Earlier today I tried to write an explanation of why WSGI technologies are useful to developers, but each attempt sounded too much like snake-oil. When you start to list what advantages you get from applying WSGI best-practises to a problem the results sound fantastic, and frankly unbelievable, but they’re all true.
Caveat emptor: I haven’t properly tested the code in this post, it should all be treated as illustrative pseudo-code. Also, this is really long, you might want to get a cup of tea.
Separation of front- and back-end
There has been a lot of noise about deliverance in the Plone community recently, many consultancy companies have deployed Deliverance-based sites and even plone.org has a deliverance front-end. The most important advantage of this pattern is that it stops front-end developers being blocked by back-end considerations.
When developing a Plone site a designer can very rarely jump straight into modifying templates on day one. For one thing, the markup isn’t all together, such as with the portal_tabs navigation bar. If there is a custom markup needed for these tabs first new viewlets and views need to be created to override the templates being used, and a test instance set up to give the markup guys something to work with. Otherwise there’d be another significant amount of work to do in integrating it later in the project.
Either way, nobody can see these new tabs in place until there has been some backend developer time allocated to the problem. When you consider the sheer number of places markup changes happen, this is clearly an untenable situation.
By using a transformational middleware like deliverance developers can start writing production markup on day one of a project. This can (and does!) happen before the backend developers have even finished talking about what technologies to use. This extra time and freedom truly is invaluable, as it means at no point will the look and feel of the site block development of the functionality or vice-versa.
Right from the start functionality and UI can go to humant testers who help develop the automated test suite. As time progresses and the deliverance theme starts incorporating more rules to integrate with the back-end testing of the integration of the two can begin. Without this separation components of the UI can’t really be tested in isolation.
Reusable components
There are two very popular Plone packages providing CAPTCHA support, collective.captcha and collective.recaptcha. To embed a CAPTCHA on your page you include:
<tal:captcha replace="here/@@captcha/image_tag" />
<input type="text" name="captcha" />and in the code that verifies your form submission you extract the value from the request and pass it back to the captcha view:
captcha = context.REQUEST.form.get("captcha", None)
if not context.restrictedTraverse("@@captcha").verify(captcha):
raise SpamBotException("CAPTCHA failed")While there are convenience methods for integrating this with the various places forms are generated in Plone, this is really quite nasty. While we have a nice, generic method for getting and verifying CAPTCHAs, it is very much tied to Plone and its various forms.
If we were to write this as a WSGI middleware we’d want to take the CAPTCHA generation and checking away from the application, so there are no import dependencies. In essence, the CAPTCHA field is just a check if the user is human, the simplest one being a simple checkbox. This isn’t going to filter out many spambots, but it does capture the essence of the problem, and is very simple to include in forms:
<input type="checkbox" name="isHuman" id="isHuman" />This can be included in any forms readily, as any form library worth its salt can create a simple checkbox, then the backing code merely ensures that it’s a required field. Simples.
However, we’ve got all these nice CAPTCHA libraries, so we create a middleware that grabs this from outgoing requests and replaces it with the full CAPTCHA that we’d otherwise have pulled directly into the form. When a request comes back through with a CAPTCHA response in it we verify it’s correct and use that to set the value of isHuman. That would look a little like this:
class CAPTCHAMiddleware(object):
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
req = Request(environ)
if "captcha" in req.params:
value = req.params["captcha"]
del req.params["captcha"]
req.params["isHuman"] = self.verify(value)
res = req.get_response(self.app)
XML = lxml.etree.XML(res.body)
captcha = XML.xpath("//input[@name='isHuman']")
if captcha:
new = lxml.etree.XML(self.getCapcha())
captcha[0].getparent().replace(captcha[0], new)
res.body = lxml.etree.tostring(XML)
return res(environ, start_response)This method can then be tested to destruction to ensure that there are no ways of circumventing the CAPTCHA. All the time the site still works with its naïve CAPTCHA in place. Any updates to the CAPTCHA product can be done, tested and deployed separately to the main site.
Let’s do a feature comparison:
| @@captcha | CAPTCHAMiddleware | |
|---|---|---|
| Can switch to other implementations | ✔ | ✔ |
| Works without customising a form | ✘ | ✘ |
| Works with any form library OOTB | ✘ | ✔ |
| No vendor lock-in | ✘ | ✔ |
Seamless upgrades of legacy sites
So, there’s a distinct advantage shown above, but it’s not mind-blowing. The second element still bothers me. This would mean that although it’s easy to create new sites that use this CAPTCHA middleware it’s not easily backported. While having this discussion with Alan Hoey of Team Rubber he suggested that he’d not use a new input method, but have a configuration file for which forms need CAPTCHAs added, and simply shield the backend from them.
At first, this sounded like a simple disagreement on implementation, until I realised that it could be elegantly implemented as a second middleware!
In this case, you’d not only have the CAPTCHA middleware I describe above, but a second middleware lower down in the stack which takes a configuration file and adds isHuman checkboxes to other forms. If the isHuman value in the request is set to False for one of these forms it will simply raise an error.
There is, however, good reason to keep with the original architecture instead of relying on searching for the isHuman checkbox, and that is it allows applications to signal that they’re aware of how CAPTCHAs work and handle them elegantly. Also, it allows careful positioning of CAPTCHAs in forms, the opportunity earlier to test they are shown everywhere that is needed and easier handling of dynamic forms, such as comments.
This new middleware, however, would make integration into older sites easy, and the feature matrix would start to look like:
| @@captcha | CAPTCHAMiddleware | Two Middlewares | |
|---|---|---|---|
| Can switch to other implementations | ✔ | ✔ | ✔ |
| Works without customising a form | ✘ | ✘ | ✔ |
| Works with any form library OOTB | ✘ | ✔ | ✔ |
| No vendor lock-in | ✘ | ✔ | ✔ |
We have a winner! By breaking the problem down into small sections we now have a very extensible system that could be applied to new and old sites with a minimum of effort. We’re not tied to Plone, or even Zope, and both the backend application and the CAPTCHA system can be readily tested in isolation.
It’s this kind of modularisation that provides the great wins for WSGI. This originally came out of discussion in #repoze in which it was suggested that the environ could hold a special environment variable and call a special function to add a CAPTCHA to a form. Such a system would fail all 4 of my tests above, and unfortunately many popular WSGI middlewares are written this way. Taking the extra time and fitting your problem to normal HTTP requests and responses makes for much more re-usable middlewares.
Testing
I’ve mentioned this a few times, but it’s worth really emphasising. Many WSGI middlewares will be barely a hundred lines long, many will be much shorter. The size of the tests for these can be orders of magnitude greater than the code itself. Many people claim that 100% test coverage just isn’t possible, but when writing this kind of function it’s not a very good idea but you can easily write a very comprehensive set of tests in a short period of time.
As the middleware can evolve on its own, separately from the application that spawned it, it can get new tests and releases as part of the development of other sites. This means that old applications see benefits of new client work faster and more often than monolithic applications.
Outside the process
Matt Hamilton’s excellent Lipstick on a Pig talk is a great example of this. WSGI middlewares can be layered onto HTTP proxies, allowing them to work on any backend, Python or not. wsapi4plone can work in the same way, it is completely agnostic of whether it is plumbed directly into Zope by a WSGI stack, or if it’s being proxied out to a different instance.
By following the WSGI best-practises of depending on HTTP requests and responses rather than making Python calls between layers this kind of flexibility is built directly into your system. This means a legacy site can easily have WSGI middlewares layered on-top of it without even restarting the Zope process.
Cool, huh?
Tags: bestpractise, Plone, python, wsgi
So you’re parsing and building an XML tree of the response for each request in the hope that there’s a captcha to replace there? With such a mindset, no wonder zope/plone has a reputation for very poor performance. And no wonder WSGI is perceived as snake oil if that’s all you can come up with.
If only I’d thought to add “illustrative pseudocode” somewhere in that post. First-off, it’s obvious to anyone that’s written any code professionally that this isn’t a suggested piece of production code. I mean, just look at it, it’s clearly a proof of concept that took about 5 minutes to write. For one thing it would fail on the vast majority of HTTP traffic, it was written in a browser to show one possible way of doing it. Would you perhaps feel more comfortable if I said “XSLT compilation”, or maybe “regular expression”? Not that it matters, anyway, the real issue with this example is that it treats the response body as a string, not as an iterable. You try doing that on your sites, that’s the real killer, not lxml’s C. Even in this naïve case, I’d estimate it would add maybe 10ms to each request.
If you’re coding for fastest-possible responses right from the start you’re doing it wrong. The whole point of this is that by using an architecture like this your app and the CAPTCHA system can evolve independently. Once you’ve made an implementation that is caching-aware you’re down to nothing at all. What you have at the end, however, is far superior to what you’d have if you’d built yet another one-shot CAPTCHA system and you’re still up on time compared to integrating somebody else’s with the FooFormLibrary that you’re using.
So, yes, I’m happy to be of the mindset that it’s better to take the hit of parsing some responses to get a better system overall, otherwise I’d be knowingly producing inferior code because the superior method makes me feel uncomfortable.
@SK – your moronic side-swipe aside, parsing an output document with the lxml HTML parser is relatively effecient. Most web pages consist of only a few kb of HTML at most, and lxml is fast. Working with a well-defined data structure is a lot safer and potentially more efficient than doing a lot of regex matching, for instance. Using something like repoze.xmliter, you’ve also got a way to do that which allows you to only parse once even if you have multiple middleware layers.
As Matt pointed out, there are situations where layering this functionality on top of an application makes a lot of sense. Response post-processing is a very commonly used technique in a lot of frameworks. WSGI makes it nicer than most.
Martin
Aspelli: not the point really. Processing and monkeypatching whole responses every time they are generated just to insert a CAPTCHA is stupid. Generating an XML tree rather than using a regexp is just a variation on insanity.
Wilkes: yes, premature optimization is bad, but refusing to bother about efficiency at the design phase is even worse.
There’s potentially a very nice extension to this if you’re looking for a performance boost – have the middleware inject an ESI rather than the new form field(s). Then you can cache the entire form and keep the CAPTCHA generation in the fast part of the pipeline. Obviously you need to be running something like Varnish that understands ESI to do this but it would probably make a measurable difference where you’re serving a lot of forms.
Alan: Yeah, that’s what I was thinking when I mentioned making it caching-aware. With a carefully tuned grace time on your objects no users will even see the (slight) slowdown.
I used to be all about WSGI middleware as componentry. After much gnashing of teeth, these days, not so much. TBH, I’d probably instead built this functionality into each application rather than trying to develop a generic captcha middleware thing.
As Martin pointed out lxml is fast. Its added processing time is negligible addition to the time of actually serving up the request in most case. And if adding 10ms to your app is a big deal, then you probably wouldn’t be doing this approach anyway.
My ‘Lipstick on a Pig’ system that Matt mentioned above is a good example of this. Admittedly, the original .NET system we are proxying is insanely slow, but we then add on a WSGI pipeline of about 8-10 middleware stages. Most of these stages at the moment do an lxml parse/unparse step at each step (no optimisation done yet… this is a prime candidate for repoze.xmliter), yet the added delay is so small it can’t be measured with any statistical certainty. This combined with the fact we are then sending about a 1/3 of the resultant HTML compared to previously, and it is well formed (original .NET produced HTML was not) means that it *seems* faster to the end user as the download and render speeds for the browser are much quicker.
Also coupled with the fact we were able to break our problem down in to many small steps (quite a few are re-used as we have about 6 different pipelines, each with a variation of which middleware is called in each one) that we could test independently means that overall we have a better system than we would have done had we not had the flexibility of WSGI.
-Matt
Please don’t take this as trolling, but simply constructive criticism.
The post was great, but the way the fonts render on Chrome for Mac is seriously almost unreadable.
Thanks for that Benjamin! Unfortunately I’ve not got much time to put into this site and my front-end knowledge is pretty limited. The fact that the footer navigation is just “hi” and the inconsistent use of colour on links is another issue. Not to mention the fact that the front page of the website is just a holding page.
It’s very much a work in progress, it’s just limited by how much work I have on. Hopefully it’ll be better by the next time you come here.
hello guys
I just want to say hi
Very good page.