Comments on: Why WSGI? http://www.circulartriangle.com/blog/?p=26 Blog Mon, 24 May 2010 22:09:03 +0000 http://wordpress.org/?v=2.9 hourly 1 By: Matthew Wilkes http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-68 Matthew Wilkes Wed, 03 Feb 2010 02:54:26 +0000 http://www.circulartriangle.com/blog/?p=26#comment-68 Thanks for that Benjamin! Unfortunately I've not got much time to put into this site and my front-end knowledge is pretty limited. The fact that the footer navigation is just "hi" and the inconsistent use of colour on links is another issue. Not to mention the fact that the front page of the website is just a holding page. It's very much a work in progress, it's just limited by how much work I have on. Hopefully it'll be better by the next time you come here. Thanks for that Benjamin! Unfortunately I’ve not got much time to put into this site and my front-end knowledge is pretty limited. The fact that the footer navigation is just “hi” and the inconsistent use of colour on links is another issue. Not to mention the fact that the front page of the website is just a holding page.

It’s very much a work in progress, it’s just limited by how much work I have on. Hopefully it’ll be better by the next time you come here.

]]> By: Benjamin http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-67 Benjamin Wed, 03 Feb 2010 00:19:15 +0000 http://www.circulartriangle.com/blog/?p=26#comment-67 Please don't take this as trolling, but simply constructive criticism. The post was great, but the way the fonts render on Chrome for Mac is seriously almost unreadable. Please don’t take this as trolling, but simply constructive criticism.
The post was great, but the way the fonts render on Chrome for Mac is seriously almost unreadable.

]]> By: Matt Hamilton http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-65 Matt Hamilton Tue, 02 Feb 2010 10:48:20 +0000 http://www.circulartriangle.com/blog/?p=26#comment-65 As Martin pointed out lxml is fast. Its added processing time is negligible addition to the time of actually serving up the request in most case. And if adding 10ms to your app is a big deal, then you probably wouldn't be doing this approach anyway. My 'Lipstick on a Pig' system that Matt mentioned above is a good example of this. Admittedly, the original .NET system we are proxying is insanely slow, but we then add on a WSGI pipeline of about 8-10 middleware stages. Most of these stages at the moment do an lxml parse/unparse step at each step (no optimisation done yet... this is a prime candidate for repoze.xmliter), yet the added delay is so small it can't be measured with any statistical certainty. This combined with the fact we are then sending about a 1/3 of the resultant HTML compared to previously, and it is well formed (original .NET produced HTML was not) means that it *seems* faster to the end user as the download and render speeds for the browser are much quicker. Also coupled with the fact we were able to break our problem down in to many small steps (quite a few are re-used as we have about 6 different pipelines, each with a variation of which middleware is called in each one) that we could test independently means that overall we have a better system than we would have done had we not had the flexibility of WSGI. -Matt As Martin pointed out lxml is fast. Its added processing time is negligible addition to the time of actually serving up the request in most case. And if adding 10ms to your app is a big deal, then you probably wouldn’t be doing this approach anyway.

My ‘Lipstick on a Pig’ system that Matt mentioned above is a good example of this. Admittedly, the original .NET system we are proxying is insanely slow, but we then add on a WSGI pipeline of about 8-10 middleware stages. Most of these stages at the moment do an lxml parse/unparse step at each step (no optimisation done yet… this is a prime candidate for repoze.xmliter), yet the added delay is so small it can’t be measured with any statistical certainty. This combined with the fact we are then sending about a 1/3 of the resultant HTML compared to previously, and it is well formed (original .NET produced HTML was not) means that it *seems* faster to the end user as the download and render speeds for the browser are much quicker.

Also coupled with the fact we were able to break our problem down in to many small steps (quite a few are re-used as we have about 6 different pipelines, each with a variation of which middleware is called in each one) that we could test independently means that overall we have a better system than we would have done had we not had the flexibility of WSGI.

-Matt

]]> By: Chris McDonough http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-63 Chris McDonough Mon, 01 Feb 2010 19:46:10 +0000 http://www.circulartriangle.com/blog/?p=26#comment-63 I used to be all about WSGI middleware as componentry. After much gnashing of teeth, these days, not so much. TBH, I'd probably instead built this functionality into each application rather than trying to develop a generic captcha middleware thing. I used to be all about WSGI middleware as componentry. After much gnashing of teeth, these days, not so much. TBH, I’d probably instead built this functionality into each application rather than trying to develop a generic captcha middleware thing.

]]> By: Matthew Wilkes http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-62 Matthew Wilkes Mon, 01 Feb 2010 14:20:26 +0000 http://www.circulartriangle.com/blog/?p=26#comment-62 Alan: Yeah, that's what I was thinking when I mentioned making it caching-aware. With a carefully tuned grace time on your objects no users will even see the (slight) slowdown. Alan: Yeah, that’s what I was thinking when I mentioned making it caching-aware. With a carefully tuned grace time on your objects no users will even see the (slight) slowdown.

]]> By: Alan Hoey http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-61 Alan Hoey Mon, 01 Feb 2010 13:20:46 +0000 http://www.circulartriangle.com/blog/?p=26#comment-61 There's potentially a very nice extension to this if you're looking for a performance boost - have the middleware inject an ESI rather than the new form field(s). Then you can cache the entire form and keep the CAPTCHA generation in the fast part of the pipeline. Obviously you need to be running something like Varnish that understands ESI to do this but it would probably make a measurable difference where you're serving a lot of forms. There’s potentially a very nice extension to this if you’re looking for a performance boost – have the middleware inject an ESI rather than the new form field(s). Then you can cache the entire form and keep the CAPTCHA generation in the fast part of the pipeline. Obviously you need to be running something like Varnish that understands ESI to do this but it would probably make a measurable difference where you’re serving a lot of forms.

]]> By: SK http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-60 SK Mon, 01 Feb 2010 12:27:36 +0000 http://www.circulartriangle.com/blog/?p=26#comment-60 Aspelli: not the point really. Processing and monkeypatching whole responses every time they are generated just to insert a CAPTCHA is stupid. Generating an XML tree rather than using a regexp is just a variation on insanity. Wilkes: yes, premature optimization is bad, but refusing to bother about efficiency at the design phase is even worse. Aspelli: not the point really. Processing and monkeypatching whole responses every time they are generated just to insert a CAPTCHA is stupid. Generating an XML tree rather than using a regexp is just a variation on insanity.

Wilkes: yes, premature optimization is bad, but refusing to bother about efficiency at the design phase is even worse.

]]> By: Martin Aspeli http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-59 Martin Aspeli Mon, 01 Feb 2010 05:46:34 +0000 http://www.circulartriangle.com/blog/?p=26#comment-59 @SK - your moronic side-swipe aside, parsing an output document with the lxml HTML parser is relatively effecient. Most web pages consist of only a few kb of HTML at most, and lxml is fast. Working with a well-defined data structure is a lot safer and potentially more efficient than doing a lot of regex matching, for instance. Using something like repoze.xmliter, you've also got a way to do that which allows you to only parse once even if you have multiple middleware layers. As Matt pointed out, there are situations where layering this functionality on top of an application makes a lot of sense. Response post-processing is a very commonly used technique in a lot of frameworks. WSGI makes it nicer than most. Martin @SK – your moronic side-swipe aside, parsing an output document with the lxml HTML parser is relatively effecient. Most web pages consist of only a few kb of HTML at most, and lxml is fast. Working with a well-defined data structure is a lot safer and potentially more efficient than doing a lot of regex matching, for instance. Using something like repoze.xmliter, you’ve also got a way to do that which allows you to only parse once even if you have multiple middleware layers.

As Matt pointed out, there are situations where layering this functionality on top of an application makes a lot of sense. Response post-processing is a very commonly used technique in a lot of frameworks. WSGI makes it nicer than most.

Martin

]]> By: Matthew Wilkes http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-57 Matthew Wilkes Sun, 31 Jan 2010 22:52:59 +0000 http://www.circulartriangle.com/blog/?p=26#comment-57 If only I'd thought to add "illustrative pseudocode" somewhere in that post. First-off, it's obvious to anyone that's written any code professionally that this isn't a suggested piece of production code. I mean, just look at it, it's <em>clearly</em> a proof of concept that took about 5 minutes to write. For one thing it would fail on the vast majority of HTTP traffic, it was written in a browser to show one possible way of doing it. Would you perhaps feel more comfortable if I said "XSLT compilation", or maybe "regular expression"? Not that it matters, anyway, the real issue with this example is that it treats the response body as a string, not as an iterable. You try doing that on your sites, that's the real killer, not lxml's C. Even in this naïve case, I'd estimate it would add maybe 10ms to each request. If you're coding for fastest-possible responses right from the start you're <strong>doing it wrong</strong>. The whole point of this is that by using an architecture like this your app and the CAPTCHA system can evolve independently. Once you've made an implementation that is caching-aware you're down to nothing at all. What you have at the end, however, is far superior to what you'd have if you'd built yet another one-shot CAPTCHA system and you're still up on time compared to integrating somebody else's with the FooFormLibrary that you're using. So, yes, I'm happy to be of the mindset that it's better to take the hit of parsing some responses to get a better system overall, otherwise I'd be knowingly producing inferior code because the superior method makes me feel uncomfortable. If only I’d thought to add “illustrative pseudocode” somewhere in that post. First-off, it’s obvious to anyone that’s written any code professionally that this isn’t a suggested piece of production code. I mean, just look at it, it’s clearly a proof of concept that took about 5 minutes to write. For one thing it would fail on the vast majority of HTTP traffic, it was written in a browser to show one possible way of doing it. Would you perhaps feel more comfortable if I said “XSLT compilation”, or maybe “regular expression”? Not that it matters, anyway, the real issue with this example is that it treats the response body as a string, not as an iterable. You try doing that on your sites, that’s the real killer, not lxml’s C. Even in this naïve case, I’d estimate it would add maybe 10ms to each request.

If you’re coding for fastest-possible responses right from the start you’re doing it wrong. The whole point of this is that by using an architecture like this your app and the CAPTCHA system can evolve independently. Once you’ve made an implementation that is caching-aware you’re down to nothing at all. What you have at the end, however, is far superior to what you’d have if you’d built yet another one-shot CAPTCHA system and you’re still up on time compared to integrating somebody else’s with the FooFormLibrary that you’re using.

So, yes, I’m happy to be of the mindset that it’s better to take the hit of parsing some responses to get a better system overall, otherwise I’d be knowingly producing inferior code because the superior method makes me feel uncomfortable.

]]> By: SK http://www.circulartriangle.com/blog/?p=26&cpage=1#comment-56 SK Sun, 31 Jan 2010 21:09:52 +0000 http://www.circulartriangle.com/blog/?p=26#comment-56 So you're parsing and building an XML tree of the response for each request in the hope that there's a captcha to replace there? With such a mindset, no wonder zope/plone has a reputation for very poor performance. And no wonder WSGI is perceived as snake oil if that's all you can come up with. So you’re parsing and building an XML tree of the response for each request in the hope that there’s a captcha to replace there? With such a mindset, no wonder zope/plone has a reputation for very poor performance. And no wonder WSGI is perceived as snake oil if that’s all you can come up with.

]]>