Redirect killed the nginx star

How we solved a rather unusual problem presenting itself when using nginx, xdv, django, uwsgi and Plone

Redirect killed th nginxI, in my official role of Inspector of Server Police, was called out at the beginning of this week to help solve a rather unusual murder case: that of an nginx worker process by a 302 Found.

The situation

  • The setup we had was formed by the following components:
  • A django/Satchmo site providing an ecommerce frontend
  • A Plone instance providing a CMS

Both themed via xdv, and served via nginx. In detail, the setup was for nginx to either proxy to uwsgi via a socket or to the Plone instance via a normal redirect, and apply xslt transformations if the output was of text/html type. The configuration is fairly similar, and in fact largely copied off from, the one suggested on the xdv PyPi page.

The problem

All worked like a breeze (okay, minus our mistakes, but hey) except for one thing: redirects would not work. Each time Django returned a 302 Found response, the nginx worker process would die.

I had earlier noticed that Django, when issuing a redirect response, doe not provide a response body (but substantially just the headers), and therefore I suspected that the xslt transformation segfaulted on an empty buffer (I do not have a high opinion of libxml2, implementation-wise).

The hunt

I then dug myself deeper into nginx code, starting from the last debug message before the segfault, and I noticed two things:

  • Django did not set a Content-Length header
  • the xslt module checked the Content-Length header and did not proceed with the transformation if it was 0 (whether this is due to a bug in libxml2, or just because the following module code needed a non-empty buffer, I did not investigate)
It is to be noted that it is not so idiotic from Django (or rather, its WSGI handler, as the builtin server does set the Content-Length) not to set the Content-Length header: this way it makes it simpler for an upstream WSGI middleware to rewrite the output without having to worry about also resetting Content-Length.
Most WSGI servers will try to set Content-Length themselves if it is not set, although this implies reading and buffering all the response (remember that any WSGI callable return an iterator!) which is not optimal, and something that uWSGI does not do. For similar reasons, using a WSGI middleware for setting the Content-Length might have proved difficult.

The solution

Well, fact is, Django has its own middleware implementation, which differs slightly from WSGI in that it has a richer API and does handle the response as a single object (not an iterator).
I then created this small middleware inside a file named middleware.py within one of my project packages:
class ContentLengthWriter(object):
    """A simple middleware that writes content-length
    """
    def process_response(self, request, response):
        if not response.has_header('Content-Length'):
            response['Content-Length'] = str(len(response.content))
        return response
I then added this middleware as the first of the middleware classes in settings.py. Keeping it in first position is important, as any upstream middleware that rewrites the response and does not set Content-Length will lead to funky disaligment and horrid, incomprehensible bugs:
MIDDLEWARE_CLASSES = (
    'my.package.middleware.ContentLengthWriter',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.transaction.TransactionMiddleware',
    'django.middleware.locale.LocaleMiddleware',
[...]
}

To be completely fair, the final setup was slightly different due to us using djc.recipe, and having this middleware class appear only within the production settings.

And that was all: redirects stopped killing the nginx star (with many apologies to Buggles).

Share this on

Share |

On same topics

Comments

comments powered by Disqus