- X-Sendfile (http://flask.pocoo.org/docs/api/#flask.Flask.use_x_sendfile)

Seems to be supported by Flask, but needs an Apache module. This causes the Python app to just emit the headers of the response, including a special X-Sendfile one. The X-Sendfile header tells the Apache process the filesystem path of the file and lets it handle the actual sending of the data, including kernel-level sendfile support, so the file isn’t even loaded into Apache’s memory. Nginx implements the same idea (http://wiki.nginx.org/XSendfile).

- FileWrapperExtension (http://code.google.com/p/modwsgi/wiki/FileWrapperExtension)

This is a similar Python/WSGI specific idea of the above, but there seem to be more cases where it fails or needs extra configuration.

- Use Python 3.2′s new GIL (http://docs.python.org/3/whatsnew/3.2.html#multi-threading)

It seems mod_wgsi 3 supports Python 3. If the application doesn’t do much else, you could try running it under Python 3.2 to take advantage of the new GIL / thread scheduling. I’d expect this to give more uniform response times and help with the 90%+ times.
If you are interested in the back story, there’s lots of information in Dave Beazley’s blog posts about the GIL (http://www.dabeaz.com/blog.html).

- Use taskset to pin processes to CPU’s

The taskset command allows one to pin a process to a specific CPU. If you have multiple processes and threads, the default scheduling often ends up trying to schedule one or more threads of the same Python process on different CPU cores. But the GIL often prevents them from actually executing in parallel. The net result for any given thread is, that it’s more or less randomly executed on all CPU cores, which destroys all the L1/L2 caches or even memory access times in NUMA systems.

For the number of processes/threads, it’s indeed best to do actual experimentation and measurements. Most general advice out there suggests to oversubscribe and have twice as many threads as CPU cores. But that usually only holds true for applications with a mixed CPU and i/o load.

So, I’m basically one attribute-set and an Apache module away from doing that. If I try it, I’ll update the results here.

I wouldn’t expect network speed to play much into the results, since either Apache or Python should be able to keep up with any reasonable pipe. However, slow readers could plug up Python threads, reducing the pool for everybody else.

That said, there are certainly many other variables to explore!

and also studying the variability along different file sizes.

Finally the last thing would be to see how it behaves and how much the performance is impacted in a test with different type of bandwidth. The latency of networks usually dwarfes the local performance issues.