Pushed by those cutting-edge users on the Nginx mailing list, I've quickly worked out the "chunkin" module which adds HTTP 1.1 chunked input support for Nginx without the need of patching the core:

    http://github.com/agentzh/chunkin-nginx-module

This module registers an access-phase handler that will eagerly read and decode incoming request bodies when a "Transfer-Encoding: chunked" header triggers a 411 error page in Nginx (hey, that's what you have to pay for avoiding patching the core ;)). For requests that are not in the "chunked" transfer encoding, this module is a "no-op".

To enable the magic, just turn on the "chunkin" config option like this:

    chunkin on;
    location /foo { ... }
    ....

No other modification is required in your nginx.conf file. (The "chunkin" directive is not allowed in the location block BTW.)

This module is still considered highly experimental and there must be some serious bugs lurking somewhere. But you're encouraged to play and test it in your non-production environment and report any quirks to me :)

Efforts have been made to reduce data copying and dynamic memory allocation, thus unfortunately raising the risk of potential buffer handling bugs caused by premature optimizations :P

This module is not supposed to be merged into the Nginx core because I've used Ragel to generate the chunked encoding parser for joy :)

The following Nginx versions have been successfully tested by this module's (very limited) test suite:

   0.8.0 ~ 0.8.24
   0.7.21 ~ 0.7.63

The test suite definitely needs more test cases and the code is hacky in various places. If you're willing to contribute, feel free to ask me for a commit bit in a private email :)

Update: I've also added a wiki page for it: http://wiki.nginx.org/NginxHttpChunkinModule

I've been reading the ``Types and Programming Languages'' ( TAPL ) book bought from Amazon.com and I've been playing with functional languages. The TAPL book uses OCaml for sample interpreter implementations and I started by learning OCaml in the hope of using it in $work. Sadly OCaml's json-static and dbd-pg/postgresql libraries gave me too much pain to install and OCaml's restricted type system made me rather unhappy. So I turned to Haskell . Well, I should have said ``returned'' ;)

The syntax of Haskell is even cleaner than OCaml and its type system is much more powerful. I quickly got familiar with Parsec and monads in general by going through the famous tutorial ``Write yourself a scheme in 48 hours''. Furthermore, the TAPL book's introduction to the lambda-calculus language helped me understand the functional programming paradigms ubiquitous in Haskell.

Great. Within just a few days, I came up with a working minisql parser for OpenResty which parses simple SQL select statements using Parsec. Building combinator parsers is really enjoyable. The whole parser is composed by several smaller ones which are in turn built upon even smaller ones and so on. Each combinator parser can also take parameters if necessary because they're essentially Haskell functions after all. The ability to do infinite look-aheads in Parsec gives me a lot of freedom to disambiguate my grammar without much hassles. Moreover, Parsec does not backtrack by default, so the parsing speed is still pretty good. Compared to the old minisql parser written in Perl (via the Parse::Yapp module), the Haskell version is at least 1 order of magnitude faster. Besides, the source of the Haskell one is also shorter and cleaner.

In the next few days, I happily mudded with the AST constructed by the parser and built four emitters atop it. These emitters are required by the new version of the OpenResty View API.

I successfully used Perl along with its modules Test::Base and IPC::Run3 to build a test scaffold for my Haskell compiler. The test suite is completely declarative and extensible. Surprisingly, I could still use the Haskell Program Coverage ( hpc ) toolchain to obtain a pretty HTML report for the code coverage of the whole test suite driven by Perl. So glad to have such a Devel::Cover equivalent on the Haskell land!

The source repository for my Haskell version of minisql compiler is located at

http://svn.openfoundry.org/openapi/trunk/haskell/

In the following journal or two I'd like to share the experience of optimizing my minisql compiler and making it several times faster.

I finally get a chance to sit down and compare OpenResty with the recently announced Google App Engine product. I've hesitated to do so in OpenResty's documentation in the fear of comparing apples to oranges. Well, they're very different things from the perspective of a platform engineer.

The key difference might be summarized by the following remarks:

   "We run your applications!"
                       -- Google App Engine

   "You run your applications; we just fill in the blanks with REST"
                       -- OpenResty

For what it's worth, both SimpleDB and CouchDB fall into the same category as OpenResty.

The true buzzword for Google App Engine, however, is the BigTable monster behind it. But unfortunately it's not accessible by means of web services (directly).

Following this reasoning, it would be really really cool if we turn the Google App Engine (thus BigTable per se) into an OpenResty storage backend, just like the existing Pg and PgFarm backends. But sadly OpenResty will have to downgrade into some kind of "reduced mode" in terms of functionalities due to the non-relational nature of BigTable. Joining two database tables (or "models" in Google App Engine's terminology) will result in a map/reduce process immediately, which may well explain why it does NOT have join operations in its GQL language. (Well, feel free to prove me wrong ;)) And that's exactly why developers for Google App Engine have been manually doing lots of stuff which used to be done automatically by a traditional RDBMS, so as to gain amazingly good data scalability offered by BigTable. It's all about balance ;) Other semi-structured data storage solutions like CouchDB and SimpleDB are suffering from similar issues here as well.

It's worth pointing out that I'm by no means saying BigTable is weak. Please don't take me wrong ;) BigTable is indeed a godsend for what it's good for.

OpenResty's PgFarm backend distributes its user data across multiple Pg nodes in terms of "accounts". Joining tables in different accounts on the PL/Proxy servers would be painful and (very) limited as well. It's a fair game :) Just different distribution granularity, isn't it? ;)