Saturday, November 26, 2011

IOLib Considered Harmful to Lisp Today

...and perhaps a boon tomorrow.

Common Lisp is a very nice language to develop all kinds of servers and clients; even though there is no real standard network API. Over the years many different wrapper and portability libraries (CLOCC, ACL-COMPAT, TRIVIAL-SOCKETS, USOCKET) appeared. Often, this libraries were made by the principle of the least common denominator. For simple applications this is good enough, but complex applications have special needs.

Typical Example: Comet, event based HTTP

Webservers like NGINX are based on a event based architecture and do not consume threads by the number of concurrent connections. That way, they can manage many concurrent connections while needing much less resources than traditional worker thread based web servers. Linux, BSD (OS X too) offer means like epoll and kqueue, to implement such event based I/O multiplexers in a very efficient manner. Reading and writing needs to be non-blocking too. All this facilities are much beyond the reach of LCD libraries that just try to clean up the API differences between the different Common Lisp implementations.

The FFI to the Rescue

This were certainly part of the reasons which played a part at the conception of IOLib (http://common-lisp.net/project/iolib/). IOLib is well documented and it is integrated in Quicklisp, so it should be really easy to deploy. As always when the rescue is sought in something like the FFI I have some doubts, that the result is actually accessible and usable. Often makes many things much more complicated and portability not really easier.

Whats the Problem?

The idea is seductive: Instead of dealing with numerous incompatible APIs of CL implementations, one only needs to deal with some OSes - which mainly means "Linux" and perhaps "BSD". With much luck it will work on OS X too, but Windows is seldom a popular target for this projects. There is nothing wrong with that - if there is demand for Windows, someone could do the necessary work and if not - why bother? Well - including Windows into the mix makes such a library just LCD again, because the concepts are different enough that an API would have to be designed to be good and not just taken from one of the OSes. The solution to that is simple: Let the libraries just be OS dependent and build more abstract layers on top of them.

How is that harmful to Lisp?

Just to make one point clear: I think IOLib is a very good idea and I hope this project will be a great success. But up to now there are many problems with it, that make it a worse solution to simpler problems than USOCKET is: It reduces your possibilities to some OpenSource OSes (Linux, FreeBSD) and some OpenSource Lispsystems (CMUCL, SBCL, CLISP, ClozureCL). I've tried a whole hacking weekend to get IOLib working on LispWorks on Mac OS X - no positive results yet. Even the ClozureCL didn't work and already broke down in the CFFI-Grovel phase (I got over that using LispWorks after hammering on my system but it still doesn't work on ClozureCL). Up to now - getting IOLib running seems the be much of a PITA. So actually HOW is that harmful to Lisp? Well - I begin to see a trend that more and more CL projects begin to switch to IOLib now, which makes them only usable by parts of the community. Think about when Hunchentoot would decide to switch to IOLib tomorrow: Well - my ClozureCL based linux servers would probably have no problem with that - my LispWorks ones will stop working. My development platform is OSX - a switch to IOLib would rule that out too.

So what to do to stop the harm?

  • Think twice... better thrice before switching your project to IOLib
  • If you can't resist: Make a working fallback to something with broader support
  • If you got some time: Help hacking IOLib to work flawlessly on OS X, Windows, LispWorks and AllegroCL a.s.o.

I'm very willing to help on OS X, ClozureCL and Lispworks. Here's what I found so far:

  • CFFI-Grovel really doesn't like Spaces in paths (like /Volumes/Macintosh HD/Users...)
  • LispWorks MAKE-ARRAY has an option :allocation :static - which is needed when doing FFI pointers to arrays. CFFI knows that IOLib not

I get IOLib compiled on LispWorks for Mac OS X, but at least the DNS functions read arbitrary chunk into the buffers.

Happy Hacking