Saturday, November 26, 2011

IOLib Considered Harmful to Lisp Today

...and perhaps a boon tomorrow.

Common Lisp is a very nice language to develop all kinds of servers and clients; even though there is no real standard network API. Over the years many different wrapper and portability libraries (CLOCC, ACL-COMPAT, TRIVIAL-SOCKETS, USOCKET) appeared. Often, this libraries were made by the principle of the least common denominator. For simple applications this is good enough, but complex applications have special needs.

Typical Example: Comet, event based HTTP

Webservers like NGINX are based on a event based architecture and do not consume threads by the number of concurrent connections. That way, they can manage many concurrent connections while needing much less resources than traditional worker thread based web servers. Linux, BSD (OS X too) offer means like epoll and kqueue, to implement such event based I/O multiplexers in a very efficient manner. Reading and writing needs to be non-blocking too. All this facilities are much beyond the reach of LCD libraries that just try to clean up the API differences between the different Common Lisp implementations.

The FFI to the Rescue

This were certainly part of the reasons which played a part at the conception of IOLib ( IOLib is well documented and it is integrated in Quicklisp, so it should be really easy to deploy. As always when the rescue is sought in something like the FFI I have some doubts, that the result is actually accessible and usable. Often makes many things much more complicated and portability not really easier.

Whats the Problem?

The idea is seductive: Instead of dealing with numerous incompatible APIs of CL implementations, one only needs to deal with some OSes - which mainly means "Linux" and perhaps "BSD". With much luck it will work on OS X too, but Windows is seldom a popular target for this projects. There is nothing wrong with that - if there is demand for Windows, someone could do the necessary work and if not - why bother? Well - including Windows into the mix makes such a library just LCD again, because the concepts are different enough that an API would have to be designed to be good and not just taken from one of the OSes. The solution to that is simple: Let the libraries just be OS dependent and build more abstract layers on top of them.

How is that harmful to Lisp?

Just to make one point clear: I think IOLib is a very good idea and I hope this project will be a great success. But up to now there are many problems with it, that make it a worse solution to simpler problems than USOCKET is: It reduces your possibilities to some OpenSource OSes (Linux, FreeBSD) and some OpenSource Lispsystems (CMUCL, SBCL, CLISP, ClozureCL). I've tried a whole hacking weekend to get IOLib working on LispWorks on Mac OS X - no positive results yet. Even the ClozureCL didn't work and already broke down in the CFFI-Grovel phase (I got over that using LispWorks after hammering on my system but it still doesn't work on ClozureCL). Up to now - getting IOLib running seems the be much of a PITA. So actually HOW is that harmful to Lisp? Well - I begin to see a trend that more and more CL projects begin to switch to IOLib now, which makes them only usable by parts of the community. Think about when Hunchentoot would decide to switch to IOLib tomorrow: Well - my ClozureCL based linux servers would probably have no problem with that - my LispWorks ones will stop working. My development platform is OSX - a switch to IOLib would rule that out too.

So what to do to stop the harm?

  • Think twice... better thrice before switching your project to IOLib
  • If you can't resist: Make a working fallback to something with broader support
  • If you got some time: Help hacking IOLib to work flawlessly on OS X, Windows, LispWorks and AllegroCL a.s.o.

I'm very willing to help on OS X, ClozureCL and Lispworks. Here's what I found so far:

  • CFFI-Grovel really doesn't like Spaces in paths (like /Volumes/Macintosh HD/Users...)
  • LispWorks MAKE-ARRAY has an option :allocation :static - which is needed when doing FFI pointers to arrays. CFFI knows that IOLib not

I get IOLib compiled on LispWorks for Mac OS X, but at least the DNS functions read arbitrary chunk into the buffers.

Happy Hacking


  1. As always, I accept patches or pecuniary contributions for improving IOlib(I've received both), since I cannot currently justify buying an ACL/LW license or an OSX machine.

    As for the features of IOlib, much of them are not present in any other CL library; so if one needs such features there's little choice(one could always develop a new library from scratch).

    As for the portability to certain OSes, if your deployment OS is Linux, why not develop in a virtual machine ? I find it awkward how people stubbornly insist in developing on OSX, given how awful it is as a Unix implementation

  2. I don't think this it's fair to single out IOLib in particular (it does prominently feature that it's limited to certain implementations and platforms), but libraries claiming to be "Lisp-this" or "Lisp-that" but relying on FFI in general. That's true even for things like math libraries, where there's no need for FFI or platform-dependent stuff, but a lot of "Lisp-X" math libraries really do most of their work by calling out to C or Fortran.

  3. @fenlix
    You actually do not have to buy a LW license to use it as an IDE. The restrictions of the personal edition mostly hinder commercial deployment and thats not really the issue here. I still think that LW is the best Lisp IDE today. It's feels like a modernized variant of Emacs in many ways. Since 6.0 they even ported the IDE over to GTK. I think there is actually no reason not to have a LW personal installation around - its a really nice piece of software.

    Please note the subtitle I wrote in the posting. I tried hard to make clear that IOLib offers unique features and is a good choice when there is a need for that. My point is, that - today - IOLib is not a good generic choice if something like usocket would be enough. Its implementation base is still to small.

    Well - I use Linux, MS Windows and Mac OS X on a daily base. I would never call colleagues "stubborn" over their personal choice. I've used BSD and then Linux as my main development machine for many years. Then I actually wanted to stop wasting time being an Administrator for my development machine and fix kernels to get suspend working. Since that time I use OS X and it is a much more efficient working environment for me. I'm a first class Lisper and only second class "Unixer" though. ;)

  4. @Vladimir
    I didn't write this post to single out IOLib in a negative way. I wrote it, because I think it offers some important things and I really want that to be part of the Lisp world. Thats why I suggested interested parties getting involved in making IOLib a broader experience. I've spent time on this by myself and wouldn't have done it if I just want to "bash" something like you seem to imply.

    Your point though: Yes I also think that using FFI to callout to C or Fortran would also be only a good choice if a Lisp-native implementation would not be feasible. The parallel to the IOLib case would be, that it is a bad idea for a library implementor to rely on such special case FFI libraries for things that are also offered by portable libraries. Another example would be cryptography. Ironclad offers alot - why to the hell call out to C or even call out to an executable using RUN-PROGRAM?

    Thats one of the things that made ASDF-INSTALL bad and that makes QuickLisp much much better.

  5. What ironclad offers is insufficient and, outside SBCL, very slow. Try comparing ironclad/Clisp or ironclad/LW to icc(or even just gcc) and see the difference

  6. I meant openssl/icc

  7. @fenlix
    Calling out to FFI can be very very slow. You want to avoid FFI as long as you can in long running high scaling lisp servers. Using a rock solid, supported lisp implementation is always a good idea though.

    Ironclad is actually very fast on LW 64 bit. The main problem for LW 32 bit with ironclad is boxed 32bit integer arithmetic. Ironclad is slower as LW 64 on ClozureCL but not unusable.

    Ironclad is at least readable code ( can't say that about the mess that is OpenSSL). Ironclad alone is no replacement for OpenSSL yet; For most cryptographic work in Lisp, Ironclad is the better choice though.

    Why do you actually use Lisp at all? Wouldnt it be easier for you to use C directly?

  8. funny to stumble on this after spending the weekend to get hu.dwim suite running on my Darwin box. IOlib was certainly the most problematic component very stubborn to play nicely... That said, I do think there has been a great deal of creative thought and effort put into it and I appreciate that even if I may disagree on some of the specifics here or there. A project with a scope as large as iolib would definitely be improved by getting more developers on board specializing in the various platforms.