Tuesday, May 15, 2012

CLOS Class Naming

The classes of the Common Lisp spec, the MOP and big CLOS projects like CLIM use some naming conventions which may not make sense at first. I'm speaking of the STANDARD- and BASIC- prefixes. What is the idea behind them?

Protocol Oriented Programming

In CLOS one defines "protocols" using classes and generic functions. Remember: Methods do belong to generic functions and not to classes. There are no dedicated means to define protocols - it is idiomatic. A common practice is to define "protocol classes" which often do not have any slots at all. They serve as superclasses to all classes which implement the protocol.

Basics and Standards

In the set of protocol implementation classes there may be one, which contains a minimal complete implementation and/or there might be one which is something like the default implementation. Assuming a protocol for FOOBAR defined through the protocol-class FOOBAR the minimal implementation could be called BASIC-FOOBAR and the default one STANDARD-FOOBAR. The BASIC-** one if often called FUNDAMENTAL-** too.


It is often a good idea to implement STANDARD-** by inheriting the BASIC-** implementation, overriding the necessary parts. A protocol user who wants to implement his own implementation can decide on which level he wants to get in: He could inherit from STANDARD-** if his implementations is only a small variant. He may inherit from BASIC-** if the implementation is more complicated and perhaps incompatible to what STANDARD-** does. If even BASIC-** is to strict, he could try to implement the protocol class completely on its own - but this is then more like implementing interfaces in classical object oriented languages, where you cannot inherit behaviour at all. You have to provide methods for any necessary generic function the protocol is made of. This is the most work and implementing a protocol is often more than just following the interfaces.

Saturday, November 26, 2011

IOLib Considered Harmful to Lisp Today

...and perhaps a boon tomorrow.

Common Lisp is a very nice language to develop all kinds of servers and clients; even though there is no real standard network API. Over the years many different wrapper and portability libraries (CLOCC, ACL-COMPAT, TRIVIAL-SOCKETS, USOCKET) appeared. Often, this libraries were made by the principle of the least common denominator. For simple applications this is good enough, but complex applications have special needs.

Typical Example: Comet, event based HTTP

Webservers like NGINX are based on a event based architecture and do not consume threads by the number of concurrent connections. That way, they can manage many concurrent connections while needing much less resources than traditional worker thread based web servers. Linux, BSD (OS X too) offer means like epoll and kqueue, to implement such event based I/O multiplexers in a very efficient manner. Reading and writing needs to be non-blocking too. All this facilities are much beyond the reach of LCD libraries that just try to clean up the API differences between the different Common Lisp implementations.

The FFI to the Rescue

This were certainly part of the reasons which played a part at the conception of IOLib (http://common-lisp.net/project/iolib/). IOLib is well documented and it is integrated in Quicklisp, so it should be really easy to deploy. As always when the rescue is sought in something like the FFI I have some doubts, that the result is actually accessible and usable. Often makes many things much more complicated and portability not really easier.

Whats the Problem?

The idea is seductive: Instead of dealing with numerous incompatible APIs of CL implementations, one only needs to deal with some OSes - which mainly means "Linux" and perhaps "BSD". With much luck it will work on OS X too, but Windows is seldom a popular target for this projects. There is nothing wrong with that - if there is demand for Windows, someone could do the necessary work and if not - why bother? Well - including Windows into the mix makes such a library just LCD again, because the concepts are different enough that an API would have to be designed to be good and not just taken from one of the OSes. The solution to that is simple: Let the libraries just be OS dependent and build more abstract layers on top of them.

How is that harmful to Lisp?

Just to make one point clear: I think IOLib is a very good idea and I hope this project will be a great success. But up to now there are many problems with it, that make it a worse solution to simpler problems than USOCKET is: It reduces your possibilities to some OpenSource OSes (Linux, FreeBSD) and some OpenSource Lispsystems (CMUCL, SBCL, CLISP, ClozureCL). I've tried a whole hacking weekend to get IOLib working on LispWorks on Mac OS X - no positive results yet. Even the ClozureCL didn't work and already broke down in the CFFI-Grovel phase (I got over that using LispWorks after hammering on my system but it still doesn't work on ClozureCL). Up to now - getting IOLib running seems the be much of a PITA. So actually HOW is that harmful to Lisp? Well - I begin to see a trend that more and more CL projects begin to switch to IOLib now, which makes them only usable by parts of the community. Think about when Hunchentoot would decide to switch to IOLib tomorrow: Well - my ClozureCL based linux servers would probably have no problem with that - my LispWorks ones will stop working. My development platform is OSX - a switch to IOLib would rule that out too.

So what to do to stop the harm?

  • Think twice... better thrice before switching your project to IOLib
  • If you can't resist: Make a working fallback to something with broader support
  • If you got some time: Help hacking IOLib to work flawlessly on OS X, Windows, LispWorks and AllegroCL a.s.o.

I'm very willing to help on OS X, ClozureCL and Lispworks. Here's what I found so far:

  • CFFI-Grovel really doesn't like Spaces in paths (like /Volumes/Macintosh HD/Users...)
  • LispWorks MAKE-ARRAY has an option :allocation :static - which is needed when doing FFI pointers to arrays. CFFI knows that IOLib not

I get IOLib compiled on LispWorks for Mac OS X, but at least the DNS functions read arbitrary chunk into the buffers.

Happy Hacking

Sunday, December 20, 2009

On Immutable Minds

I've wanted to write about Clojure for a while. It has not happened yet because I just didn't find the time for it. Its still not time for it, but today I thought about a phenomenon I observed since Clojures new popularity more than before. There is a particular kind of people who seemingly have problems to live with multiple options. I've called them the "Immutable Minds" because they are incapable of changing their mind without completely replacing all things who were there before. I called it also that way, because I think immutable is such a nice tagword this times ;-). This isn't anything special with Clojure (there is nothing wrong with it)

The Immutable Mind is single-minded

When Common Lisp was fresh (actually refreshed) the Immutable Minds found it and replaced whatever they had in their minds before. But what they found seemed like a trap: It was the obvious "single option" for them, but Common Lisp itself was full of things "multi": It had multi-methods, multi-inheritance, multiple namespaces. It even had multiple implementations (yes that really got critized!) and even worse: Multiple opinions. They couldn't use it but it had replaced what was there before (which was already declared "dead" and they've lost the reference), so they did the only thing they still could do: Pestering people outside the community about Lisp being "The One And Only (tm)" while annoying the community members by constant requests on how to change lisp to be less multi but more single. Single namespace, single inheritance, single implementation and the most important: Single opinion (theirs). When they didn't get what they wanted they got angry and filled with bitterness. Then came Clojure.

Clojure has a single mind (not really)

They saw Clojure and saw that it was a single platform, single implementation by one guy (Rich Hickey). A dream has come true. So they switched their mind again, obsoleting Common Lisp and replacing it by Clojure. Since Common Lisp is still a Lisp they had a problem. As long as Common Lisp exists they cannot use Clojure as the "One and Only Future Lisp". So they started pestering all the world about Clojure being a replacement for Common Lisp. They didn't see the second trap. Clojure may be driven single-mindedly and there is a single mind which controls much of its drive, but regardless of all that things: It's actually quite multi.

They'll move on

They will not touch a line of Clojure like they didn't with Common Lisp (or OCaml, Haskell and many other living languages too). They will be wearisome and pesky, but don't despair dear Clojurians - they will move on and Clojure will thrive to be a multi-language. Perhaps this kind of people are the same ones, who think there is only one god to believe in – without realizing that the world would be a better one without religion.

Tuesday, December 1, 2009

A Lisp for the JVM

In the current issue of Toolbox my article »Lisp für die JVM« (Lisp for the JVM) was published. It's about ABCL, the not too old JVM-based Common Lisp. Since I wrote the article, a lot of things happened with ABCL. Now, one can read about first successful experiments to get this Lisp on Googles AppEngine. Also, the core of the system - the compiler - was continuously improved. ABCL is developing from an experimental prototype to a full CL alternative for java based environments.Thats incredible news for me; not only as an author, but also as a software developer.

And Clojure?

If one currently speaks about a Lisp for the JVM, everybody seems to think of Clojure. Ich already have Clojure on my screen and will very likely publish an article about it. Its just that ABCL was the bigger news to me. A practical and full implementation of Common Lisp on the JVM is a great achievement!

I dont' share the view, that a modern new dialect of lisp has to use an existing runtime like the JVM. I'm not even certain, if Rich Hickey actually meant this point as a fortune against Common Lisp. For me, his statement makes only sense if applied to languages like Python or Ruby: Many such scripting languages appeared in the last years; most of them came with their own runtimes. For such languages, defined by ad hoc reference implementations, it probably doesn't make much sense any more to write your own runtime.

JVM is not a USP

Common Lisp got specified independent from a concrete platform. The ANSI standard doesn't try to make the platform fix. So why not develop a Common Lisp for the JVM or the CLR? ABCL proved that it is possible and that the standard works for this task. If I'll write an article about Clojure, a fixation on the JVM or CLR will not be important. It would be more likely counted negatively. Clojure is currently in its design phase to emancipate from its VM. Yes - it was practical and useful, but the JVM is not a USP for Clojure. In an article about Clojure I would like to check what there actually is about all the hype and what actually is innovative about it.

Friday, January 25, 2008

New Worlds And Lispy Solutions

I really think that Gary brought up the discussion about Lisp library packaging at the right time. My logs definitely seem to hint at the existence of a formidable interest about this topic. I've taken a look at Matthew Kennedy's Lispy project, which strives to be a next generation package manager for common lisp and Gary brought up another topic, which I've thought about some years ago: an abstraction of modules above single systems (I called it "worlds").


DISCLAIMER: I've just looked very briefly at the code of Lispy!

Matthew Kennedy's Lispy aims to be simple and easy to use and is based on concepts which are proven to work for Linux distributions. Lispy is not really centrally organized, but it doesn't seem to emphasize distributed facilities very much. It seems to be quite easy though to setup a Lispy repository - all you need is some web space and a mapping file which stores metadata about all provided packages. As far as I understood it, versions are handled as chronological markers and there is a distinction between versions which come from package authors and versions which are managed by the Lispy system. There are MD5 sums of the asdf-system tarballs to enable efficient ways for verification of integrity and authentication. Matthew's design is based on his experience from managing Lisp packages for the Gentoo portage project and it actually is designed as a system to manage package distributions. In regard to my find/acquire/verify/authenticate quadruple he decided not to conflate loading of packages and installation of packages into one operation (acquire); this is maybe indeed the better thing to do. I didn't understand really who is responsible for the creation of the mapping-files; particularly where the :depends-on information is coming from. My opinion about Lispy is, that it looks very well designed. On the first sight it seems to be more suited for package maintainers than for developers. That may sound a bit weird, what I actually mean is that it seems to be more suited to manage released versions of a package. What should I do if I need a particular (perhaps locally patched) version of a package for only a particular project? If I understand Lispy right, then I would define a repository (mapping-file) for each such project. It also doesn't seem to support fetching packages out of scm repositories (cvs, darcs, svn...) and defines asdf-packages as tarballs. I still think the MD5/SHA digest should be computed from a defined order of the files of a source package and not from a tarball. Perhaps there could be an ASDF operation which computes the digest instead of compiling the files - so the digest would be defined only by the files relevant to the ASDF-System.

Worlds Apart

To me a world in the common lisp system management context is a logical bundle of systems used in a particular project or setting. It defines the context needed for development and deployment. A system-version could be uniquely identified by its system-name and a digest from the defined order of the content files. This would also mean, that world is actually defined by the a set of particular versions of asdf-packages. If one changes one of the packages it is not the same world anymore. This is similar to the concept of identity in git (see e. g. Nikodemus Siivola's explanation) Gary's command-set for working with worlds (he called them "environments") looks a bit to procedural to me. I would prefer a more declarative syntax. So instead of writing CREATE-WORLD and CLONE-WORLD as a user I would rather use a DEFINE-WORLD macro which defines the dependencies of a world. A world should know anything needed to upgrade the systems if needed. So if a system in my world is defined to track the changes of a particular branch in a scm system - so be it. I think worlds could be quite similar to the shelfs of cl-librarian. The main difference between "worlds" and "mapping repositories" (like in lispy) is that the first is a set of asdf-systems from the standpoint of a asdf-package user and the latter is from the standpoint of a package-publisher. The only reason to publish a world definition is actually to readily deployable (batteries included) projects. So the packages within a world should in some way depend on each other while the packages within a mapping repository (a "distribution") may have the only common point that they are maintained by the same person.

Tuesday, January 22, 2008

ASDF Load Syntax Sucks?

An often raised complaint against ASDF is the quite unfriendly syntax of one of its most basic operations: Loading a system.

(asdf:operate 'asdf:load-op :my-system)

I think this can not really be an issue, can it? How often does one actually have to type this thing? Systems are typically loaded as dependents of other systems. So I wouldn't actually use ASDF:OPERATE! I just type the system-name in the :depends-on clause of my system definition.

Interactive Development!

The sole exception to this is interactive development. It is quite common to load a system to try something out without officially adding it as an dependency to your system. Perhaps you do not even have a system for your code yet. The point here is: Loading a system while doing interactive development does not really need to be by directly using the programmatic interface of the ASDF-library; I'm using an editor command within the LispWorks IDE to interactively load a system. This even provides me with name-completion for asdf-systems. Writing something similar for slime should be trivial (if it doesn't exist yet).

Why Not Colon Commands?

A recent discussion on comp.lang.lisp about colon commands on the lisp REPL showed that most common lisp implementations provide such a REPL command feature; why not just define an :asdf, :load-asdf or :la colon-command?

Monday, January 21, 2008

Ideas for Asdf and Asdf-install

Gary King, a fellow lisper and diligent blogger started a discussion about making asdf-install better. First: I do not use asdf-install either. Before I tell you why I don't use it let me try to think about the points Gary raised.

1. It doesn't play well with Windows

Gary's first argument is that asdf-install doesn't work well with Windows, because of it's use of Unix utilities for tasks like archiving or signing/verifying. I agree that this is an issue, but nothing that couldn't be solved by just using ports of those utilities for windows.

2. It doesn't play well with itself

The second argument just describe some obvious short-comings of asdf-install. It doesn't handle updating your libraries in any way and it can only download and install tarballs. Gary is right here too; asdf-install is just not enough.

What do I think?

Gary is a smart guy and definitely is on the right track with his idea. I do not agree is his idea to make asdf-install better, but only because I think that asdf-install isn't a good base for anything. I think managing asdf-packages is too important to be left outside of lisp. I think there are enough high-quality libraries to handle the necessary tasks in common lisp and without airy callouts to shell utilities.

Well... but asdf-install is written in lisp!

Technically yes, but if you look into other general purpose lisp-libraries, you will see that a good lisp library should be flexible enough to adapt to many different use cases. A good lisp package manager needs to be extensible in at least four directions:

  1. How to find packages (cliki, cl-user.net...)
  2. How to acquire packages (e. g. tarball, zip, CVS, Darcs, SVN, git, e-mail, local folder)
  3. How to verify integrity of packages
  4. How to authenticate the identity of the supplier of a package

Finding a package should be a decentralized task. It should be possible to easily setup your own package registry. There could be different technologies used, but I think there should be at least a web-based one which doesn't demand more than some static web space. An interesting point here is, that finding a package might not necessarily be a proactive step. Maybe one could register to some typical package feeds, and as soon as a new version is announced, it will be acquired by the system. If the protocols of this component are done right, than there is no perceivable difference in finding a package somewhere in the internet or already installed on your harddrive. There should be no need for symlinks or *central-registry* hacking. The installed packages happen just to be cached on your harddrive.

There is nothing special behind the technologies to acquire the packages. One important thing is the change of notion for an asdf-package. Up to now it was actually a tarball with a certain structure. This does not hold anymore if we want to acquire packages through other channels like version control systems. Maybe an asdf-package should defined as the decompressed folder. The handling of a compressed asdf-package seems obvious to me, regardless of it being archived using tar/gz, zip or whatever.

The most obvious way to verify the integrity of an asdf-package seems to be a digest algorithm like MD5 or SHA. But if we define asdf-packages to be just folders and not tarballs anymore, we have the problem, that there is no obvious way how to compute the digest (because there is no defined order of its components). One idea may be to include a manifest-file with each asdf-package and use the order defined by this manifest to compute the digest; there may certainly be different ideas. A manifest-file could be useful for other things too though. It should be a very easy one step action to bless a package as released and create the necessary stuff like the digest and perhaps the manifest.

I've not thought very much about authentication. There are some evident solutions possible; one of it actually used already by asdf-install. The crux is to find a way which will actually be used, instead of just switched off by most users.

The package manager should understand different versions of a package. In the past the version-info was always encoded in the system-definition. I'm not sure if this is the right place for it. Regardless of this; the package manager has to handle packages with and without versions. The version-number is probably a good idea to order the different package-versions by age. But time is only one dimension in which different versions of a package could be placed. I think there should be something like a unique identifier for each version of a package and the best candidate for this would be the digest for the concrete asdf-package.

The package manager needs to handle dependencies between asdf-packages in a better way than just trying to load it and acquire the package on fail (like it is done in asdf-install). Actually this approach seems caused by a misfeature of asdf itself; there doesn't seem to be a way to ask an asdf-system for its dependencies.

My wishlist in very short: A new package-manager for asdf should:

  1. be written in common lisp
  2. be able to use different package sources
  3. be written in common lisp
  4. not mandate airy-fary use of OS-dependent stuff like symlinks
  5. be written in common lisp
  6. neither makes downloading nor finding packages a centralized task
  7. be written in common lisp
  8. have well-defined CLOS protocols for future extension
  9. and of course: It should be written in plain common lisp!

Thanks for your time! I really want to hear what you think about this ideas.