Friday, January 25, 2008

New Worlds And Lispy Solutions

I really think that Gary brought up the discussion about Lisp library packaging at the right time. My logs definitely seem to hint at the existence of a formidable interest about this topic. I've taken a look at Matthew Kennedy's Lispy project, which strives to be a next generation package manager for common lisp and Gary brought up another topic, which I've thought about some years ago: an abstraction of modules above single systems (I called it "worlds").


DISCLAIMER: I've just looked very briefly at the code of Lispy!

Matthew Kennedy's Lispy aims to be simple and easy to use and is based on concepts which are proven to work for Linux distributions. Lispy is not really centrally organized, but it doesn't seem to emphasize distributed facilities very much. It seems to be quite easy though to setup a Lispy repository - all you need is some web space and a mapping file which stores metadata about all provided packages. As far as I understood it, versions are handled as chronological markers and there is a distinction between versions which come from package authors and versions which are managed by the Lispy system. There are MD5 sums of the asdf-system tarballs to enable efficient ways for verification of integrity and authentication. Matthew's design is based on his experience from managing Lisp packages for the Gentoo portage project and it actually is designed as a system to manage package distributions. In regard to my find/acquire/verify/authenticate quadruple he decided not to conflate loading of packages and installation of packages into one operation (acquire); this is maybe indeed the better thing to do. I didn't understand really who is responsible for the creation of the mapping-files; particularly where the :depends-on information is coming from. My opinion about Lispy is, that it looks very well designed. On the first sight it seems to be more suited for package maintainers than for developers. That may sound a bit weird, what I actually mean is that it seems to be more suited to manage released versions of a package. What should I do if I need a particular (perhaps locally patched) version of a package for only a particular project? If I understand Lispy right, then I would define a repository (mapping-file) for each such project. It also doesn't seem to support fetching packages out of scm repositories (cvs, darcs, svn...) and defines asdf-packages as tarballs. I still think the MD5/SHA digest should be computed from a defined order of the files of a source package and not from a tarball. Perhaps there could be an ASDF operation which computes the digest instead of compiling the files - so the digest would be defined only by the files relevant to the ASDF-System.

Worlds Apart

To me a world in the common lisp system management context is a logical bundle of systems used in a particular project or setting. It defines the context needed for development and deployment. A system-version could be uniquely identified by its system-name and a digest from the defined order of the content files. This would also mean, that world is actually defined by the a set of particular versions of asdf-packages. If one changes one of the packages it is not the same world anymore. This is similar to the concept of identity in git (see e. g. Nikodemus Siivola's explanation) Gary's command-set for working with worlds (he called them "environments") looks a bit to procedural to me. I would prefer a more declarative syntax. So instead of writing CREATE-WORLD and CLONE-WORLD as a user I would rather use a DEFINE-WORLD macro which defines the dependencies of a world. A world should know anything needed to upgrade the systems if needed. So if a system in my world is defined to track the changes of a particular branch in a scm system - so be it. I think worlds could be quite similar to the shelfs of cl-librarian. The main difference between "worlds" and "mapping repositories" (like in lispy) is that the first is a set of asdf-systems from the standpoint of a asdf-package user and the latter is from the standpoint of a package-publisher. The only reason to publish a world definition is actually to readily deployable (batteries included) projects. So the packages within a world should in some way depend on each other while the packages within a mapping repository (a "distribution") may have the only common point that they are maintained by the same person.

Tuesday, January 22, 2008

ASDF Load Syntax Sucks?

An often raised complaint against ASDF is the quite unfriendly syntax of one of its most basic operations: Loading a system.

(asdf:operate 'asdf:load-op :my-system)

I think this can not really be an issue, can it? How often does one actually have to type this thing? Systems are typically loaded as dependents of other systems. So I wouldn't actually use ASDF:OPERATE! I just type the system-name in the :depends-on clause of my system definition.

Interactive Development!

The sole exception to this is interactive development. It is quite common to load a system to try something out without officially adding it as an dependency to your system. Perhaps you do not even have a system for your code yet. The point here is: Loading a system while doing interactive development does not really need to be by directly using the programmatic interface of the ASDF-library; I'm using an editor command within the LispWorks IDE to interactively load a system. This even provides me with name-completion for asdf-systems. Writing something similar for slime should be trivial (if it doesn't exist yet).

Why Not Colon Commands?

A recent discussion on comp.lang.lisp about colon commands on the lisp REPL showed that most common lisp implementations provide such a REPL command feature; why not just define an :asdf, :load-asdf or :la colon-command?

Monday, January 21, 2008

Ideas for Asdf and Asdf-install

Gary King, a fellow lisper and diligent blogger started a discussion about making asdf-install better. First: I do not use asdf-install either. Before I tell you why I don't use it let me try to think about the points Gary raised.

1. It doesn't play well with Windows

Gary's first argument is that asdf-install doesn't work well with Windows, because of it's use of Unix utilities for tasks like archiving or signing/verifying. I agree that this is an issue, but nothing that couldn't be solved by just using ports of those utilities for windows.

2. It doesn't play well with itself

The second argument just describe some obvious short-comings of asdf-install. It doesn't handle updating your libraries in any way and it can only download and install tarballs. Gary is right here too; asdf-install is just not enough.

What do I think?

Gary is a smart guy and definitely is on the right track with his idea. I do not agree is his idea to make asdf-install better, but only because I think that asdf-install isn't a good base for anything. I think managing asdf-packages is too important to be left outside of lisp. I think there are enough high-quality libraries to handle the necessary tasks in common lisp and without airy callouts to shell utilities.

Well... but asdf-install is written in lisp!

Technically yes, but if you look into other general purpose lisp-libraries, you will see that a good lisp library should be flexible enough to adapt to many different use cases. A good lisp package manager needs to be extensible in at least four directions:

  1. How to find packages (cliki,
  2. How to acquire packages (e. g. tarball, zip, CVS, Darcs, SVN, git, e-mail, local folder)
  3. How to verify integrity of packages
  4. How to authenticate the identity of the supplier of a package

Finding a package should be a decentralized task. It should be possible to easily setup your own package registry. There could be different technologies used, but I think there should be at least a web-based one which doesn't demand more than some static web space. An interesting point here is, that finding a package might not necessarily be a proactive step. Maybe one could register to some typical package feeds, and as soon as a new version is announced, it will be acquired by the system. If the protocols of this component are done right, than there is no perceivable difference in finding a package somewhere in the internet or already installed on your harddrive. There should be no need for symlinks or *central-registry* hacking. The installed packages happen just to be cached on your harddrive.

There is nothing special behind the technologies to acquire the packages. One important thing is the change of notion for an asdf-package. Up to now it was actually a tarball with a certain structure. This does not hold anymore if we want to acquire packages through other channels like version control systems. Maybe an asdf-package should defined as the decompressed folder. The handling of a compressed asdf-package seems obvious to me, regardless of it being archived using tar/gz, zip or whatever.

The most obvious way to verify the integrity of an asdf-package seems to be a digest algorithm like MD5 or SHA. But if we define asdf-packages to be just folders and not tarballs anymore, we have the problem, that there is no obvious way how to compute the digest (because there is no defined order of its components). One idea may be to include a manifest-file with each asdf-package and use the order defined by this manifest to compute the digest; there may certainly be different ideas. A manifest-file could be useful for other things too though. It should be a very easy one step action to bless a package as released and create the necessary stuff like the digest and perhaps the manifest.

I've not thought very much about authentication. There are some evident solutions possible; one of it actually used already by asdf-install. The crux is to find a way which will actually be used, instead of just switched off by most users.

The package manager should understand different versions of a package. In the past the version-info was always encoded in the system-definition. I'm not sure if this is the right place for it. Regardless of this; the package manager has to handle packages with and without versions. The version-number is probably a good idea to order the different package-versions by age. But time is only one dimension in which different versions of a package could be placed. I think there should be something like a unique identifier for each version of a package and the best candidate for this would be the digest for the concrete asdf-package.

The package manager needs to handle dependencies between asdf-packages in a better way than just trying to load it and acquire the package on fail (like it is done in asdf-install). Actually this approach seems caused by a misfeature of asdf itself; there doesn't seem to be a way to ask an asdf-system for its dependencies.

My wishlist in very short: A new package-manager for asdf should:

  1. be written in common lisp
  2. be able to use different package sources
  3. be written in common lisp
  4. not mandate airy-fary use of OS-dependent stuff like symlinks
  5. be written in common lisp
  6. neither makes downloading nor finding packages a centralized task
  7. be written in common lisp
  8. have well-defined CLOS protocols for future extension
  9. and of course: It should be written in plain common lisp!

Thanks for your time! I really want to hear what you think about this ideas.