CODEartist (en)

CLOS Class Naming

noreply@blogger.com (Jochen H. Schmidt) — Tue, 15 May 2012 08:59:00 +0000

The classes of the Common Lisp spec, the MOP and big CLOS projects like CLIM use some naming conventions which may not make sense at first. I'm speaking of the STANDARD- and BASIC- prefixes. What is the idea behind them?

Protocol Oriented Programming

In CLOS one defines "protocols" using classes and generic functions. Remember: Methods do belong to generic functions and not to classes. There are no dedicated means to define protocols - it is idiomatic. A common practice is to define "protocol classes" which often do not have any slots at all. They serve as superclasses to all classes which implement the protocol.

Basics and Standards

In the set of protocol implementation classes there may be one, which contains a minimal complete implementation and/or there might be one which is something like the default implementation. Assuming a protocol for FOOBAR defined through the protocol-class FOOBAR the minimal implementation could be called BASIC-FOOBAR and the default one STANDARD-FOOBAR. The BASIC-** one if often called FUNDAMENTAL-** too.

Extensibility

It is often a good idea to implement STANDARD-** by inheriting the BASIC-** implementation, overriding the necessary parts. A protocol user who wants to implement his own implementation can decide on which level he wants to get in: He could inherit from STANDARD-** if his implementations is only a small variant. He may inherit from BASIC-** if the implementation is more complicated and perhaps incompatible to what STANDARD-** does. If even BASIC-** is to strict, he could try to implement the protocol class completely on its own - but this is then more like implementing interfaces in classical object oriented languages, where you cannot inherit behaviour at all. You have to provide methods for any necessary generic function the protocol is made of. This is the most work and implementing a protocol is often more than just following the interfaces.

IOLib Considered Harmful to Lisp Today

noreply@blogger.com (Jochen H. Schmidt) — Sat, 26 Nov 2011 13:48:00 +0000

...and perhaps a boon tomorrow.

Common Lisp is a very nice language to develop all kinds of servers and clients; even though there is no real standard network API. Over the years many different wrapper and portability libraries (CLOCC, ACL-COMPAT, TRIVIAL-SOCKETS, USOCKET) appeared. Often, this libraries were made by the principle of the least common denominator. For simple applications this is good enough, but complex applications have special needs.

Typical Example: Comet, event based HTTP

Webservers like NGINX are based on a event based architecture and do not consume threads by the number of concurrent connections. That way, they can manage many concurrent connections while needing much less resources than traditional worker thread based web servers. Linux, BSD (OS X too) offer means like epoll and kqueue, to implement such event based I/O multiplexers in a very efficient manner. Reading and writing needs to be non-blocking too. All this facilities are much beyond the reach of LCD libraries that just try to clean up the API differences between the different Common Lisp implementations.

The FFI to the Rescue

This were certainly part of the reasons which played a part at the conception of IOLib (http://common-lisp.net/project/iolib/). IOLib is well documented and it is integrated in Quicklisp, so it should be really easy to deploy. As always when the rescue is sought in something like the FFI I have some doubts, that the result is actually accessible and usable. Often makes many things much more complicated and portability not really easier.

Whats the Problem?

The idea is seductive: Instead of dealing with numerous incompatible APIs of CL implementations, one only needs to deal with some OSes - which mainly means "Linux" and perhaps "BSD". With much luck it will work on OS X too, but Windows is seldom a popular target for this projects. There is nothing wrong with that - if there is demand for Windows, someone could do the necessary work and if not - why bother? Well - including Windows into the mix makes such a library just LCD again, because the concepts are different enough that an API would have to be designed to be good and not just taken from one of the OSes. The solution to that is simple: Let the libraries just be OS dependent and build more abstract layers on top of them.

How is that harmful to Lisp?

Just to make one point clear: I think IOLib is a very good idea and I hope this project will be a great success. But up to now there are many problems with it, that make it a worse solution to simpler problems than USOCKET is: It reduces your possibilities to some OpenSource OSes (Linux, FreeBSD) and some OpenSource Lispsystems (CMUCL, SBCL, CLISP, ClozureCL). I've tried a whole hacking weekend to get IOLib working on LispWorks on Mac OS X - no positive results yet. Even the ClozureCL didn't work and already broke down in the CFFI-Grovel phase (I got over that using LispWorks after hammering on my system but it still doesn't work on ClozureCL). Up to now - getting IOLib running seems the be much of a PITA. So actually HOW is that harmful to Lisp? Well - I begin to see a trend that more and more CL projects begin to switch to IOLib now, which makes them only usable by parts of the community. Think about when Hunchentoot would decide to switch to IOLib tomorrow: Well - my ClozureCL based linux servers would probably have no problem with that - my LispWorks ones will stop working. My development platform is OSX - a switch to IOLib would rule that out too.

So what to do to stop the harm?

Think twice... better thrice before switching your project to IOLib
If you can't resist: Make a working fallback to something with broader support
If you got some time: Help hacking IOLib to work flawlessly on OS X, Windows, LispWorks and AllegroCL a.s.o.

I'm very willing to help on OS X, ClozureCL and Lispworks. Here's what I found so far:

CFFI-Grovel really doesn't like Spaces in paths (like /Volumes/Macintosh HD/Users...)
LispWorks MAKE-ARRAY has an option :allocation :static - which is needed when doing FFI pointers to arrays. CFFI knows that IOLib not

I get IOLib compiled on LispWorks for Mac OS X, but at least the DNS functions read arbitrary chunk into the buffers.

Happy Hacking

On Immutable Minds

noreply@blogger.com (Jochen H. Schmidt) — Sun, 20 Dec 2009 17:42:00 +0000

I've wanted to write about Clojure for a while. It has not happened yet because I just didn't find the time for it. Its still not time for it, but today I thought about a phenomenon I observed since Clojures new popularity more than before. There is a particular kind of people who seemingly have problems to live with multiple options. I've called them the "Immutable Minds" because they are incapable of changing their mind without completely replacing all things who were there before. I called it also that way, because I think immutable is such a nice tagword this times ;-). This isn't anything special with Clojure (there is nothing wrong with it)

The Immutable Mind is single-minded

When Common Lisp was fresh (actually refreshed) the Immutable Minds found it and replaced whatever they had in their minds before. But what they found seemed like a trap: It was the obvious "single option" for them, but Common Lisp itself was full of things "multi": It had multi-methods, multi-inheritance, multiple namespaces. It even had multiple implementations (yes that really got critized!) and even worse: Multiple opinions. They couldn't use it but it had replaced what was there before (which was already declared "dead" and they've lost the reference), so they did the only thing they still could do: Pestering people outside the community about Lisp being "The One And Only (tm)" while annoying the community members by constant requests on how to change lisp to be less multi but more single. Single namespace, single inheritance, single implementation and the most important: Single opinion (theirs). When they didn't get what they wanted they got angry and filled with bitterness. Then came Clojure.

Clojure has a single mind (not really)

They saw Clojure and saw that it was a single platform, single implementation by one guy (Rich Hickey). A dream has come true. So they switched their mind again, obsoleting Common Lisp and replacing it by Clojure. Since Common Lisp is still a Lisp they had a problem. As long as Common Lisp exists they cannot use Clojure as the "One and Only Future Lisp". So they started pestering all the world about Clojure being a replacement for Common Lisp. They didn't see the second trap. Clojure may be driven single-mindedly and there is a single mind which controls much of its drive, but regardless of all that things: It's actually quite multi.

They'll move on

They will not touch a line of Clojure like they didn't with Common Lisp (or OCaml, Haskell and many other living languages too). They will be wearisome and pesky, but don't despair dear Clojurians - they will move on and Clojure will thrive to be a multi-language. Perhaps this kind of people are the same ones, who think there is only one god to believe in – without realizing that the world would be a better one without religion.

A Lisp for the JVM

noreply@blogger.com (Jochen H. Schmidt) — Mon, 30 Nov 2009 23:14:00 +0000

In the current issue of Toolbox my article »Lisp für die JVM« (Lisp for the JVM) was published. It's about ABCL, the not too old JVM-based Common Lisp. Since I wrote the article, a lot of things happened with ABCL. Now, one can read about first successful experiments to get this Lisp on Googles AppEngine. Also, the core of the system - the compiler - was continuously improved. ABCL is developing from an experimental prototype to a full CL alternative for java based environments.Thats incredible news for me; not only as an author, but also as a software developer.

And Clojure?

If one currently speaks about a Lisp for the JVM, everybody seems to think of Clojure. Ich already have Clojure on my screen and will very likely publish an article about it. Its just that ABCL was the bigger news to me. A practical and full implementation of Common Lisp on the JVM is a great achievement!

I dont' share the view, that a modern new dialect of lisp has to use an existing runtime like the JVM. I'm not even certain, if Rich Hickey actually meant this point as a fortune against Common Lisp. For me, his statement makes only sense if applied to languages like Python or Ruby: Many such scripting languages appeared in the last years; most of them came with their own runtimes. For such languages, defined by ad hoc reference implementations, it probably doesn't make much sense any more to write your own runtime.

JVM is not a USP

Common Lisp got specified independent from a concrete platform. The ANSI standard doesn't try to make the platform fix. So why not develop a Common Lisp for the JVM or the CLR? ABCL proved that it is possible and that the standard works for this task. If I'll write an article about Clojure, a fixation on the JVM or CLR will not be important. It would be more likely counted negatively. Clojure is currently in its design phase to emancipate from its VM. Yes - it was practical and useful, but the JVM is not a USP for Clojure. In an article about Clojure I would like to check what there actually is about all the hype and what actually is innovative about it.

New Worlds And Lispy Solutions

noreply@blogger.com (Jochen H. Schmidt) — Fri, 25 Jan 2008 22:14:00 +0000

I really think that Gary brought up the discussion about Lisp library packaging at the right time. My logs definitely seem to hint at the existence of a formidable interest about this topic. I've taken a look at Matthew Kennedy's Lispy project, which strives to be a next generation package manager for common lisp and Gary brought up another topic, which I've thought about some years ago: an abstraction of modules above single systems (I called it "worlds").

Lispy

DISCLAIMER: I've just looked very briefly at the code of Lispy!

Matthew Kennedy's Lispy aims to be simple and easy to use and is based on concepts which are proven to work for Linux distributions. Lispy is not really centrally organized, but it doesn't seem to emphasize distributed facilities very much. It seems to be quite easy though to setup a Lispy repository - all you need is some web space and a mapping file which stores metadata about all provided packages. As far as I understood it, versions are handled as chronological markers and there is a distinction between versions which come from package authors and versions which are managed by the Lispy system. There are MD5 sums of the asdf-system tarballs to enable efficient ways for verification of integrity and authentication. Matthew's design is based on his experience from managing Lisp packages for the Gentoo portage project and it actually is designed as a system to manage package distributions. In regard to my find/acquire/verify/authenticate quadruple he decided not to conflate loading of packages and installation of packages into one operation (acquire); this is maybe indeed the better thing to do. I didn't understand really who is responsible for the creation of the mapping-files; particularly where the :depends-on information is coming from. My opinion about Lispy is, that it looks very well designed. On the first sight it seems to be more suited for package maintainers than for developers. That may sound a bit weird, what I actually mean is that it seems to be more suited to manage released versions of a package. What should I do if I need a particular (perhaps locally patched) version of a package for only a particular project? If I understand Lispy right, then I would define a repository (mapping-file) for each such project. It also doesn't seem to support fetching packages out of scm repositories (cvs, darcs, svn...) and defines asdf-packages as tarballs. I still think the MD5/SHA digest should be computed from a defined order of the files of a source package and not from a tarball. Perhaps there could be an ASDF operation which computes the digest instead of compiling the files - so the digest would be defined only by the files relevant to the ASDF-System.

Worlds Apart

To me a world in the common lisp system management context is a logical bundle of systems used in a particular project or setting. It defines the context needed for development and deployment. A system-version could be uniquely identified by its system-name and a digest from the defined order of the content files. This would also mean, that world is actually defined by the a set of particular versions of asdf-packages. If one changes one of the packages it is not the same world anymore. This is similar to the concept of identity in git (see e. g. Nikodemus Siivola's explanation) Gary's command-set for working with worlds (he called them "environments") looks a bit to procedural to me. I would prefer a more declarative syntax. So instead of writing CREATE-WORLD and CLONE-WORLD as a user I would rather use a DEFINE-WORLD macro which defines the dependencies of a world. A world should know anything needed to upgrade the systems if needed. So if a system in my world is defined to track the changes of a particular branch in a scm system - so be it. I think worlds could be quite similar to the shelfs of cl-librarian. The main difference between "worlds" and "mapping repositories" (like in lispy) is that the first is a set of asdf-systems from the standpoint of a asdf-package user and the latter is from the standpoint of a package-publisher. The only reason to publish a world definition is actually to readily deployable (batteries included) projects. So the packages within a world should in some way depend on each other while the packages within a mapping repository (a "distribution") may have the only common point that they are maintained by the same person.

ASDF Load Syntax Sucks?

noreply@blogger.com (Jochen H. Schmidt) — Tue, 22 Jan 2008 15:34:00 +0000

An often raised complaint against ASDF is the quite unfriendly syntax of one of its most basic operations: Loading a system.

(asdf:operate 'asdf:load-op :my-system)

I think this can not really be an issue, can it? How often does one actually have to type this thing? Systems are typically loaded as dependents of other systems. So I wouldn't actually use ASDF:OPERATE! I just type the system-name in the :depends-on clause of my system definition.

Interactive Development!

The sole exception to this is interactive development. It is quite common to load a system to try something out without officially adding it as an dependency to your system. Perhaps you do not even have a system for your code yet. The point here is: Loading a system while doing interactive development does not really need to be by directly using the programmatic interface of the ASDF-library; I'm using an editor command within the LispWorks IDE to interactively load a system. This even provides me with name-completion for asdf-systems. Writing something similar for slime should be trivial (if it doesn't exist yet).

Why Not Colon Commands?

A recent discussion on comp.lang.lisp about colon commands on the lisp REPL showed that most common lisp implementations provide such a REPL command feature; why not just define an :asdf, :load-asdf or :la colon-command?

Ideas for Asdf and Asdf-install

noreply@blogger.com (Jochen H. Schmidt) — Mon, 21 Jan 2008 16:33:00 +0000

Gary King, a fellow lisper and diligent blogger started a discussion about making asdf-install better. First: I do not use asdf-install either. Before I tell you why I don't use it let me try to think about the points Gary raised.

1. It doesn't play well with Windows

Gary's first argument is that asdf-install doesn't work well with Windows, because of it's use of Unix utilities for tasks like archiving or signing/verifying. I agree that this is an issue, but nothing that couldn't be solved by just using ports of those utilities for windows.

2. It doesn't play well with itself

The second argument just describe some obvious short-comings of asdf-install. It doesn't handle updating your libraries in any way and it can only download and install tarballs. Gary is right here too; asdf-install is just not enough.

What do I think?

Gary is a smart guy and definitely is on the right track with his idea. I do not agree is his idea to make asdf-install better, but only because I think that asdf-install isn't a good base for anything. I think managing asdf-packages is too important to be left outside of lisp. I think there are enough high-quality libraries to handle the necessary tasks in common lisp and without airy callouts to shell utilities.

Well... but asdf-install is written in lisp!

Technically yes, but if you look into other general purpose lisp-libraries, you will see that a good lisp library should be flexible enough to adapt to many different use cases. A good lisp package manager needs to be extensible in at least four directions:

How to find packages (cliki, cl-user.net...)
How to acquire packages (e. g. tarball, zip, CVS, Darcs, SVN, git, e-mail, local folder)
How to verify integrity of packages
How to authenticate the identity of the supplier of a package

Finding a package should be a decentralized task. It should be possible to easily setup your own package registry. There could be different technologies used, but I think there should be at least a web-based one which doesn't demand more than some static web space. An interesting point here is, that finding a package might not necessarily be a proactive step. Maybe one could register to some typical package feeds, and as soon as a new version is announced, it will be acquired by the system. If the protocols of this component are done right, than there is no perceivable difference in finding a package somewhere in the internet or already installed on your harddrive. There should be no need for symlinks or *central-registry* hacking. The installed packages happen just to be cached on your harddrive.

There is nothing special behind the technologies to acquire the packages. One important thing is the change of notion for an asdf-package. Up to now it was actually a tarball with a certain structure. This does not hold anymore if we want to acquire packages through other channels like version control systems. Maybe an asdf-package should defined as the decompressed folder. The handling of a compressed asdf-package seems obvious to me, regardless of it being archived using tar/gz, zip or whatever.

The most obvious way to verify the integrity of an asdf-package seems to be a digest algorithm like MD5 or SHA. But if we define asdf-packages to be just folders and not tarballs anymore, we have the problem, that there is no obvious way how to compute the digest (because there is no defined order of its components). One idea may be to include a manifest-file with each asdf-package and use the order defined by this manifest to compute the digest; there may certainly be different ideas. A manifest-file could be useful for other things too though. It should be a very easy one step action to bless a package as released and create the necessary stuff like the digest and perhaps the manifest.

I've not thought very much about authentication. There are some evident solutions possible; one of it actually used already by asdf-install. The crux is to find a way which will actually be used, instead of just switched off by most users.

The package manager should understand different versions of a package. In the past the version-info was always encoded in the system-definition. I'm not sure if this is the right place for it. Regardless of this; the package manager has to handle packages with and without versions. The version-number is probably a good idea to order the different package-versions by age. But time is only one dimension in which different versions of a package could be placed. I think there should be something like a unique identifier for each version of a package and the best candidate for this would be the digest for the concrete asdf-package.

The package manager needs to handle dependencies between asdf-packages in a better way than just trying to load it and acquire the package on fail (like it is done in asdf-install). Actually this approach seems caused by a misfeature of asdf itself; there doesn't seem to be a way to ask an asdf-system for its dependencies.

My wishlist in very short: A new package-manager for asdf should:

be written in common lisp
be able to use different package sources
be written in common lisp
not mandate airy-fary use of OS-dependent stuff like symlinks
be written in common lisp
neither makes downloading nor finding packages a centralized task
be written in common lisp
have well-defined CLOS protocols for future extension
and of course: It should be written in plain common lisp!

Thanks for your time! I really want to hear what you think about this ideas.

The Darkside is Here

noreply@blogger.com (Jochen H. Schmidt) — Thu, 06 Dec 2007 20:59:00 +0000

For most of my projects I'm using the distributed revision control system darcs. I particularly like how easy and streamlined it is. Creating repositories, branches or pushing single patches around is just a matter of a few minutes in learning time. Another nice feature is, that you can commit only parts of the changes in a single file; darcs will just ask you about each change. Choosing the changes one wants to record on the command line can be a tedious process though. This is why I wrote us an internal tool called "Darkside" which allows us to easily pick all changes using a straightforward dialog. The following screenshot shows the "Patch Recorder" tool of Darkside. An editor pane below the change-tree shows which lines were removed (red) and which were added (green). Darkside is written in LispWorks using CAPI.

The next screenshot shows the second tab-pane of the interface. Here one can write the log-message for the current record. I extended darcs' notion of a patch name a little bit by providing a combo-box with certain tags like "Feature", "Change", "Bugfix" and so on. The selected tag gets enclosed in brackets and then prepended before the patch-name. This is actually just a convention we developed using darcs. The list-pane below the log-editor shows other patches which have been recorded before. It is possible to double-click those lines to paste the log-text of this patch into the log-editor.

CL-Darcs: A Common Lisp Implementation

The original darcs (which is used in Darkside too) is written in Haskell, a statically typed functional programming language. The installation is simple: just copy the executable somewhere into your path. Wouldn't it be nice to have an implementation of darcs in common lisp? Well, we have luck! Thanks to Magnus Henoch's cl-darcs there is already a common lisp library which aims to be compatible with the original darcs implementation. The code is nicely structured and good readable. One really missing thing for a cl-darcs to ensure compatibility to the original darcs is an extensive test-suite. But Magnus implements the command-line interface of darcs on top of the common lisp API so that one can use the official darcs test suite. I think cl-darcs is a very promising project.

Steve is Not an Acceptable Lisper

noreply@blogger.com (Jochen H. Schmidt) — Sun, 15 Apr 2007 21:10:00 +0000

Steve Yegge raised an interesting point in his blog. He mentions that some time ago Eric Kidd wrote an article called Why Ruby is an Acceptable Lisp and Steves article is meant as a response to that claiming that Lisp itself may not be an acceptable Lisp at all.

Steve said in his introduction:

I guarantee you there isn't one single Lisp programmer out there who uses exclusively Lisp. Instead we spend our time hacking around its inadequacies, often in other languages.

Well I've used Lisp almost exclusively in the last 6 years. I do not think it has to be this way - I have absolutely no problem using another programming language if it fits better to the demands of my tasks. I also learned many other languages in this six years; but for most problems that came up I did use Lisp and it never was a real problem.

Steve seems a bit obsessed about "length" in all kinds of shades so I will try to give an elaborate overview of the points he raised:

Problem 1: Which Lisp

In short this section talks about how difficult it is to choose a Lisp because there are so many implementations of it. I never quite understood what the problem was with this. I'm quite happy to have a multidimensional choice out of commercial/non-commercial, bytecompiled, native, Mac, Windows, Linux, Solaris, big, small, slow, fast, embeddable, whatever implementations of Lisp. According to Steve, choosing a Lisp seems difficult too, because Paul Graham planned to create his own Lisp (Arc) and this would mean something like "don't use Lisp!". I always thought choosing a programming language should be done by analysing ones needs and what is offered from the available options.

Steve mentioned another difficulty arising, when a hypothetical newbie chooses a Common Lisp implementation:

Eventually, if they stick with Lisp at all, they learn they can override most of these defaults in nonportable ways, which makes things infinitesimally more bearable.

Well - it seems Steve himself is not yet at the point were he realizes that exactly this adaptability is what makes Lisp the language it is. For me "CL" means not only "Common Lisp", but also "Compiler Language". I know no other language which makes it so easy to adapt it to that kind of language you want and need.

Problem 2: Worthless Spec

This section argues that the Common Lisp spec (ANS X3.226-1994) is ancient. Well it's twelve years from that time - maybe thats really a long time for a language spec. He claims that everytime someone proposes to update the standard someone else does not agree to that. That's true - there are many people who think redoing the official standards process which lead to ANSI Common Lisp might be to much work for the gain. What I do not understand with Steve's concerns about this topic is, why other languages like Ruby or Python are allowed to evolve without any official standard at all and Common Lisp is only allowed to change through a labour intensive ANSI process?

Some years ago someone suggested creating a COMMON-LISP-2006 package (I've updated the year 'cause I cannot remember when the proposal was) and extending the language the way one wants within this package. This technique is used by many Lispers to adapt CL to their needs and it could also get used to adapt "Common Lisp the Language". Even better would be an approach which would be modular; allowing CL2006 implementations to implement only parts of the CL2006 package tree. But I think such an approach is not really needed. As with any current programming language: Solutions grow out of actual problems. If there is no solution for a particular problem, someone will write a solution for it if he needs it. Interoperability and reuse are programming problems too and people will write solutions.It's not a new thing for Lisp - not even for any other Programming Language - that there are people who like more to talk, argue and discuss than to simply sit down and solve their problems. Fortunately the Lisp community has quite a lot talented programmers who have contributed in a constructive and helpful way to create useful solutions for Lisp-using people.

Problem 3: CLOS

Steve claims some things which are factually not true with CLOS. I recommend any Lisper to get a copy of the book "The Art of the Metaobject Protocol" - IMHO this is an essential lecture for any programmer who wants to understand object oriented programming; not only in Lisp. For the curious: you can read what Steve got wrong with CLOS (and other things) by reading the comments to his blog entry. I don't think its necessary to duplicate what already was said.

Problem 4: Macros

Steve builds up an analogy between CL Macros and duct tape. While being entertaining to read, I don't see were this story fits with the topic of Lisp being no acceptable Lisp. The points raised are old: "hygienic" vs "not-hygienic" macros, whining about the syntax - the only new and innovative one is Steve's complete misunderstanding that CLOS is made as just a set of macros.

Problem 4: Type System

(Yes the numbering is quoted right - Steve has chosen to use this for an analogy between type systems and incorrectly numbering headings) In this section Steve promises to talk more about type systems - but not now. There is nothing more than a loose mention that the CL type system may be flexible but not enough. After his claim that Haskell and OCaml are faster because of their type systems my intention to read up his next posting about type systems dropped below zero.

I think this whole exercise in trying to write a rant about something you do not even fully comprehend is entertaining but useless.