Setting up Lisp & Hunchentoot on a web server

For this post i decided to explain how i’ve setup Common Lisp Hunchentoot webserver running on Ubuntu 8.10 and how to set it up for remote interaction using Emacs and SLIME (Superior Lisp Interaction Mode for Emacs).

Obviously i couldn’t have done it all by myself, these are the documents i used as reference for this task, “Running Common Lisp behind Apache” and “Install Hunchentoot 1.0 on Ubuntu”.

For this post i will assume you are working in the remote box/server so i assume you are logged in by ssh or something.

We’re going to use SBCL but the Ubuntu package is not up-to-date, we’ll use clbuild to download and install the latest SBCL version.

Step 1 – Install clbuild requirements

You might have noticed we are installing sbcl package after i just said it’s too dated, well in order to compile sbcl one needs another lisp running and in this case it’s easier to use the ubuntu sbcl package. After we compile the latest sbcl this package can (and probably should) be removed.

Step 2 – Get clbuild

Now you have clbuild installed you can check if you have all the required dependencies by using:

Step 4 – Download latest SBCL / compile it

Step 5 – Downloading Hunchentoot and Cl-Who

We explicitly load cl-unicode and trivial-bactrace because when compiling hunchentoot it will throw some errors if these libraries are not present. cl-who is a pretty cool library for emitting xml/html but it is optional, here i opted for installing it as well. This will prompt you if you want to install all the dependencies, just go ahead and say yes.

After all this i think it’s best if we compile all these libraries.

This will compile/recompile all the projects installed so far.

Step 6 – Getting latest SLIME

Slime is composed of Slime itself, the Emacs code, and Swank the Common Lisp code that will accept connections from Slime.

Let’s just compile swank so that it doesn’t need to be compiled the first time it’s loaded.

Step 7 – creating the lisp user

We want to create a special user for running our lisp code, since this post is about setting up hunchentoot let’s create a user for that. This user will have it’s password disabled and will belong to the nogroup group.Let’s add the user:

Step 8 – System script for starting Hunchentoot

We want to create a system script to allow our Hunchentoot server to be automatically started when the system (re)starts. Let’s create the /etc/init.d/hunchentoot file script. I use emacs but you can use whatever editor you feel more comfortable with.

Edit: The following code has been updated to a real initialization script for Ubuntu, since i noticed that the previous version wasn’t restarting my hunchentoot when the system restarted. The old original version is still available for download here.

And inside let’s put the following code

To make life easier you can just download this script here: /etc/init.d/hunchentoot (move it to /etc/init.d/ and rename it to hunchentoot).

Remember to set the script to executable.

When starting, this script will simply delegate to the start-hunchentoot script in the hunchentoot user’s home dir (in this case /var/lib/hunchentoot), it will be ran as the hunchentoot user. When stopping, the script telnets to the local port where the lisp process will be listening for a connection to close the system (you can set this port to whatever value you want).

Step 9 – Launching the Lisp process

We need to get a Lisp process running in the background, for that we’ll use detachtty which is a lightweight tool quite like screen but with less functionalities and more suited here. We’ll just use the ubuntu package.

Next we’ll create the directory structure we will be use for detachtty logs and runtime information, we also set the correct permissions for these directories.

Now simply use your editor and create the file /var/lib/hunchentoot/start-hunchentoot with the following code:

You can download this script here: /var/lib/hunchentoot/start-hunchentoot
(just put it in your hunchentoot user home dir and rename it to start-hunchentoot).

Again, let’s setup the correct permissions for the script.

This will launch the lisp process and instruct it to load the init.lisp file which will contain the code to actually start Hunchentoot and Swank (Slime server).

Step 10 – Lisp initialization code (init.lisp)

Using your editor create the /var/lib/hunchentoot/init.lisp file with the code:

Edit: I had to comment out the call to stop the hunchentoot server because it was freezing lisp thus never actually ending the process (i believe this issue is solved in latest version of hunchentoot so you can try and leave that line uncommented).

If you want you can download init.lisp.txt (just rename it to init.lisp on your server).

Step 11 – Testing if it runs

So the time to actually test if this whole setup works is now, simply issue the command:

If you’re lucky then if you connect from your browser to http://_your_domain_:8000 you should see Hunchentoot’s default page. If for some reason you need to stop the server simply use:

Step 12 – Troubleshooting

If after doing all the steps here something is not working here are some of the methods i used to see if everything is correctly setup.

  1. Check if the detachtty is running: also check if the files hunchentoot.pid and hunchentoot.sock exist in the directory /var/lib/hunchentoot/run/. if either of these approaches fail then detachtty failed to start or has already exited.
  2. If it’s a lisp error you can find the error inside /var/lib/hunchentoot/hunchentoot.dribble.
  3. Check ownership and permissions on all the files and directories we talked about.

Step 13 – Remotely interacting with Lisp

Now that everything is setup we can remotely interact with our Lisp process by using emacs and slime in our local machine. For this to work we need to install emacs and slime in our local machine, we can do this by installing the ubuntu packages:

But the server part of slime, the swank server running on the remote machine, only listens on a local port so in order to remotely connect you must first create an ssh tunnel into the remote machine this is accomplished by something like this:

This will perform an ssh login as _user_ and bind your local port 4005 to port 4006 on the _remote_box_, all the communications from your local machine to the remote machine are behind ssh and thus more secure.

After this from emacs all you need to do is to invoke slime-connect and connect to Host: 127.0.0.1 Port: 4005. If everything goes ok you should now see inside an emacs buffer a lisp REPL like:

Now all you need to do is type lisp expressions that will be evaluated on the remote machine lisp process. i.e:

Conclusions

As i stated in the first post, the idea behind this blog is mostly of helping me not to forget how i’ve done somethings before. These were basically the steps i took to get me to my goal of running lisp web pages. I really hope this might end up being useful to others as well. Feel free to comment/critic on the quality of the post and/or the code.

12 thoughts on “Setting up Lisp & Hunchentoot on a web server”

  1. Thanks so much for your Lisp SBCL/Swank setup blog article. I found it to be well written and very helpful. Currently, I am working on the problem of updating code in the wild without disrupting service. I am used to how this is done in Ruby-land using Capistrano, but haven’t worked out my flow for Lisp yet. I haven’t yet drunk the Koolaid on updating a live Lisp process in a REPL. To me it seems unwise to not have rollback and I am used to having. If you have any thoughts to share on that, I would love to hear them, i.e., how Lispers handle that aspect of deployment/update.

    Again, thanks.

  2. Hi, i never actually developed any Ruby production code so i can’t really make any kind of comparison but here’s how i do it in Common Lisp:

    First i develop my code locally keeping track of changes using git. When i feel my code is ready to be integrated into the production environment then i use slime/swank to update the running process.

    Depending on the depth of the changes i sometimes leave them running process with the updated code (via slime/swank) run for 24hrs or more before actually replacing any source code files that might exist on the server, this way if my changes crash sbcl/hunchentoot when these restart they will reload my still-not-updated source code files and i have had immediate rollback.

    If something fails after i have replaced the source code files, then i just use the previous version of the code from my git repository, load them in the running process through slime/swank again (if possible) and/or replace the source code files to their previous state.

    In my code i have never used any kind of database migrations (ala Rails style) so my workflow is very simple.

  3. On step 5 when I try to install the server I get this error:

    Initialized empty Git repository in ~/clbuild/source/cl-base64/.git/
    git.b9.com[0: 216.184.11.2]: errno=No route to host
    fatal: unable to connect a socket (No route to host)

    error: update was interrupted.
    Use “clbuild update –resume” to retry. (See also “clbuild skip PROJECT”).

    I can ping the site its trying to access just fine, so I really don’t know whats going on. –resume does not fix.

    Any guidance would be much appreciated.

  4. Hi smitty, unfortunately there seems to be a problem with b9.com host, i tried it myself and several packages hosted there seem to throw an error (ie: md5, clsql).

    You can find alternate sources for some of these packages, for hunchentoot and it’s dependencies you can check the BKNR third-party SVN repository, that’s the where Hunchentoot main development happens and it also includes all the Hunchentoot dependency libraries (cl-base64, md5, etc…).

    After you get an alternate repository locations for these packages you can edit the ‘projects’ file in clbuild folder and replace the current repo settings, after that clbuild should be able to find and install these libraries again.

    Hope this helps.

  5. Yup, that was pretty much the problem. Here are the changes I make to the ‘projects’ file starting on line 67:

    puri get_svn http://bknr.net/svn/trunk/thirdparty/puri #URI library

    md5 get_svn http://bknr.net/svn/trunk/thirdparty/md5 #MD5 digest implementation

    cl-base64 get_svn http://bknr.net/svn/trunk/thirdparty/cl-base64 #Base64 encoder/decoder

    clsql get_svn http://bknr.net/svn/trunk/thirdparty/clsql #database interface Library

    rt get_svn http://bknr.net/svn/trunk/thirdparty/rt #regression test driver

    Thanks for your help, I haven’t progressed through step 5 so I might well have more questions.

  6. Hi again smitty, glad to see you managed to solve that issue. I think i’ll drop a line to clbuild’s maintainers (if i can find their addresses) and let them know of this problem. I’m not sure how many people rely on clbuild but this should be affecting several others.

    If you have more questions i’ll be glad to help out (if i’m able to because i’m not an advanced Common Lisp user).

  7. Alright, first off there is a bit of an error in the post of the start-hunchentoot script, it seems to depend on monitor resolution but not all the code is displayed in the browser… which gave me a few problems because i copied and pasted without paying enough attention 😉

    After I fixed that things worked much better, hunchentoot actually shows up in the browser and such, but now I am having trouble with getting into a remote REPL in emacs. When ever I use M-x slime-connect, 127.0.0.1, 4005 I get this error:

    make client process failed: connection refused, :name, SLIME Lisp, :buffer, nil\
    , :host, 127.0.0.1, :service, 4005

    Got any idea what’s going on?

  8. Hi smitty, first off you’re right, i should have already fixed the blog theme, it’s quite messed 😛

    About the error you’re getting, i sometimes get that error when connecting to my remote Lisp process if i forget to first create (and leave open) an SSH tunnel. Please check that you have successfully run something like:

    $ ssh -L 4005:127.0.0.1:4006 _user_@_remote_box_

    and that process is actually left running.

    Basically this opens an SSH connection to _remote_box_ as _user_ and sets everything in a way that all the data sent to your local machine port 4005 will be sent to _remote_box_, Your Lisp process will then see that data as coming from 127.0.0.1 (locally on the remote server) on port 4006 (which was the port i setup swank on).

    Hope this helps a bit, sorry if my explanation of SSH tunneling is not so good…

  9. First, thank you for this guide, it’s the only I have found for this process!

    There are two things I don’t understand, and simply ignored last time I followed it:

    * the adduser line.
    I can’t get it to work. adduser doesn’t understand –system here but tries to create a user called “–system”. useradd however knows “–system”, but not –disabled-password! I can’t find any clues about creating a password-less user. Any advice? This is on Arch Linux, why would it work differently?

    * the . /lib/lsb line in the init-script.
    What is it for? It’s not mentioned in the guide. I don’t have that folder, is it something Ubuntu-specific? Should I just remove that line?

    Again, great guide, wish there were more like it.

  10. Hi maomao,

    As you have realized this post is quite old and also ubuntu specific. I expect most of it can be reused on other distros with minor changes. So,

    1) You can try using useradd command instead since adduser is a wrapper script and the behavior varies from distro to distro. My advice is that you consult the man page for useradd, try the following on the console:

    $ man useradd

    2) the line you mention is a ubuntu specific file that contains some common functions for init-scripts. I believe the only function from these that I use in my script is status_of_proc in hunchentoot_status()

    I believe archlinux provides the file /etc/rc.d/functions that has a similar purpose but with different syntax.

    Sorry but I’m not experienced enough in archlinux to provide much valuable help in porting the script to archlinux.

  11. Thank you for your , I got it to work now!
    Some conclusions:
    * –disabled-password disables login by password, which isn’t really important, but it can be done in Arch Linux by commenting out some lines around 388 in /usr/sbin/useradd.
    * the status_of_proc line, I just commented it out for now, don’t really need it.
    * “detachtty” works from the shell, but needed full path in the script to run.
    * Not sure what the benefit of using clbuild is, I use sbcl in the init-script instead.
    * (declare (ignore addr port)) in the multiple-value-bind prevents lots of warning text in the
    dribble file.
    * hunchentoot:stop can be uncommented, works fine for me.

    Again, thanks. 🙂

    1. Hi, I’m glad you managed to get it working and thanks for the update.
      Regarding your question about using clbuild at the time I was using clbuild to build sbcl and at that time it also provided a nice way of maintaining libraries updated. Remember these were the times before we all had ASDF 2 and Quicklisp.
      Regarding the line with the commented hunchentoot:stop, this was a problem with the specific hunchentoot version from clbuild at the time I wrote the article. Everything seems to work nicely without the comment with more recent versions.

Leave a Reply

Your email address will not be published. Required fields are marked *