Inanity 2: From RAGS to Riches
------------------------------
This is the second post in my planned epic three part tale.
The following is poorly written, contains loads of spelling
and gramatical errors, and is likely of interest to almost
nobody. Proceed at your own risk!!
* * *
After being asked several times by various people whether I had my own
gemlog, I finally spent some time last weekend doing something about
this. Of course, in the spirit of this The Lambda Lab this meant
implementing a new server in Scheme. :-)
The goal was to have something like Scratchy [1] (my gopher server)
but for gemini. Honestly I wasn't expecting this to be very hard at
all. The only thing I was unsure of was the TLS aspect.
I've never had anything to do with TLS, besides the experience getting
elpher to initiate TLS connections to supporting gopher servers and
then gemini servers. Some searching revealed that there already was
an egg (that's what Chicken calls its packages :-) ) providing openssl
bindings. Sadly, although it is nicely documented, the documentation
lacks any examples. While this isn't usually a problem with scheme
libraries, it was a show-stopper for me as - honestly - I didn't know
what half of the options to the procedures used to listen for
incomming connections actually meant.
Eventually I went hunting through the source code of the one package
that seemed to use this library: the HTTP library, Spiffy. Therein I
was able to find the necessary examples, which honestly seemed too
simple - most of the options to the listener-creation procedures being
apparently unnecessary. I was able to use this to cobble together a
simple test that would accept TLS connections on a port and send a
"Hello, World!\n" when a client connected. After managing to get this
working, replaced the hello world with a gemini header string and
successfully managed to get av98 to connect to the port and process
the result.
At this point I figured I was home free. I spent an enjoyable hour or
two writing the document delivery code. This was a wonderful
experience which again drove home how much nicer it is to deal with
gemini, both on the server and the client side, than gopher. The
resulting server is at least as capable as Scratchy, but both shorter
and less complex. (The ability of servers to indicate the mime-type
of the content they produce makes so many things easier.)
Now I must digress for a moment to gush about how lisps make the
creation of extensible programs just incredibly trivial. Here's the
definition of the procedure which reads a a scheme function from a
file, then applies that function to the URI requested by the client:
(define (serve-script config uri)
(let* ((path (document-path config uri))
(proc (eval (with-input-from-file path read))))
(with-current-working-directory
(pathname-directory (document-path config uri))
(lambda () (proc uri)))))
If you're not used to lispy languages, this might just look like a
mess of parentheses, but just look at the let binding up the top. The
expression (eval (with-input-from-file path read)) is responsible for
opening the file and parsing the contents as lisp expression, then
evaluating the result. That file can have comments, the lisp
expression can be arbitrarily long: it can have it's own imports,
procedure definitions, etc etc. Talk about concise! All due to the
power of `read'. There's a document [2] on Richard Stallman's web page
where he discusses his various opinions on programming languages,
among other things, and where he states that "Most other [non-lisp]
languages have nothing comparable to `read', nothing comparable to
`eval', and nothing comparable to `print'. What gaping deficiencies!"
Even after a few short years hacking on elisp and scheme, I tend to
agree.
(Of course, the above procedure comes with a _bunch_ of security
concerns, but (a) it's just a hobby server, and (b) as long as I
carefully vet the extension scripts I run and take steps to ensure
nothing I haven't vetted is ever run, I think I'm pretty safe.)
Okay, so once the document serving code was complete, I quickly hooked
it up to the TLS front-end, fired up a local instance and verified
that av98 could connect and retrieve documents. Too easy! I got
excited and, in a bout of prescient madness, gave the server its name:
RAGS, the Right-Awful Gemini Server. All that remained was to
cross-compile a static binary for deployment on my VPS...
$ csc_linux -static rags.scm
/usr/bin/ld: /usr/local/lib/chicken/11/openssl.o: in function `stub1057':
openssl.static.c:(.text+0xaf3b): undefined reference to `SSL_get_peer_certificate'
/usr/bin/ld: openssl.static.c:(.text+0xaf52): undefined reference to `X509_get_issuer_name'
/usr/bin/ld: openssl.static.c:(.text+0xaf5f): undefined reference to `X509_NAME_oneline'
/usr/bin/ld: openssl.static.c:(.text+0xaf6a): undefined reference to `X509_free'
...
Okay, you get the idea.
Now, I've had trouble statically compiling my chicken projects before.
A couple of times I've even had to ask for help on the #chicken
freenode channel. There's always been a solution, and the Chicken
community is one of the most notoriously helpful and friendly around,
so it's always been a pleasant experience. However in this instance I
suspected that the problem was more severe than normal due to the
involvement of openssl, and the fact that my server code relied on an
egg which provided bindings to this library but didn't include the
library itself. I thus shelved the whole idea of producing a static
binary.
You might be asking - why am I bothering with static binaries? Well,
running a Chicken binary with shared library dependencies requires, of
course, the Chicken libraries to be installed. This on its own isn't
a problem: my VPS runs Debian and there's, of course, a chicken-bin
package containing the required files. However my VPS is (was)
running Debian 8, for which the most recent Chicken version available
was 4.something. There's are some pretty big difference between
Chicken 4 and 5, and all of my code is written assuming Chicken 5.
Then why not just compile Chicken 5 on the VPS? Well, I tried that
when I was just getting set up a little over a year ago, but here's
the thing - my VPS only has 128MB of RAM. Which is enough for an
amazing number of things but, apparently, not enough to compile
Chicken 5 with gcc.
Thus, for the whole of the last year, I've been hacking together
gopher, mail, corewar servers, I've been using Chicken's -static
option to produce hulking 5MB ELFs that run wherever. (In combination
with a Docker container allowing me to build such executables on the
distinctly non-GNU/Linux machine I've been suffering with, for my
sins.)
So... what now?
Just for kicks, I decided to do what I should have tried ages ago - I
tried to see how far I could get by compiling a dynamic executable and
then simply copying .so files over to the VPS. It turns out that
there really weren't many that needed to be copied: just libchicken.so
and those corresponding to the individual eggs. And it worked! I
shouldn't have been surprised, but honestly you doubt everything when
you get to my age. I quickly replaced all of my old static executables
for dynamic versions and saved a bunch of memory across the board.
And, once I installed the required openssl libs on the server, I was
able to get RAGS running too.
Fantastic! Now I just needed to check that av98 could connect to
thelambdalab.xyz...
.. but no. No it doesn't. It refuses to connect. The server reports
that it and av98 failed to agree on a cypher to use for the connection.
AAArrrrgh!!!
Okay, deep breaths.
The first thing I tried was commenting out the line of the av98 python
code that restricts the choice of cyphers. It was then able to connect
without trouble, meaning that it really was an issue with the required
cyphers not being made available by my server. But why was this only
a problem on my VPS? After all, I was using the same openssl
libraries there as in my local environment, wasn't I?
To try to understand this, I used a script to spam the server with TLS
connections restricted to particular cyphers so that I could build up
a list of which cyphers were being made available. Repeating this on
the server when running on my localhost confirmed that the cyphers
supported by the server running on the VPS formed a much smaller
subset of the ones made available by the server running on my home
machine.
At this point I began to suspect the problem. Debian 8 ships with
libssl 1.0.1, while on my local machine I'm using libssl 1.1.0.
Happily, my VPS provider sent me an email sometime over the last few
months indicating that I could now upgrade to Debian 10 with a click
of a button. After confirming that Debian 10 would indeed allow me to
easily install later versions of this library (but sadly not
Chicken..) I figured that today was the day to press this button.
After an amazingly short time (seriously, maybe 30 seconds? a minute?)
I was able to ssh into the freshly re-imaged server and begin the
painful process of putting pieces of my Rube Goldberg configuration
back together. It took nearly half a day, in the end, and this is
despite having anticipated this nightmare from the beginning and
fastidiously backing up every one of the small number of configuration
files that I've modified over the last year. Oh how I wish I were
running Guix or Nix.
Anyway, _finally_ I was able to test RAGS on the new configuration and
confirm that it was able to meet the cypher requirements of av98. I
quickly put together a Gemini landing page, together with a
dynamically generated (using those wonderful scheme extension
scripts!) list of my recent phlog posts and an atom feed generator.
After checking that everything ran smoothly and rendered nicely in
elpher, bombadillo and av98, I went to bed.
Which, if you made it through all of that sorry tale, I'm sure you'll
want to do too! 0.o
If you're keen to see what the state of things is now, head over to
gemini://thelambdalab.xyz! And if you want to glance over the RAGS
source, comment on it or submit patches, visit the project's gopher
page at gopher://thelambdalab.xyz/1/projects/rags.
Until next time...
---
[1]: gopher://thelambdalab.xyz/1/projects/scratchy/
[2]: https://stallman.org/stallman-computing.html