2024-07-21 What was hard about the Oddµ architecture?
======================================================

If you know Go, I think you'll find the architecture of Oddmu to be
very traditional: There is a list of actions it knows how to handle
when it serves a site on the web and a list of commands it understands
when called from the shell. Everything involving HTML or the RSS feed
uses a Go template.

There were two things that were tricky, in hindsight. I didn't know
enough Go to know how best to do it and so it took me a long time to
figure them out.

The first was what data to keep in memory. I didn't want search to
involve a database so I experimented a lot with different ways of
indexing the content. Trigrams? Full-text? And what about scoring? In
the end, I dropped all of that. The only thing I keep in memory is a
map of page names and page titles as well as a map of hashtags and
pages. I find that searching titles and hashtags is what I do most
often. For everything else, I search the files. This, of course, is
much slower but for a site of ten thousand pages it's still fast
enough. So figuring out how to do search was hard. Once I had this
figured out, I discovered that keeping maps in memory when there are
multiple Go routines being used by the web server means that I need
locks. Both of my maps are in fact data structures that each contain a
map and a lock. This was new to me and it took me a long time to
realize that having the lock was important. For a while, I didn't have
those locks. 😅

The second thing that took me a long time to get right is that I
wanted pages to be served from Oddµ when it runs as a web server and I
wanted to be able to edit files locally and upload them (using rsync,
for example) while it was still running on the server. I didn't know
about watching the filesystem for changes and had to learn how to do
this: a watcher per directory, spawning new watchers when directories
are created, and updating those maps I keep in memory as files are
added, deleted or edited.

Watching the files introduced another complication, however: What if a
user edited a page via the web? I already had code in place that would
update my maps. Now the file system watcher would report the same file
being written to, resulting in two updates. And what about a change to
a file that results in multiple events: create a file, write something
to the file, write some more to the file. Three events resulting in
three updates? The code is therefore complicated.

If Oddµ knows that a file is going to change ahead of time, like when
a user on the web saves an edit, the file is added to a map of ignored
files for a second. When the file system watcher sees those changes,
no updates are done because it knows that the edit handler already
does it.

If Oddµ sees a file change because of somebody editing files directly,
a one second timer is started. If another event for the same file
arrives, the timer is reset to one second. Only when a second has
elapsed without changes does the watcher do the updates to the
internal data structures. That is to say, changes made to the file
system regarding page title changes or hash tag changes take an effect
on the web with a one second delay.

So now, with all that in place, think of all the go routines running
in parallel. Every one second timer is a go routine. Every go routine
accessing a map like the map if files to ignore means that this map
needs to be a data structure containing a map and a lock.

When I look at that part of the code, I still get dizzy. 😵‍💫

​#Oddµ #Programming ​#Programming

2024-07-21. @bouncepaw@merveilles.town asked about usability issues,
specially in the context of using Oddµ for picture galleries. In my
case, I think the hardest part is finding particular pictures.
Consider the page of flowers in 2020. I know that there's a
sanseveria in bloom on that page, for example. But how can I find it?
I can find the page if I search for "bogenhanf #pictures", and then I
can search for the same word on the page, and then I have to wait for
a bit as the browser lazy-loads pictures.

Perhaps, if I could get into the habit of putting the image
description into the Markdown alt text, I could index all the images
and their alt text, providing direct access to the images. I guess I
would also have to add special code to the Markdown renderer that made
the alt text visible for regular visitors (as title attribute or as
regular paragraphs). That'd be interesting.

Later. I have implemented something useful, I hope. If one of the
search terms (excluding hashtags and predicates) matches an alt-text
of an image, that image is included in the result.

Of course, almost none of my galleries use the alt-text, so right now
there aren't many pages that will show results for a query like
"bogenhanf #pictures".

My expectations regarding alt texts and title attributes for images
have changed significantly as I've spent time on fedi, so I'm really
not sure what to do about Markdown in this respect. Right now I'm in
the habit of adding neither alt-text nor title attribute, trying to
provide enough text in the paragraphs above or below. With this new
search option, that might have to change, too. As a sighted person, I
sort of expect the title attribute to be the same as the alt text.