How I Killed the Plusser and Why It Had It Coming
For those not familiar, the anonradio room in com has a few helper
bots that run in the background. One very useful bot prints out
metadata from the DJ as audio is streamed (track info, titles, etc).
Another prints the time every 15 minutes. And then there was (past
tense) the plusser.
The plusser was a bot that kept "scores" for users. Anytime a line of
chat started with an active username and a postfix operator (++, --,
+-, -+) the bot would "award" the mentioned user an increment or
decrement point (or a NOP), and print a new score. Using the ! key
(repurposed from an earlier b1fffst3r mode) would print a Top 20
DAMGUD CYBERCHATTER table with current leaders. Scores were often used
to "award" the active DJ points, but were also handed out like candy
for jokes, good tidbits of info, or even general greetings.
At 10 points a user would receive a code for an award token that could
ostensibly be redeemed with membership for prizes (stickers), but the
actual receipt of such prizes was hit or miss.
The plusser was not an original feature of com, but was added at the
request of a user quite some time back (before I went active in com).
It was not universally popular. Some users hated that it brought a
social-media-esque "like button/karma" feature to com. Some users
disliked the extra lines of text (often many, many lines of text) it
added to the com log. But a major problem was that the plusser was
STRONGLY (though never provably) linked with the dreaded "com lag":
periods were com would seem to hang up solidly, sometimes for minutes.
tob was not a fan
-----------------------------------------------------------------------
[tob] YAY \o/ the effing plusser is broken!
[16:56:49] [14/37/128] (aNONradio): Kaneko Ayano - Joshou
[ingrix] tob++ not a fan of the plusser? ;)
[tob] Maybe it will die a respectful death, finally.
[ingrix] I didn't realize it wasn't always around
[tob] ingrix++ It almost makes us as cool as facebook and reddit!
-----------------------------------------------------------------------
others were ambivalent
-----------------------------------------------------------------------
[Thu 03-Oct-24 14:15:00]
[maxime] hi
[14:16:37] maxime@sdf has left anonradio
[johnm] quintin-- plusser back?
[14:21:25] johnm -- quintin down to -10000002759
[johnm] indeed it is, finally
[ingrix] dang it, I kind of liked having it gone
-----------------------------------------------------------------------
I was suspicious
-----------------------------------------------------------------------
[Thu 03-Oct-24 15:00:00]
[15:01:06] [14/41/67] (gregf): repeating gregf from 26-Aug-2021
[scm] Yeah.
<zoerhoff is suffering through an early meeting. Thankfully I only have one more today. Unfortunately, it's at 2130 local time tonight.>
[zoerhoff] com lag right after the plusser comes back? Highly sus.
[scm] Indeed.
-----------------------------------------------------------------------
As the code is not open, it was never clear *how* the plusser stored
its score data. As com appears to be otherwise file-based, I formed a
hypothesis that the plusser was also file-based, with each user's
score kept in a file in some central directory, but I had no way to
test my hypothesis. Until recently.
At some point, someone (I don't recall who) discovered that prefixing
the username with a / still incremented/decremented the score just
fine. This led me to my first potential test of my file-based scoring
hypothesis:
The Beginning of the End
-----------------------------------------------------------------------
[Tue 22-Oct-24 01:15:00]
...
[zoerhoff] /scm++
[01:25:03] zoerhoff ++ /scm up to 1053
[scm] lol
[scm] :D
[quintin] scm++ == -- (cover all the bases)
[01:25:21] quintin ++ scm up to 1054
[01:25:31] 13 listeners with a daily peak of 30 and 130 peak for the month.
[zoerhoff] .scm++
[scm] :>
[zoerhoff] *scm++
[01:25:49] zoerhoff ++ *scm up to 1055
[zoerhoff] hah!
[01:26:01] 13 listeners with a daily peak of 30 and 130 peak for the month.
<zoerhoff confirms (?) a suspicion about how com stores scores>
...
[jwh] is it stripping non-alpha prefixes?
[zoerhoff] I think each score is a file name marching the user. Leading * and / parse away
[jwh] hmm
[zoerhoff] /com/some/path/score//user would be fine
[zoerhoff] trying to examine a machine thats opaque, with no xrays or tools to open it
[jwh] &zoerhoff++
[Tue 22-Oct-24 01:30:00]
[zoerhoff] //jwh++
[01:30:28] zoerhoff ++ //jwh up to 918
[jwh] several years ago, there was some kind of glitch that reset the perms of the /sdf/com tree; for brief time, it was world-readable. i kinda kick myself for not taking a quick tar of it :(
<zoerhoff has an evil thought>
[zoerhoff] flip*++
[01:32:09] zoerhoff ++ flip* up to 1
[jwh] ooo
[zoerhoff] flipchip++
[01:32:30] zoerhoff ++ flipchip up to 7
[jwh] root*++
[quintin] that "flip" result is interesting.. since that's not an account.
[quintin] heh
[jwh] yeah
[ratxue] flip*++
[01:33:39] ratxue ++ flip* up to 1
[jwh] q*--
[01:33:43] jwh -- q* down to -1
[quintin] hmmmm
[ratxue] flip*--
[01:33:55] ratxue -- flip* down to -1
[ratxue] flip*++
[01:34:18] ratxue ++ flip* up to 1
[ratxue] lol
[jwh] whoa, it skipped 0
[ratxue] supercalifragilisticexpialidocious*--
[quintin] You didn't spell it right, so it won't work.
[ratxue] quintin---- well spotted
[01:36:37] ratxue -- quintin down to -10000002856
[quintin] ratxue++ lulz
[01:36:50] quintin ++ ratxue up to 4575
-----------------------------------------------------------------------
At this point, we were starting to figure out the fact that basic
globs (/ and *) "worked" in that they caused the plusser to do
something, but clearly that something was not entirely correct.
From here, things degraded quickly as the experiments grew ever
more complex.
The Middle of the End
-----------------------------------------------------------------------
[Tue 22-Oct-24 02:45:00]
[userfxnet] nanu nanu
[userfxnet] e*++
[userfxnet] ;/
[userfxnet] user*++
[02:51:30] userfxnet ++ user* up to 1
[userfxnet] hmm.... OH. I SEE
[userfxnet] quin*++
[02:51:51] userfxnet ++ quin* up to -10000002855
[userfxnet] riiiiight riiiiight ahahahaha
[quintin] ha!
[userfxnet] yeeeaaah, feel me?? interesting stuff. glad I caught that bit o chat earlier.
[quintin] I have been globbed
[userfxnet] I guess so have I and whoever else has the user prefix
[userfxnet] snow*++
[02:53:02] userfxnet ++ snow* up to 4380
[zoerhoff] quin*--
[02:53:04] zoerhoff -- quin* down to -1
[userfxnet] WOAH
[zoerhoff] quintin--
[02:53:25] zoerhoff -- quintin down to -10000002857
[quintin] That's... inconsistent.
[userfxnet] freaky. oooOOOoooh. halloween COM frights and sights
[zoerhoff] What a treasure trove of weirdness
[quintin] oh wait. quin* got -1, but I went from 2855 to 2857, so it is wildcarding..
[userfxnet] i think when it's trying to give an output print of some score, although wildcarding, is unable to give one from a singular source and as a result just prints either god damn nonsense or some kinda combined number i imagine?
[userfxnet] I guess give one score from a single source, as well as to, given the fact your score was affected quintin
[zoerhoff] need to think a bit about this, but I've gleaned a few things tonight
[userfxnet] nice*+-
[userfxnet] doesn't work with NOPs?
[zoerhoff] zoe+-
[zoerhoff] zoe*+-
[02:57:15] zoerhoff NOPs zoe*
[quintin] hard to tell since noop gonna noop
[userfxnet] over*-+
[02:57:39] userfxnet NOPs over*
[userfxnet] I'm confused now
[userfxnet] zoe and nice don't work, but over does??
[zoerhoff] user*++
[02:58:25] zoerhoff ++ user* up to 1
[userfxnet] /userfxnet++
[02:58:41] userfxnet ++ /userfxnet up to 3068
[userfxnet] /userfxnet--
[02:58:55] userfxnet -- /userfxnet down to 3067
[userfxnet] /user*-+
[02:59:16] userfxnet NOPs /user*
[userfxnet] o.0
[quintin] userfxnet, I'm wondering if it it's just spitting out the first $UNAME + $SCORE and not even written to expect an array/dict/list, etc.
[zoerhoff] user*++
[02:59:58] zoerhoff ++ user* up to 1
[Tue 22-Oct-24 03:00:00]
[zoerhoff] userfxnet++
[03:00:06] zoerhoff ++ userfxnet up to 3068
[userfxnet] Oh i see so just whatever it fetches first, yeah that makes sense
[zoerhoff] curioser and curioser
[quintin] I'm 100% guessing, but yeah
[userfxnet] very!
[03:01:22] [30/33/130] (openmic): Aimyon - Matryoshka
[quintin] I just noticed there is a "snow*" in the "TOP-20 DAMGUD CYBER CHATTERS" list
[quintin] one point ahead of snowdusk
[zoerhoff] o.0
[03:05:47] [27/33/130] (openmic): Aimyon - Something Blue
[03:06:17] [25/33/130] (openmic): Kaneko Ayano - Jet Coaster
[zoerhoff] Did...did we break it?
[quintin] zoerhoff++ We?
[03:07:43] quintin ++ zoerhoff up to 2853
[quintin] :)
[zoerhoff] quintin-- Fine, ufx did it, but I kinda started this
[03:08:25] zoerhoff -- quintin down to -10000002858
[quintin] zoer*++ Lettuce sea
[03:08:38] quintin ++ zoer* up to 2854
[03:08:44] [23/33/130] (openmic): GESSERIT - Cave
[zoerhoff] HAH
[quintin] Oh.. check it out. Now we have a zoer*
[zoerhoff] Is this better or worse than the buffer underrun?
[03:12:06] [21/33/130] (openmic): snowdusk - snowdusk intergalactic wasabi mix radiospot 2
[quintin] Gonna go with better since we can share the fun with multiple unames
[03:12:46] [21/33/130] (openmic): Tristeza - Futuro (Snodgrass mix)
[zoerhoff] we're totally asking for scores to get nuked...
[Tue 22-Oct-24 03:15:00]
[quintin] For providing top tier fuzzing? Well that would be a fine thank you.
[03:18:04] [21/33/130] (openmic): Kaneko Ayano - Hikari No Hou E
[thrig] any sensible shell script will "$var" to prevent the auto-split and auto-glob of $var
-----------------------------------------------------------------------
By this point, it was found that the Top 20 list was becoming corrupted
with the extra wildcard entries being added (and then effectively frozen
in place as the experiments moved on). This was beginning to get weird,
but in classic nerd hax0r fashion, it was about to get werider.
The End of the End
-----------------------------------------------------------------------
[Tue 22-Oct-24 03:30:00]
[thrig] (many shell scripts are not so sensible)
[03:30:07] [21/33/130] (openmic): Open Mic - Anyone can stream is streaming LIVE!
[ingrix] zoer`ls`++
[03:30:22] [21/33/130] (openmic): [dj: lifelog] L-Vis 1990 & Sinjin Hawke — 'The Pit'
[ingrix] zoer`echo -n hoff`++
[ingrix] zoerhof?++
[03:31:14] ingrix ++ zoerhof? up to 2854
[ingrix] zoer$(echo -n hoff)++
[zoerhoff] ahahahahahahah
[zoerhoff] There are 3 of me!
--- quintin@faeroes -------------------------------------------
tyn 2893
zoerhof? 2854
zoer* 2854
zoerhoff 2853
screwtape 2624
--- quintin@faeroes -------------------------------------------
<quintin@faeroes DUMPs in 121 characters of 10 words in 6 lines>
...
[ingrix] u?e?f?n?t++
[03:37:47] ingrix ++ u?e?f?n?t up to 3069
[ingrix] oh my god
[quintin] Yoo!!!!! hahahaha
[mcornick] attack of the globs!
[quintin] the TOP-20 list is GLORIOUS!
...
[quintin] ?n*r?x++ Very resonable
[03:41:32] quintin ++ ?n*r?x up to 1349
[ingrix] we're going to wildcard something and find a hidden username in here
[mcornick] **/*++
[ingrix] or we're going to accidentally break com lol
[userfxnet] t*++
[03:42:20] userfxnet ++ t* up to 1
[zoerhoff] just the plusser
[mcornick] now we are become Death, the destroyer of com points
[ingrix] ok, who is t*
[userfxnet] i'm just here like, who tf did it fetch first for t???
[ingrix] there are two t's in here, thrig and tob, and both have a score above 1
[mcornick] y*++
[mcornick] (there's no one here starting with y at the moment)
[ingrix] ta++
[ingrix] ta*++
[ingrix] tb*++
[userfxnet] tata--
[ingrix] tc*++
[jwh] t[a-m]++
[jwh] t[n-z]*++
[03:44:19] jwh ++ t[n-z]* up to 1
[ingrix] oh, good idea jwh
[ingrix] t[a-z]b++
[03:44:25] ingrix ++ t[a-z]b up to 10429
[userfxnet] r?t++
[ingrix] fuck
[jwh] t[a-m]*++
[03:44:36] jwh ++ t[a-m]* up to 1
[ingrix] yes! binary search!
[zoerhoff] OH NO HE DID IT
[userfxnet] marvelous
[Tue 22-Oct-24 03:45:00]
[03:45:06] [21/33/130] (openmic): Open Mic - Anyone can stream is streaming LIVE!
[quintin] check out TOP-20! hahaha
[ingrix] https://www.youtube.com/watch?v=u3CKgkyc7Qo
[mcornick] we need a bottom 20 so we can see all the quints
[03:45:34] [21/33/130] (openmic): [dj: lifelog] Rick James - Cold Blooded (Secret Mixes Fixes)
[quintin] and a flashlight to see that far into the depths
[ingrix] [j][w][h]++
[03:46:30] ingrix ++ [j][w][h] up to 920
[userfxnet] what even in the hell
[mcornick] m??????k++
[03:46:54] mcornick ++ m??????k up to 1018
[mcornick] I win
[userfxnet] u?x++
[mcornick] LOOK ON MY WORK, YE MIGHTY, AND DESPAIR!
...
[ingrix] t[a-g]*++
[ingrix] t[h-m]++
[ingrix] t[h-m]*++
[03:48:50] ingrix ++ t[h-m]* up to 1
[quintin] Oh Mmm Gee! R u a H@x0rz!?
[ingrix] t[h-k]*++
[03:49:06] ingrix ++ t[h-k]* up to 1
[zoerhoff] so 1337
[ingrix] t[h-i]*++
[03:49:32] ingrix ++ t[h-i]* up to 1
...
[mcornick] also, no one tell smj. this is hilarious and should be
kept in place
[quintin] m*o*n*c*++ for leaningout the limo window
[03:51:54] quintin ++ m*o*n*c* up to 1
[quintin] agree, gotta keep this under the radar. Real low profile.
[userfxnet] q[u-n]--
[quintin] nah.. but maybe.
[quintin] q[u-n]*+-
[03:53:28] quintin NOPs q[un]*
...
[quintin] ingri[X]++ Just how sensative
[quintin] ok, so at least case
[zoerhoff] ingri[xX]++
[04:01:51] zoerhoff ++ ingri[xX] up to 1349
[quintin] yeah, so that's x or X, so confirmed
[ingrix] ingri[xxx]+++
[04:02:29] ingrix ++ ingri[xxx] up to 1349
[ingrix] hell yeah that's my stripper name
[zoerhoff] filthy ingrix
[userfxnet] userfxne[ttttt]++
[04:03:06] userfxnet ++ userfxne[ttttt] up to 3069
[quintin] Thanks! Now com has an NC17 rating.
...
[Thu 24-Oct-24 16:00:01]
[zoerhoff] o/
[ingrix] zoerhof[f]++
[16:00:57] ingrix ++ zoerhof[f] up to 2858
[zoerhoff] plusser delenda est
-----------------------------------------------------------------------
By this point, it was clear that we had found a major hole (perhaps
several) in the plusser. Not only basic wildcards worked, but also
fairly complex glob patterns ([a-z], [aA], etc.) were "working" and
the whole score system was rapidly corrupting. Then this happened:
The Really Final We Mean It End
-----------------------------------------------------------------------
[spuos] /*++
[17:13:13] spuos ++ /Downloads /Mail /SAVE /altroot /arpa /bin /boot /boot.cfg /com /dev /emul /etc /ftp /go /kern /lib /libdata /libexec /lost+found /mail /man.tar /menarea1.tar /meta /mnt /netbsd /nnetbsd /onetbsd /oo /pkgadd /pkgadd93 /pkgsrc /proc /rescue /root /sbin /sdf /stand /sys /tftpboot /tmp /udd /usr /var /vmail /www up to 1
[ingrix] /*[/n]*++
[Fri 25-Oct-24 17:15:00]
...
[17:22:45] zoerhoff@sverige has left anonradio because com is busted
[handyc] haha
[ingrix] uh oh
[17:23:13] zoerhoff@sverige has joined anonradio
[handyc] /sdf/arpa/*++
[zoerhoff] Hmm, back now bug laggy on sverige
[handyc] /bin++
[handyc] /bin/*++
[handyc] /*bin/*++
...
[ldbeth] did that plusplus actually executes shell code
[quintin] not so far...
-----------------------------------------------------------------------
Shortly after this discovery (which let's face it is basically a code
injection vulnerability), the plusser began double-printing scores and
otherwise acting odd. At this point I made the decision to fess up to
the powers that be.
The Bullet to the plusser's head
-----------------------------------------------------------------------
Date: Fri, 25 Oct 2024 20:09:26 +0000
From: Kristian M Zoerhoff <zoerhoff@sdf.org>
To: SDF Membership <membership@sdf.org>
Subject: plusser in com anonradio room
During some recent "explorations" of the limits of the anonradio
room's plusser, things have become rather broken. The bot currently
prints all scrore changes twice, and the DGCC top 20 board is
rather...unusual...as shown below.
The plusser probably needs a restart and some manual cleanup of the
top 20 board. There do seem to be a few bugs found as part of this
process, but without source code access I can't provide any useful
suggestions. One user did suggest that some variables related to
usernames might benefit from quoting ("$var" instead of $var).
TOP-20 DAMGUD CYBER CHATTERS
----------------------------
tob 10462
t[oppybirthdaytoyou]b 10451
t[o,birthday]b 10451
t[a-z]b 10429
mnw 5261
gef 4665
[g]ef 4663
g?f 4661
ratxue 4601
snow* 4380
snowdusk 4379
publius 3935
ffog 3310
stug 3309
userfxne[jackjumpedoveracandlestick] 3097
userfxnet 3096
userfxne[ttttt] 3069
u?e?f?n?t 3069
tyn 2904
zoerhoff 2866
----------------------------
-----------------------------------------------------------------------
This did not, in fact, kill the plusser. Membership first reset the
scores, but this did not last long.
Why Won't You Die!?!?
-----------------------------------------------------------------------
[ingrix] let's see if anything got patched
[ingrix] [q]uintin--
[02:14:41] ingrix -- [q]uintin down to -7
[02:14:41] ingrix -- [q]uintin down to -7
[lykaina] what's the top 20 list?
[ingrix] /*++
[02:14:56] ingrix ++ /Downloads /Mail /SAVE /altroot /arpa /bin /boot /boot.cfg /com /dev /emul /etc /ftp /go /ke rn /lib /libdata /libexec /lost+found /mail /man.tar /menarea1.tar /meta /mnt /netbsd /nnetbsd /onetbsd /oo /pkgadd /pkgadd93 /pkgsrc /proc /rescue /root /sbin /sdf /stand /sys /tftpboot /tmp /udd /usr /var /vmail /www up to 1
[Sat 26-Oct-24 02:15:00]
...
[Sat 26-Oct-24 22:30:00]
[22:34:15] rusty1@faeroes has joined anonradio
[ratxue] rusty1++ o/
[rusty1] o
[ratxue] hmm
[rusty1] plusser no worky?
[ratxue] not for you it would seem :o/
[rusty1] perhaps i am not worthy
[rusty1] ratxue++
[ratxue] plusser is in ruins
[zoerhoff] shuffled off this mortal coil
[rusty1] quitin's fault, too many negatives ?
[zoerhoff] my fault, found some "features" that led to mass experimentation run amok
-----------------------------------------------------------------------
And at long last, on 26-Oct-24 22:30 UTC (ish) the plusser died. There
has been one request in BBOARD to bring it back, but other users have
objected, and interestingly, there have been no measurable incidents
of com lag since its departure.
Am I sorry? Nope. The plusser had some serious bugs and we managed to
find them with no source code access, no documentation, and no map of
what the internal data structures look like. It was a fabulous,
low-stakes vulnerability test, and it was wildly successful. Unless
and until those bugs are patched, it has no business being online,
even a controlled environment like SDF. The fact that its mere
existence has proven divisive should be another nail in the coffin, as
should the performance issues with a file-based system in an NFS
cluster that's already stressed on a good day. It was a fun little toy
while it lasted, but its time is past.
I hope.