2024-07-20 Bots again
=====================
Today I couldn't log into the server. ssh just sat there, waiting for
the server. Some web page loaded a few bytes and stopped. I knew it:
some idiot had once again written a bot that was hitting my web apps
(also known as "expensive endpoints") because they ignored all the
rules. They also didn't get caught by fail2ban because of they were
using cloud services, of course. And the only thing that helps against
these fuckers is banning the whole network. Time to get working.
First, I had to reboot the server via the website of my service
provider.
Then I had to figure out which app it was. It's impossible for me to
find out. All my expensive apps are from a directory called
/home/alex/farm.
When I look at the system log file I, find entries like these:
grep "Out of memory" /var/log/syslog | tail | cut -b 72-
Out of memory: Killed process 1525 (/home/alex/farm) total-vm:95736kB, anon-rss:7700kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:208kB oom_score_adj:0
Out of memory: Killed process 521 (node) total-vm:1268068kB, anon-rss:63440kB, file-rss:0kB, shmem-rss:0kB, UID:118 pgtables:2112kB oom_score_adj:0
Out of memory: Killed process 12302 (/home/alex/farm) total-vm:92464kB, anon-rss:54820kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:208kB oom_score_adj:0
Out of memory: Killed process 12622 (/home/alex/farm) total-vm:92420kB, anon-rss:53040kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:208kB oom_score_adj:0
Out of memory: Killed process 12628 (/home/alex/farm) total-vm:92712kB, anon-rss:51460kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:212kB oom_score_adj:0
Out of memory: Killed process 12632 (/home/alex/farm) total-vm:92556kB, anon-rss:51916kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:208kB oom_score_adj:0
Out of memory: Killed process 12206 (/home/alex/farm) total-vm:92320kB, anon-rss:57256kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:208kB oom_score_adj:0
Out of memory: Killed process 12016 (/home/alex/farm) total-vm:92292kB, anon-rss:57740kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:208kB oom_score_adj:0
Out of memory: Killed process 12689 (/home/alex/farm) total-vm:92728kB, anon-rss:57444kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:212kB oom_score_adj:0
Out of memory: Killed process 12041 (/home/alex/farm) total-vm:92484kB, anon-rss:58288kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:208kB oom_score_adj:0
This is not helping. Who is 12041? There are large lists of the
process identifiers and some numbers in the log, but the name is
always shortened:
grep "12041" /var/log/syslog | tail -n 3
2024-07-20T19:51:03.968678+02:00 sibirocobombus kernel: [ 3275.028117] [ 12041] 1000 12041 23121 14572 212992 4117 0 /home/alex/farm
2024-07-20T19:51:03.968882+02:00 sibirocobombus kernel: [ 3275.028363] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/monit.service,task=/home/alex/farm,pid=12041,uid=1000
2024-07-20T19:51:03.968883+02:00 sibirocobombus kernel: [ 3275.028378] Out of memory: Killed process 12041 (/home/alex/farm) total-vm:92484kB, anon-rss:58288kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:208kB oom_score_adj:0
Useless.
By that time, however, the system was slowing down again. I fired up
htop and saw a gazillion instances of the Community Wiki script.
Looking at the /etc/log/apache2/access.log file shows that bots were
requesting the edit pages (!) of all the thousands of Community Wiki
pages.
Thanks, idiots.
I needed to find where these requests where coming from. My
leech-detector script was as limited to single IP numbers as
fail2ban. So the first order of business was to add some
functionality: If given the option --networks, it now looks up the
network range the requests come from and reports those:
tail -n 50 /var/log/apache2/access.log | grep communitywiki | bin/admin/leech-detector --networks
This is expensive, of course, so I try to cache the Route View ASN
lookups but the cache doesn't persist between calls. Using a small
number of rows is the correct approach, here.
The result is a table like the one below. When I ran it, I used 400
rows or so.
+----------------+------+--------+------+---------+------------+
| IP | HITS | BANDW | REL | INTERV | STATUS |
+----------------+------+--------+------+---------+------------+
| 117.22.0.0/15 | 1 | 6K | 25% | | 200 (100%) |
| 59.172.0.0/14 | 1 | 6K | 25% | | 200 (100%) |
| 114.104.0.0/14 | 1 | 6K | 25% | | 200 (100%) |
| 220.184.0.0/13 | 1 | 6K | 25% | | 200 (100%) |
+----------------+------+--------+------+---------+------------+
If you then start using whois on the numbers, you'll see:
* 117.22.0.0 → China Telecom
* 59.172.0.0 → China Telecom
* 114.104.0.0 → China Telecom
* 220.184.0.0 → Zhejiang Telecom
I think you know where this is going.
And so I copying the network IP, calling whois, verifying that it was
Chinese Telecom or related, and added it to my ban-cidr script. (Note
how I'm using netfilter-persistent these days so that I don't have to
run ban-cidr after every server restart.)
And now I'm going to add the ranges above to the script.
Some days I feel like I'm slowly starting to ban the whole commercial
cloud service internet because nothing good ever seems to come of it.
Do you feel the same?
#Administration #Bots #Butlerian_Jihad #Bots #Butlerian_Jihad
#Butlerian_Jihad
2024-07-22. Here's another slight annoyance, for ngircd. There are a
lot of log lines for a day:
journalctl --unit ngircd.service|grep "Jul 21"|wc -l
7436
More than half of them come from a single IP hosted by Octopuce:
journalctl --unit ngircd.service|grep "Jul 21.*2001:67c:288:2::231"|wc -l
4347
They all have this format:
Jul 21 11:39:51 sibirocobombus ngircd[594]: Accepted connection 16 from "[2001:67c:288:2::231]:41240" on socket 8.
Jul 21 11:39:51 sibirocobombus ngircd[594]: Using X509 credentials from slot 0
Jul 21 11:39:51 sibirocobombus ngircd[594]: Shutting down connection 16 (SSL accept error, closing socket) with "[2001:67c:288:2::231]:41240" ...
Jul 21 11:39:51 sibirocobombus ngircd[594]: Client unregistered (connection 16): SSL accept error, closing socket.
Jul 21 11:39:51 sibirocobombus ngircd[594]: Connection 16 with "[2001:67c:288:2::231]:41240" closed (in: 0.0k, out: 0.0k).
I'm going to try and use fail2ban for this.
Looks like this is happening every 10 seconds:
journalctl --unit ngircd.service|grep "Jul 21.*Shutting down"|tail|cut -d ' ' -f 3
23:58:23
23:58:33
23:58:43
23:58:53
23:59:03
23:59:13
23:59:23
23:59:33
23:59:43
23:59:53
So here's my attempt at a filter, /etc/fail2ban/filter.d/ngircd.conf:
# Fail2Ban filter for failed ssl connections to the ngIRC daemon
[INCLUDES]
# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf
[Definition]
# Example:
# Shutting down connection 16 (SSL accept error, closing socket) with "[2001:67c:288:2::231]:44846"
_daemon = ngircd
failregex = ^%(__prefix_line)sShutting down connection [0-9]+ \(SSL accept error, closing socket\) with "<HOST>:[0-9]+" \.\.\.$
ignoreregex =
[Init]
journalmatch = _SYSTEMD_UNIT=ngircd.service + _COMM=ngircd
# Author: Alex Schroeder
And this is the jail, /etc/fail2ban/jail.d/ngircd.conf – and here I
want to ban them after four failed connects in a minute. Remember the
default ban lasts 10 minutes. Since this attack is moving so slowly, I
want to increase this ban time to an hour.
[ngircd]
enabled = true
findtime = 60
maxretry = 4
bantime = 60m
Let's give it a try!
systemctl reload fail2ban
Then again: fail2ban is only for future attacks. The current level of
persistency also deserves its own treatment:
ipset create banlist6 hash:net family inet6
ip6tables -I INPUT -m set --match-set banlist6 src -j DROP
ip6tables -I FORWARD -m set --match-set banlist6 src -j DROP
ipset add banlist6 2001:67c:288:2::231
There we go.
Oh, and as soon as I did that, it switched to 91.194.61.231, but only
once every two or three minutes. Still Octopuce, though.
ipset add banlist 91.194.60.0/23
2024-07-22. Back to Community Wiki and its enemies, however.
1.92GiB for the bots?
2024-10-17. I am not alone.
> It recently occurred to me that all my rants about “AI” training are
> rather specific and isolate certain bad actors — but that I never
> really talked about the larger issue here, namely that “AI” is the
> worst parasite the free and open web has ever seen. – Leeches,
> Leeches, Everywhere