CCDC Nationals in San Antonio, TX

CCDC blog post! We ended up getting 2nd place at Nationals.

Many good vibes received and useful lessons learned. If you're not familiar with CCDC, just read this:

The mission of the Collegiate Cyber Defense Competition (CCDC) system is to provide institutions with an information assurance or computer security curriculum a controlled, competitive environment to assess their student's depth of understanding and operational competency in managing the challenges inherent in protecting a corporate network infrastructure and business information systems. (via nccdc.org)

What? That's nonsense? Welcome to the real world. That's what CCDC tries to be-- the 'real world'-- except, you have twenty disgusting computers to manage by yourself, and you have anywhere from -5 to 10 minutes to secure them before being kerplunked by twelve instances of Cobalt Strike.

I've been trying to qualify for CCDC Nationals for three years, since my first "season" in 2019-2020. Finally, this year, UCF stayed in their region, and we didn't red team ourselves to death with SLAs. It's harder than it looks.

The National event was ton of fun, and we did pretty alright. Of course, we also made a bunch of mistakes.

Strategy

Generally, the strategy is: change passwords ASAP, automate everything, and cheese as much as you can.

We had two people (including myself) on Linux, so we had two first priorities: database/webapp password changes, and root password changes. I wrote a tool to run bash scripts on all systems (kind of like an off-brand Ansible, you can see it here), which I used to run password change scripts and run through basic box securing (e.g., if php.ini, disable these functions, change shells to rbash, and so forth). The other Linux person worked on the database, and changing all creds related to it.

I would've written a lot more scripts, and improved the tooling for automatically configuring new boxes (e.g., correctly escalate to root with sudo/doas). BUT, we were prohibited from making changes to our script repository since 30 days before the regional event. The amount of foresight they expect college students to have is baffling. I didn't even know where I would be or what I would be doing in two months, let alone making sure my scripts were "nats ready." If you qualify, they should let you make changes until 30 days before Nationals as well.

As an ancillary task, I manage one or two of the (typically) three ESXi boxes. For Nationals, this became my first priority. (This turned out to be a waste of time-- ESXI systems were out of scope. I'll talk about this more later, but this severely reduces the red team skill ceiling, and makes things less fun.)

Giga Genius Cheese

Our strategy above intentionally prioritizes root password changes and non-scored credential changes, because scored credentials are only needed for a small number of checks, and you can often make them entirely "secure" without even changing the creds! Case in point, I never changed SMTP credentials and we had 100% uptime on it (though I will admit that wasn't intentional).

Consider this. After you change all non-scored password services, what's left?

• SMTP/POP3
• FTP
• SSH
• SAMBA (maybe)

Scored services on fileshares are always read-only. FTP and SAMBA done. SMTP/POP3, they could delete emails, but I backed them up, and it's scored on sending new emails, not retrieving old ones. Also consider that you could chattr +i the mail directories. That leaves SSH, as the hardest one.

However, if you change all shells to rbash, and export an empty path in the bashrc for non-root users... the experiment is complete. Dwayne's engine, Green. My SSH, essentially nonfunctional. Scored credentials, unchanged. CIAS_Alex, out of a job?

Team Composition

We chose to adopt a slightly unconventional team composition. We had three M$ caretakers, two Linux (like I mentioned above), an inject handler, and a red teamer(!). The red team person was in charge of scanning our network, testing our fixes, and otherwise finding additional flaws and holes that may be easier to find from the "outside."

If I were to compose a team again, I would probably just go for three Linux people where one is in charge of red-team-esque duties, rather than having a separate job, especially given how Linux-focused the environments are.

Regionals

Services

The regionals environment was themed as "Malachor", an energy company.

As with previous years, the vast majority of scored services are hosted on Linux (thankfully). We were given two injects to migrate services, and we chose to use them on DOLIBARR and WWW. The database was mostly centralized on peragus, which simplified database/webapp password changes. The only two sites that were not connected to that database were DOLIBARR and BLOG (Wordpress), both of which were using local databases.

Red Team

Historically, we have not felt too much heat from the red team. During the competition, I noticed two malicious binaries running as bfett (because we didn't change the scored user passwords immediately). After killing those sessions and moving the binaries, I didn't hear anything from them again.

Alas, when we received our red team report, we saw that they were able to dump two databases and we lost another 200 points for an exposed web server with PII. I'm almost certain this is DOLIBARR and WWW (Opencart)-- both of these Windows machines had databases, and DOLIBARR had a lot of employee PII. We also lost another ~1000 points from red team login access, which I would guesstimate around 200 of is on Linux, because we deprioritized scored service password changes for user-level accounts.

Injects

Injects were handled very well. Having one or two inject people definitely pays off. I only had to do a couple things so I can't complain. Last year, where I tried and failed to set up Elastic for the entire competition, was a different story.

Lessons Learned

Overall, I think we did really well, and my Linux partner and I especially got pretty close to optimal score.

However, we could improve many things. Just focusing on Linux, most notably, we could've improve our tooling and scripts. I was typing the root password (which was the same for every machine...) every time I wanted to run a script. If I were able to use pubkey authentication, it would be impossible to replay my password or otherwise keylog like it would be with password authentication. (I added this to our script, but as I was complaining about before, we had to use the older version that was submitted before regionals).

Finally, we didn't prioritize ESXI security at all, since we knew the passwords were unique, which I figured wouldn't be the same at the national event.

Nationals

Finally, after three years of trying, here's the juice... Nationals. All-expenses-paid trip to a unnecessarily large hotel/resort in San Antonio, Texas. It used to be in Orlando, this one might be easier for UTSA's travel.

Travel wasn't too bad, only one of our flights got delayed, so I'll call it a win. Going with this team of people was a lot of fun. Like with all (most) of the events I do, I feel super lucky to have them.

We got in early in the day (arriving around 1PM) on Thursday, so we spent the day screwing around. The actual competition is ~10AM to ~6PM on Friday and Saturday, awards on Sunday.

Services

At the start of Day 1, we came into the big banquet hall after getting our team packets, and sat through the mandatory "don't be stupid, and here's how CCDC works" presentation. We then waited outside our rooms (we were fortunate to have one close to the ops room), and the competition started at 10AM sharp.

The first thing I did, before putting my bag down, or breathing, was to change the ESXI password. The ESXI was on a particularly chunky laptop. Of course, it turns out this was for naught, because ESXIs were out of scope.

Here's the team packet picture. Hard to read? Same IRL.

And the grind began. The first two hours are always super frantic, since it feels like a race, regardless of if people are actually in your machines or not. I thought I locked myself out of my PC in the first five minutes, but I was just typing the new root password wrong (typical panic and tunnel-vision).

I spent all of my time in tmux-- blessed it be-- hopping between different boxes, renaming tabs, having sessions die as our firewall guy moved the boxes behind the firewall. I ran a bunch of scripts through the tool I wrote (mentioned above), and it Just Worked. It was awesome having a simple capability to run scripts on all systems, and very satisfying that something I wrote actually worked.

Passwords changed, firewall scripts ready to be customized. An occasional beep from the point-of-sale system. Green team wants to see the price of Golden Eye. Can you give me share access to drop the screenshot? Yeah, for the failed logins inject. You should have access now. FTP is down on the scoreboard! Did you turn the firewall back on? Yeah I did, I'll turn it off again, don't know why that isn't working. OrangeHRM DB password is changed, but I can't log in to the webapp. Just change the DB hashes by one letter, they don't actually need login access. Blog is down! Restarted, fixed, was some weird PHP thing. Can you look at this, is this process legit? Yeah that's normal. Can you take care of firewall on Osirus? He's using GUI on Airen, don't kill GDM. Anything not working? Nope, Palo looks good. New inject, 'reverse this binary.' I can take a look at that later, remind me in an hour? Why can I not change passwords on Valentine? Shop is down? You have it? Yeah I'll fix it. Look at this, super sus, a systemd user service running as brettc. I killed and changed creds, you want to set up rbash for that box? Screenshot share is still not working, did you add my IP? I'm working on fixing SSH on the suse box, I'll run them all at once. Did you find any new boxes in the environment? Nope, looks like the map has everything. All webapps are up. For now, don't jinx it.

The eight hour days go by quick.

Red Team

Each team at Nationals has three red teamers, but they'll "spread the love" (as they say) and help one another, point out new vulnerabilities found, share credentials, share exploits, and such.

The stereotypical CCDC experience, the one they want and publicize, is that Day 1 is extremely quiet-- the thought being that the red team acts like a real APT and is extremely stealthy-- and Day 2 is when they take destructive action.

In reality, it was pretty noisy Day 1 on Linux, but Windows did see basically no activity after the first hour. On Linux, we saw constant login attempts, and about an hour in, default cred abuse (for the ones we didn't change, due to our strategy).

This would've worked fine, but I got lazy, and it came back to bite us. The environment had a bunch of game servers with docker containers, and since they were only checked by orange team (human volunteers), I didn't care enough to prioritize them. If orange team needed to check it, it'd probably be up (since I didn't think they had a way to escalate), and we could always revert the image if they did find a way. But it wasn't worth the time to dig out all the persistence for a trash game server.

But in the last hour, all of traffic from the ESXi (where the game servers were hosted) started to drop, like we'd only get maybe one out of every four packets. The clever thing the red teamers did was change the IP addresses of the game servers to be the same as the gateway, meaning all ESXi traffic was sent between four different boxes. They didn't have any access to our critical services, but they still took them down. Which is a lesson in humility and thoroughness, I suppose. I still don't know how they escalated.

We were talking to our red teamers after, and they said that they were only able to do one thing (i.e., shut off a service) before we kicked them out entirely. I was proud of that outcome, but also felt at some level that they shouldn't have been able to do that one thing at all. Since it turned out that one thing was very destructive.

Injects

Like regionals, I didn't have to think about injects too much. Thanks inject handler!

Fun

Just like with HackThePort, I think one of the best parts was meeting all the other competitors. I recognized a lot of faces, and they were all super nice. Even my mortal enemies from UCF were friendly and fun to talk to. Chatting with another team's coach until midnight on Day 1 was definitely one of the trip highlights.

San Antonio itself was a good experience too. Besides the resort pools/slides and such, there were a bunch of interesting restaurants and tourist attractions (not that we had time for many).

I also really enjoyed taking to the red teamers at the Day 1 dinner and the Day 2 social. It's hard to portray or give a summary of these intangibles in blog-form (or perhaps I lack the literary skill), but it's like a computer convention, but more focused and enjoyable.

Panoply

There's a just-for-funsies event on Sunday, after the competition closes, called Panoply. It's king of the hill, where they use their previous environments' images. You got two hours to be the red team. The credentials aren't easily guessable (with the exception of some webapps), but you do know what services are scored. At 192.168.100.100 internally, you get a page that looks just like the public one, except there are scores for each of the ~50 boxes-- either up, up and compromised (each team has a different icon), or down. You get scored if your flag is accessible (either in HTTP, FTP, SMB, whatever the check it). You don't get any points if it's down.

The ideal strategy here seems to be to find the easy boxes (anonymous write FTP, MS17-10, random CVE) and automate exploiting them, since chances are that they'll get taken down later by people fighting over them. Boxes that are down for a couple checks get automatically get reverted. That might be enough to win, but to lock it in, you should leverage your access to dump creds (Invoke-Mimikatz, /etc/shadow, whatever), and use that to get access to the "hard" boxes (the SSH and updated RDP only ones).

Of course, my words are weightless since we did not win this event. We got fourth, which I would blame on not automating or going for the easy boxes at all. We got some credentials from a Linux box, which I was able to get login on one machine for, and got some points towards the end. But the effect of getting a box early, or multiple boxes, and letting it compound, is much more impactful.

This event didn't have "prizes", but now that I think about it, CCDC doesn't offer any real prizes either. Weird that every other "prestigious" cyber security event has some kind of prize, typically monetary. I guess CCDC rides on its history.

Why Did We Lose?

So why did we lose? We got 2nd place, UCF got first.

Well, as usual, any number of things we could have not missed or done better would have contributed to making up the point gap (which was about 700 points). If I had to point at something, it'd be misprioritizing tasks combined with tunnel vision.

The first missed opportunity was in getting the ecommerce server up, which I very briefly looked at before moving on to the standard password changes/hardening stuff. Only when our inject handler asked me to look at it again did I find the first part, which was really simple to find (just .bash_history in a user's home folder). From there you were supposed to follow the source SSH IP and find the other scripts on a different machine, but I didn't do that, since I thought just the history was good enough, and the inject wasn't a priority for me. If we got that up earlier it wouldve been ~100 points. Second major source of points lost was not leaving enough time for the RE challenges, which we could have gotten if we left more than 30 minutes for them (~400-600 points). And last was just a bummer, we submitted the flag for the last RE challenge with one character typo'd, netting us zero points (200 points).

Other than that, standard less sloppy sysadmin work and better management of the docker servers, would have been more than enough to land first place. But UCF really did a great job, and didn't leave much of a margin. Although, I'm sure they have some (smaller than ours) list of mistakes as well.

Summary

As usual, everyone prefers Linux. Fun social event. Good educational experience. If you have the chance to do CCDC or something similar, I'd definitely recommend it.