Red Teaming Through Mistakes

Note: these words are cathartic for me as this is something I’m still practicing myself — I hope there’s something you can take away as well.

I’m a big fan of the “detach” concept that Jocko Willink talks about in his books and podcast. The gist is: step back (at least mentally, if not physically), look around, observe, and employ a little empathy to understand what all parties involved are likely thinking. This is especially important when dealing with a Red Team member’s mistake.

In the case of Red Teaming, the mistake is most likely completely harmless, unless it’s one of these:

- An ethics/trust violation

- Exposure of company, client, customer, or stakeholder data

- Reduction of security controls resulting in a breach

Any of those are very severe and probably require very severe consequences — probably no second chances. That’s just life.

Notice what didn’t make the list: blowing OPSEC during a red team campaign and causing detection, response, or attribution. Nor did trying a tool that worked during testing but failed live in the production target environment.

Why not?

Because red teams aren’t real. They’re simulated. They’re a game we play to learn. The breaches are pretend. The consequences are growth, not damage. Nobody is getting hurt.

While we certainly want to maximize realism when we simulate an adversary, our goal isn’t to win — it’s to improve the detection and response capabilities of the Blue Team, if by nothing else than repetition cycles. Yes, your teammates may be frustrated for a bit if a cover story that took weeks or even months to build gets blown, but they’ll get over it. If they don’t, they probably aren’t in the right line of work.

Yes, we want our tools to always work, but enterprise environments are nothing if not complicated. Chase a broken tool to root cause and discover a new variable to work into future test cycles.

Triage

The focus should always be on creating new challenges for the blue team so they can stretch and learn. So my first advice is always to determine if the campaign can be salvaged as is. Maybe the mistake will be limited in damage to the Red Team’s plans. That’s OK. Not perfect, but OK.

If it’s only attribution that is blown (e.g. linking a red team operator to a phish or C2 domain), then simply roll with the original plan anyway, especially if there are valuable training objectives that will be met from the campaign. Just go through the cycles, since it takes a long time to get from detection to attribution anyway, and the former is more important than the latter.

Or maybe you just need to reset and try again — burn the whole pretext to the ground, create a new one, age your domains, etc. The problem with that is it takes time and time is money. Unless the strategic objectives for the campaign justify this expense, this isn’t my recommendation.

If a tool failed to perform in the live action of the campaign, spend some time to locate the root cause. Then simply move on. Own the mistake — don’t say “the stupid tool didn’t work correctly.” Instead say: “My fault, we’ll add [circumstance X] to the list of things to test for next time before we deploy that tool.” An understanding Blue Team should recognize the damage the tool could have made and still learn something from the experience.

Purple Teams & Mistakes

If the Red Teamer is afraid of looking like an amateur to the Blue Team — oh well, time to remember just how hard the Blue Team job actually is. Odds are massively stacked in favor of the adversary and the way you feel is probably how they feel every day. Stay humble, laugh it off, and don’t let your preconceived notions come into play. It’s probably good for Blue to see Red like this ocasionally. In fact, this is a great opportunity for team building during the debrief. If it helps, tell your trusted agent(s) in advance that “the Red Team” (no need to single anyone out) made a mistake that will result in premature attribution. Then you can all laugh it off later. This may put Blue in a better position to laugh off their future mistakes, which have much more dire consequences than Red anyway.

Fail Fast

The tech industry loves this phrase: “fail fast.” It’s a bit of a buzz word, but it fits well with Red Teams, too. If a mistake was made, so be it. End the campaign early. Collect notes and debrief. Discuss how the mistakes were made. Learn from it, but don’t dwell on the mistake. Just make a note and carry on. Pick a new target, get a new campaign started, and put what you learned into practice. The more cycles, the more that can be learned on both sides. Plus, the more cycles behind you, the farther into the past that mistake will travel. Failing fast helps you put the mistake behind you.

High Performing Teams

One of my favorite takeaways from the expensive Google study on high performing teams is that a team member must feel safe to make mistakes. If a team member cannot overcome a mistake quickly, laugh it off, and learn from it, without fear of persecution from other team members, then the team will break down and eventually become unproductive. The same is true for Red Teams, and even more so for Purple Teams.

If a Red Team operator is afraid to try things to make mistakes, the team will grow stagnant, repeat the same TTPs all the time, and become ineffective. Creativity is the human adversary’s best asset. Along with creativity is the ability to test a hypothesis, which will fail some percentage of the time. I joke with some of my operators that I am going to put “Commit N mistakes” on their annual performance objectives — don’t just stay safe.

“Nobody wakes up in the morning and says, ‘Today will be the day I’ll screw up my work.’”

To the operators who didn’t make the mistake: see the positive intent in your teammate. Have empathy. Your teammate may have been rushed, may have been temporarily clumsy, may have been ignorant about how something worked, but odds are they didn’t do it on purpose. Put yourself in their shoes, because if you don’t, life has a funny way of forcing you to eventually make your own mistakes. Pride comes before the fall. This is a perfect time to employ the Golden Rule: treat them the way you want to be treated when it’s your turn to say “Oops, sorry.” Be careful, but if they’re open to it, and the team has the right chemistry and relationships, laugh it off with them. Maybe tell them about a time you screwed up. But whatever you do, don’t push your teammate to the point where they don’t want to risk making mistakes in the future, because if you do, you’re killing your team one brain cell at a time.

To the leaders (whether you have direct reports or you’re just a technical/campaign lead): be sensitive and keep your finger on the pulse of a situation. An honest mistake not addressed correctly with trust within the team can ruin everything you worked hard to build. Talent doesn’t grow on trees. Recruiting good talent is hard. Retaining talent and keeping them engaged is even harder because that job never ends. Your team is watching how you respond, and you’re setting the tone whether you realize it or not.

Another great Jocko Willink phrase:

“It’s not what you preach, it’s what you tolerate.”

Tolerate mistakes, but do not tolerate the victimization of a teammate who makes a mistake.

Now go put it behind you and do what you love doing.