As much as we may wish it weren’t so, there are some things that only people, and in some cases, only certain people, can do. As one of the smartest guys in cyber security points out below, some things can’t be automated, and incident response is one of them. That’s why having an incident response team armed and ready to go - before an actual incident needs responding to, well, that’s a smart idea. A first key step is to clearly define the incident response team roles and responsibilities (we'll cover all that ground in this guide).
In fact, there are several things we’ll cover in this chapter of the Insider’s Guide to Incident Response. First of all, your incident response team will need to be armed, and they will need to be aimed. Even though we cover true “armature” in terms of incident response tools in Chapter 4, we’ll share some of the secrets of internal armor - advice that will help your team be empowered in the event of a worst-case scenario.
And second, your cyber incident response team will need to be aimed. In any team endeavor, goal setting is critical because it enables you to stay focused, even in times of extreme crisis and stress.
In this chapter, you’ll learn how to assemble and organize an incident response team, how to arm them and keep them focused on containing, investigating, responding to and recovering from security incidents.
“Incident Response needs people, because successful Incident Response requires thinking.” — Bruce Schneier, Schneier on Security
Drives and coordinates all incident response team activity, and keeps the team focused on minimizing damage, and recovering quickly.
Collects and analyzes all evidence, determines root cause, directs the other security analysts, and implements rapid system and service recovery.
Leads the effort on messaging and communications for all audiences, inside and outside of the company.
Documentation & Timeline Lead
Documents all team activities, especially investigation, discovery and recovery tasks, and develops reliable timeline for each stage of the incident.
Just as you would guess. Since an incident may or may not develop into criminal charges, it’s essential to have legal and HR guidance and participation.
We’ve put together the core functions of an incident response team in this handy graphic. Since every company will have differently sized and skilled staff, we referenced the core functions vs. the potential titles of team members. So you might find that a single person could fulfill two functions, or you might want to dedicate more than one person to a single function, depending on your team makeup. That said, here are a few other key considerations to keep in mind:
When it comes to cyber security incident response, IT should be leading the incident response effort, with executive representation from each major business unit, especially when it comes to Legal and HR. While the active members of the team will likely not be senior executives, plan on asking executives to participate in major recruitment and communications efforts.
While we’ve provided general functions like documentation, communication, and investigation, you’ll want to get more specific when outlining your team member roles. Make sure that you document these roles and clearly communicate them, so that your team is well coordinated and knows what is expected of them - before a crisis happens.
Effective communication is the secret to success for any project, and it’s especially true for incident response teams. Print out team member contact information and distribute it widely (don’t just rely on soft copies of phone directories. Chances are, you may not have access to them during a security incident). Include important external contacts as well, and make sure to discuss and document when, how, and who to contact at outside entities, such as law enforcement, the media, or other incident response organizations like an ISAC.
An incident response team analyzes information, discusses observations and activities, and shares important reports and communications across the company. The amount of time spent on any of one of these activities depends on one key question: Is this a time of calm or crisis? When not actively investigating or responding to a security incident, the team should meet at least quarterly, to review current security trends and incident response procedures. The more information that an incident response team can provide to the executive staff, the better, in terms of retaining executive support and participation when it’s especially needed (during a crisis or immediately after).
The incident response team’s goal is to coordinate and align the key resources and team members during a cyber security incident to minimize impact and restore operations as quickly as possible. This includes the following critical functions: investigation and analysis, communications, training, and awareness as well as documentation and timeline development.
Is this an incident that requires attention now? Which assets are impacted?
Determine and document the scope, priority, and impact.
Which types of security incidents do we include in our daily, weekly, and monthly reports? Who is on the distribution list? What information can we provide to the executive team to maintain visibility and awareness (e.g. industry reports, user behavioral patterns, etc.)?
Define and categorize security incidents based on asset value/impact. Document and educate team members on appropriate reporting procedures. Collect relevant trending data and other information to showcase the value the incident response team can bring to the overall business.
What’s the most effective way to investigate and recover data and functionality? How do we improve our response capabilities?
Investigate root cause, document findings, implement recovery strategies, and communicate status to team members.
Most companies span across multiple locations, and unfortunately, most security incidents do the same. While you might not be able to have a primary team member onsite at every location, strive to have local presence where the majority of business and IT operations happen. The likelihood that you’ll need physical access to perform certain investigations and analysis activities is pretty high… even for trivial things like rebooting a server or swapping out a HDD.
If an incident response team isn’t empowered to do what needs to be done during a time of crisis, they will never be successful. That’s why it’s essential to have executive participation be as visible as possible, and as consistent as possible. Otherwise, the team won’t be armed effectively to minimize impact and recover quickly… no matter what the scope of the security incident.
The key is to sell the value of these critical incident response team roles to the executive staff. No matter the industry, executives are always interested in ways to make money and avoid losing it. The stronger you can tie your team goals and activities to real, measurable risk reduction (in other words cost reduction), then the easier it will be for them to say yes, and stay engaged.
Quantifiable metrics (e.g. number of hours of work reduced based on using a new forensics tool) and reliable reporting and communication will be the best ways to keep the team front-and-center in terms of executive priority and support.
In terms of incident response team member recruitment, here are three key considerations based on NIST’s recommendations from their Computer Security Incident Handling guide.
Chances are, your company is like most, and you’ll need to have incident response team members available on a 24x7x365 basis. In fact, from my experience and those of other insiders, Friday afternoons always seemed to be the “bewitching” hour, especially when it was a holiday weekend. Please note that you may need some onsite staff support in certain cases, so living close to the office can be a real asset in an incident response team member. Be specific, clear and direct when articulating incident response team roles and responisibilities.
You may not have the ability to assign full-time responsibilities to all of your team members. With a small staff, consider having some team members serve as a part of a “virtual” incident response team. A virtual incident response team is a bit like a volunteer fire department. When an emergency occurs, the team members are contacted and assembled quickly, and those who can assist do so. Typically, the IT help desk serves as the first point of contact for incident reporting. The help desk members can be trained to perform the initial investigation and data gathering and then alert the cyber incident response team if it appears that a serious incident has occurred.
Incident response work is very stressful, and being constantly on-call can take a toll on the team. This makes it easy for incident response team members to become frazzled or lose motivation and focus. It is important to counteract staff burnout by providing opportunities for learning and growth as well as team building and improved communication. You may also want to consider outsourcing some of the incident response activities (e.g. SIEM monitoring) to a trusted partner or MSSP.
As we pointed out before, incident response is not for the faint of heart. It takes an extraordinary person who combines intellectual curiosity with a tireless passion for never giving up, especially during times of crisis. This description sounds a lot like what it takes to be a great leader. And that’s what attracts many of us insiders to join the incident response team. The opportunity to become and be seen as a leader inside and outside of your company is one that doesn’t come often, and can reap more benefits than can be imagined at first. You’ll learn things you’ve never learned inside of a data center (e.g. disclosure rules and procedures, how to speak effectively with the press and executives, etc.) and you’ll be seen as a leader throughout your company.
In addition to technical expertise and problem solving, cyber incident response team members should have strong teamwork and communication skills. Speaking and writing skills are essential because cooperation and coordination are the key to effective incident response. Intellectual curiosity and a keen observation are other skills you’ll want to hone.
Security analysis is detective work – while other technical work pits you versus your knowledge of the technology, Security analysis is one where you’re competing against an unknown and anonymous person’s knowledge of the technology. Detective work is full of false leads, dead ends, bad evidence, and unreliable witnesses – you’re going to learn to develop many of the same skills to deal with these.
Security analysis inevitably involves poring over large sets of data – log files, databases, and events from security controls. Finding leads within big blocks of information – logs, databases, etc, means finding the ‘edge cases’ and ‘aggregates’ – what is the most common thing out there, the least common – what do those groups have in common, which ones stand out?
A system may make 10,000 TCP connections a day – but which hosts did it only connect to once? When following a trail of logs, always be looking for the things you can group together, with something they have in common, then find the one that stands out.
“Don’t make assumptions,” common wisdom says – they’re right, assuming that something is there and continuing on that assumption will lead to poor results in incident response teams. But in an effort to avoid making assumptions, people fall into the trap of not making assertions. In order to find the truth, you’ll need to put together some logical connections and test them.
“If I know that this system is X, and I’ve seen alert Y, then I should see event Z on this other system.”
This is an assertion – something that is testable – and if it proves true, you know you are on the right track! (assuming your assertion is based on correct information). Always be testing.
According to good ol’ Sherlock Holmes, “When you have eliminated the impossible, whatever remains, however improbable – must be the Truth.”
You are going to encounter many occasions where you don’t know exactly what you are looking for… to the point where you might not even recognize it if you were looking directly at it. In these circumstances, the most productive way forward is to eliminate the things that you can explain away – until you are left with the things that you have no immediate answer to – and that’s where find the truth.
“Never attribute to malice, that which is adequately explained by stupidity.” – Hanlon’s Razor.
What makes incident response so rewarding is the promise of hunting down and stopping that “red letter day” intrusion before it can do the real damage. When your job involves looking for malicious activity, it’s all too easy to see it everywhere you look.
Sometimes that attack you’re sure you have discovered is just someone clicking the wrong configuration checkbox, or specifying the wrong netmask on a network range.
Experienced incident response team members, hunting down intrusions being controlled by live human attackers in pursuit of major corporate IP theft, have a skill that cannot be taught, nor adequately explained here.
From experience administrating systems, building systems, writing software, configuring networks – but also, from knowing how to break into them – you can develop that ability to ask yourself “what would I next do in their position?” – and make an assertion on that question that you can test (and it may often prove right, allowing you to ‘jump ahead’ several steps in the investigation successfully).
Bottom line: Study systems, study attacks, study attackers- understand how they think – get into their head. Be smarter than your opponent.