Chapter One
Incident Response Team: What are the Roles and Responsibilities?
As much as we may wish it weren’t so, there are some things that only people, and in some cases, only certain people, can do. As one of the smartest guys in cyber security points out below, some things can’t be automated, and incident response is one of them. That’s why having an incident response team armed and ready to go - before an actual incident needs responding to, well, that’s a smart idea. A first key step is to clearly define the incident response team roles and responsibilities (we'll cover all that ground in this guide).
In fact, there are several things we’ll cover in this chapter of the Insider’s Guide to Incident Response. First of all, your incident response team will need to be armed, and they will need to be aimed. Even though we cover true “armature” in terms of incident response tools in Chapter 4, we’ll share some of the secrets of internal armor - advice that will help your team be empowered in the event of a worst-case scenario.
And second, your cyber incident response team will need to be aimed. In any team endeavor, goal setting is critical because it enables you to stay focused, even in times of extreme crisis and stress.
In this chapter, you’ll learn how to assemble and organize an incident response team, how to arm them and keep them focused on containing, investigating, responding to and recovering from security incidents.
“Incident Response needs people, because successful Incident Response requires thinking.” — Bruce Schneier, Schneier on Security
Love This Guide?
Try USM Anywhere for Free
Accelerate your threat detection and incident response with all of the essential security controls you need in one easy-to-use console.
Incident Response Team Roles
Team Leader
Drives and coordinates all incident response team activity, and keeps the team focused on minimizing damage, and recovering quickly.
Lead Investigator
Collects and analyzes all evidence, determines root cause, directs the other security analysts, and implements rapid system and service recovery.
Communications Lead
Leads the effort on messaging and communications for all audiences, inside and outside of the company.
Documentation & Timeline Lead
Documents all team activities, especially investigation, discovery and recovery tasks, and develops reliable timeline for each stage of the incident.
HR/Legal Representation
Just as you would guess. Since an incident may or may not develop into criminal charges, it’s essential to have legal and HR guidance and participation.
Who’s on the incident response team?
We’ve put together the core functions of an incident response team in this handy graphic. Since every company will have differently sized and skilled staff, we referenced the core functions vs. the potential titles of team members. So you might find that a single person could fulfill two functions, or you might want to dedicate more than one person to a single function, depending on your team makeup. That said, here are a few other key considerations to keep in mind:
- IT leads with strong executive support & inter-departmental participation.
When it comes to cyber security incident response, IT should be leading the incident response effort, with executive representation from each major business unit, especially when it comes to Legal and HR. While the active members of the team will likely not be senior executives, plan on asking executives to participate in major recruitment and communications efforts.
- Clearly define, document, & communicate the roles & responsibilities for each team member.
While we’ve provided general functions like documentation, communication, and investigation, you’ll want to get more specific when outlining your team member roles. Make sure that you document these roles and clearly communicate them, so that your team is well coordinated and knows what is expected of them - before a crisis happens.
- Establish, confirm, & publish communication channels & meeting schedules.
Effective communication is the secret to success for any project, and it’s especially true for incident response teams. Print out team member contact information and distribute it widely (don’t just rely on soft copies of phone directories. Chances are, you may not have access to them during a security incident). Include important external contacts as well, and make sure to discuss and document when, how, and who to contact at outside entities, such as law enforcement, the media, or other incident response organizations like an ISAC.
What Does an Incident Response Team Do?
An incident response team analyzes information, discusses observations and activities, and shares important reports and communications across the company. The amount of time spent on any of one of these activities depends on one key question: Is this a time of calm or crisis? When not actively investigating or responding to a security incident, the team should meet at least quarterly, to review current security trends and incident response procedures. The more information that an incident response team can provide to the executive staff, the better, in terms of retaining executive support and participation when it’s especially needed (during a crisis or immediately after).
What’s the Goal of an Incident Response Team?
The incident response team’s goal is to coordinate and align the key resources and team members during a cyber security incident to minimize impact and restore operations as quickly as possible. This includes the following critical functions: investigation and analysis, communications, training, and awareness as well as documentation and timeline development.
Investigation/Analysis
Is this an incident that requires attention now? Which assets are impacted?
Determine and document the scope, priority, and impact.
Reporting/Communications
Which types of security incidents do we include in our daily, weekly, and monthly reports? Who is on the distribution list? What information can we provide to the executive team to maintain visibility and awareness (e.g. industry reports, user behavioral patterns, etc.)?
Define and categorize security incidents based on asset value/impact. Document and educate team members on appropriate reporting procedures. Collect relevant trending data and other information to showcase the value the incident response team can bring to the overall business.
Response/Improvement
What’s the most effective way to investigate and recover data and functionality? How do we improve our response capabilities?
Investigate root cause, document findings, implement recovery strategies, and communicate status to team members.
Where Should Incident Response Team Members Be Located?
Most companies span across multiple locations, and unfortunately, most security incidents do the same. While you might not be able to have a primary team member onsite at every location, strive to have local presence where the majority of business and IT operations happen. The likelihood that you’ll need physical access to perform certain investigations and analysis activities is pretty high… even for trivial things like rebooting a server or swapping out a HDD.
How Can Incident Response Team Members Be Armed?
If an incident response team isn’t empowered to do what needs to be done during a time of crisis, they will never be successful. That’s why it’s essential to have executive participation be as visible as possible, and as consistent as possible. Otherwise, the team won’t be armed effectively to minimize impact and recover quickly… no matter what the scope of the security incident.
The key is to sell the value of these critical incident response team roles to the executive staff. No matter the industry, executives are always interested in ways to make money and avoid losing it. The stronger you can tie your team goals and activities to real, measurable risk reduction (in other words cost reduction), then the easier it will be for them to say yes, and stay engaged.
Quantifiable metrics (e.g. number of hours of work reduced based on using a new forensics tool) and reliable reporting and communication will be the best ways to keep the team front-and-center in terms of executive priority and support.
Incident Response Capabilities Needed >
How Should I Choose the Right Incident Response
Team Members?
In terms of incident response team member recruitment, here are three key considerations based on NIST’s recommendations from their Computer Security Incident Handling guide.
Aim for 24/7 Availability
Chances are, your company is like most, and you’ll need to have incident response team members available on a 24x7x365 basis. In fact, from my experience and those of other insiders, Friday afternoons always seemed to be the “bewitching” hour, especially when it was a holiday weekend. Please note that you may need some onsite staff support in certain cases, so living close to the office can be a real asset in an incident response team member. Be specific, clear and direct when articulating incident response team roles and responsibilities.
Consider Virtual or Volunteer Team Members (if full-time isn’t an option)
You may not have the ability to assign full-time responsibilities to all of your team members. With a small staff, consider having some team members serve as a part of a “virtual” incident response team. A virtual incident response team is a bit like a volunteer fire department. When an emergency occurs, the team members are contacted and assembled quickly, and those who can assist do so. Typically, the IT help desk serves as the first point of contact for incident reporting. The help desk members can be trained to perform the initial investigation and data gathering and then alert the cyber incident response team if it appears that a serious incident has occurred.
Monitor & Bolster Employee & Team Morale
Incident response work is very stressful, and being constantly on-call can take a toll on the team. This makes it easy for incident response team members to become frazzled or lose motivation and focus. It is important to counteract staff burnout by providing opportunities for learning and growth as well as team building and improved communication. You may also want to consider outsourcing some of the incident response activities (e.g. SIEM monitoring) to a trusted partner or MSSP.
Why Participate on an Incident Response Team?
As we pointed out before, incident response is not for the faint of heart. It takes an extraordinary person who combines intellectual curiosity with a tireless passion for never giving up, especially during times of crisis. This description sounds a lot like what it takes to be a great leader. And that’s what attracts many of us insiders to join the incident response team. The opportunity to become and be seen as a leader inside and outside of your company is one that doesn’t come often, and can reap more benefits than can be imagined at first. You’ll learn things you’ve never learned inside of a data center (e.g. disclosure rules and procedures, how to speak effectively with the press and executives, etc.) and you’ll be seen as a leader throughout your company.
For those who are new to cyber security & incident response.
What skills are needed? What are the important
things to look for?
In addition to technical expertise and problem solving, cyber incident response team members should have strong teamwork and communication skills. Speaking and writing skills are essential because cooperation and coordination are the key to effective incident response. Intellectual curiosity and a keen observation are other skills you’ll want to hone.
Security analysis is detective work – while other technical work pits you versus your knowledge of the technology, Security analysis is one where you’re competing against an unknown and anonymous person’s knowledge of the technology. Detective work is full of false leads, dead ends, bad evidence, and unreliable witnesses – you’re going to learn to develop many of the same skills to deal with these.
Five Tips For Incident Response Team Members:
Look for the Common Denominators.
Look for the Common Exceptions.
Security analysis inevitably involves poring over large sets of data – log files, databases, and events from security controls. Finding leads within big blocks of information – logs, databases, etc, means finding the ‘edge cases’ and ‘aggregates’ – what is the most common thing out there, the least common – what do those groups have in common, which ones stand out?
A system may make 10,000 TCP connections a day – but which hosts did it only connect to once? When following a trail of logs, always be looking for the things you can group together, with something they have in common, then find the one that stands out.
Make Assertions, Not Assumptions.
“Don’t make assumptions,” common wisdom says – they’re right, assuming that something is there and continuing on that assumption will lead to poor results in incident response teams. But in an effort to avoid making assumptions, people fall into the trap of not making assertions. In order to find the truth, you’ll need to put together some logical connections and test them.
“If I know that this system is X, and I’ve seen alert Y, then I should see event Z on this other system.”
This is an assertion – something that is testable – and if it proves true, you know you are on the right track! (assuming your assertion is based on correct information). Always be testing.
Eliminate the Impossible.
According to good ol’ Sherlock Holmes, “When you have eliminated the impossible, whatever remains, however improbable – must be the Truth.”
You are going to encounter many occasions where you don’t know exactly what you are looking for… to the point where you might not even recognize it if you were looking directly at it. In these circumstances, the most productive way forward is to eliminate the things that you can explain away – until you are left with the things that you have no immediate answer to – and that’s where find the truth.
Always Look for a Simpler Explanation.
“Never attribute to malice, that which is adequately explained by stupidity.” – Hanlon’s Razor.
What makes incident response so rewarding is the promise of hunting down and stopping that “red letter day” intrusion before it can do the real damage. When your job involves looking for malicious activity, it’s all too easy to see it everywhere you look.
Sometimes that attack you’re sure you have discovered is just someone clicking the wrong configuration checkbox, or specifying the wrong netmask on a network range.
Imagine things from the Attacker's Perspective.
What would you do in their position?
Experienced incident response team members, hunting down intrusions being controlled by live human attackers in pursuit of major corporate IP theft, have a skill that cannot be taught, nor adequately explained here.
From experience administrating systems, building systems, writing software, configuring networks – but also, from knowing how to break into them – you can develop that ability to ask yourself “what would I next do in their position?” – and make an assertion on that question that you can test (and it may often prove right, allowing you to ‘jump ahead’ several steps in the investigation successfully).
Bottom line: Study systems, study attacks, study attackers- understand how they think – get into their head. Be smarter than your opponent.
Navigate Your Journey:
Can you share some of the lessons you’ve learned from surviving a data breach?
Here are the things you should know about what a breach looks like, from ground zero, ahead of time. Stress levels will be at an all-time high, interpersonal conflicts will boil to the surface, that dry-run disaster planning drill you've been meaning to do for months, but never found the time for? That one minor change request your senior engineers have had sitting on the table for weeks that consistently got deferred in favor of deploying that cool new app for the sales team? You betcha, good times.
Here are some of the things you can do, to give yourself a fighting chance:
Don’t Let Security Be an Island
IT departments (and engineers) are notorious for the ‘ivory tower’ attitude, we invented the term ‘luser’ to describe the biggest problem with any network. Create some meetings outside the ‘IT Comfort Zone” every so often; the first time you meet the legal and PR teams shouldn’t really be in the middle of a five-alarm fire.
Bring some of the people on the ground into the incident response planning process - soliciting input from the people who maintain the systems that support your business processes every day, can give much more accurate insight into what can go wrong for your business/than any book full of generic examples can. These are the people that spend their day staring at the pieces of the infrastructure that are held together with duct-tape and chicken wire.
Give people a place to talk
Nondisclosure agreements will be flying left and right, stress levels will be high, and the PR and legal secrecy machine will be in full force. Many employees may have had such a bad experience with the whole affair, that they may decide to quit. Keeping secrets for other people is a stress factor most people did not consider when they went into security as a career choice. Invite your HR department staff to join any NDA discussions, and give employees a place to vent their concerns confidentially and legally. You’ll be rewarded with many fewer open slots to fill in the months following a breach.
Let others learn from your mistakes
If you are required to disclose a breach to the public, work with PR and legal to disclose information in a way that the rest of the world can feel like they have learned something from your experiences. Adam Shostack points out in ‘The New School of Information Security’ that no company that has disclosed a breach has seen its stock price permanently suffer as a result. However the fallout of intentionally vague and misleading disclosures may hang over a company’s reputation for some time. Sharing lessons learned can provide enormous benefits to a company’s reputation within their own industries as well as the broader market.
It gets better
Famously overheard at a recent infosec conference - “We’re only one more breach away from our next budget increase!”. There’s nothing like a breach to put security back on the executive team’s radar. Take this as an opportunity for new ideas and approaches, not just “We’re finally getting that thing we’ve been asking for, all year”. Use the opportunity to consider new directions beyond the constraints of the ‘old normal’. Now is the time to take “Misfortune is just opportunity in disguise’ to heart.
Test for Impact, not Vulnerabilities
If you are spending money on third-party penetration testing, you should be expecting more in return than the output of a vulnerability scanner and some compromised systems - expect reports that show results in terms of impact to business operations, bottom lines and branding - these are the things your executives need to be aware of - either you look for and determine them ahead of time, or your attacks do.
Don’t Panic!
Murphy’s Law will be in full effect. The information the executive team is asking for, was only being recorded by that one system that was down for its maintenance window, the report you need right now, will take another hour to generate and the only person with free hands you have available, hasn’t been trained on how to perform the task you need done before the lawyers check in for their hourly status update. Panic generates mistakes, mistakes get in the way of work. This advice works from both ends of the command chain - if your executive team is expecting a fifteen-minute status update conference call every hour, that’s 25% less work the people on the ground are getting done. Calm Heads Rule The Day - set expectations early on and don’t go into a disaster recovery plan that principally operates on the impossible expectations.