Incident Response Guide
This document outlines Login.gov’s process for responding to incidents. It outlines roles and responsibilities during and after incidents, and it lays out the steps we’ll take to resolve them.
This is the full explicit document!
In a situation? Check the Incident Response Checklist for a quick reference.
The formal version from the FedRAMP Agency ATO package can be found here Login.gov Incident Response Plan
Login.gov incident response operates under the GSA Incident Response framework. See GSA IT - IT Security Procedural Guide: Incident Response for detailed GSA IR guidance.
- Response process
- Incident Severities
- Frequently Asked Questions and Expectations
At a high level, incident response follows this process:
An TTS staff member inside or outside the Login.gov team (the reporter) notices and reports a Login.gov-related incident, using the TTS incident response process and notifying the Login.gov team in the #login-situation Slack channel.
- The first responder on the Login.gov team (which could be the reporter if the reporter is on the team) becomes the initial Situation Lead (SL).
The SL follows this Login.gov IR Plan and may additionally reference TTS incident response process
- The SL notifies #login-situation that an incident has been declared
- Responders assemble in the War Room ((See the Topic in #login-situation channel for the link)
- SL requests more responders if needed:
- During business hours:
- Call in on-call members using the @login-appdev-oncall and @login-devops-oncall handles in Slack
- Use @here in #login-situation if still understaffed
- After hours:
- Slack or OpsGenie used to alert additional responders (See Emergency Contacts if needed)
- During business hours:
- Roles are assigned when possible as responders join the incident:
- Situation Lead (SL): - Responsible for ensuring all following steps are completed
- Technical Lead (TL): Leads technical investigation and mitigation
- Comms Lead (CL): Coordinates communication outside of #login-situation, within GSA, and if needed, with partners and the public
- Scribe (SC): Relays information discussed in war room (hangout) to #login-situation and aids Situation Lead in recording incident
Roles proceed as follows:
- Situation Lead (SL):
- Requests additional responders as needed, including a new SL if they need to cycle off
- Ensures roles and team are coordinated and have what they need
- Shares context on what is happening and asks clarifying questions
- Tech Lead (TL):
- Leads technical response, delegating technical tasks as needed
- Checks for relevant Incident Response Runbooks and initiates use
- Ensures screen sharing and other methods are used
- Scribe (SC):
- Records significant activities in #login-situation channel to create a timeline
- Asks for links to resources/extra information to record as needed
- Relays information to help someone NOT in the war room who wants to understand the incident
- Comms Lead (CL):
- Creates the official tracking issue for the incident: Incident Template
- Creates the Incident Review document by copying Incident Review Google Doc and shares a link in #login-situation
- Uses GSA IR Email Template to create and send notice to GSA Incident Response firstname.lastname@example.org, IT Service Desk email@example.com (or GSA IT Helpline called), and our GSA ISSO and ISSM within 1 hour of start of incident
- If incident is an outage (problem impacting users’ ability to use Login.gov), SL updates the Login.gov Statuspage via the Statuspage Admin Interface (View Sample Message)
- The SL forms a team (responders) to determine if the event is actually a confirmed incident, and if so assesses the severity (investigating).
- The SL sends out an initial situation report (sitrep), or a false-alarm notification.
- If the cause of the incident is determined to be an outage in a 3rd party vendor, turn on the vendor outage page in the idp via the Vendor Outage Response Process
- The SL coordinates, communicates, and tracks the investigation and remediation.
The responders work to contain and remediate the issue; timelines vary based on the assessed severity.
- Initial stopgap fix
- Longer-term solution
- Impact Discovery
- Send GSA IR (firstname.lastname@example.org) and partner notifications/updates if relevant
- If breach and theft is determined involve GSA IR for forensics
- Other remediation
Discuss closing out urgent response
- Two phases to close out: finished urgent response, and fully closed
- For an incident to remain in the active emergency phase, it must need all hands on deck no matter the time of day/night. If the initial remediation is complete, then urgent response can conclude, which ends the discussion in #login-situation and clears the topic in that channel.
- This doesn’t mean response is finished. There may still be more follow up tasks required during business hours for the team to fully close out the incident ticket.
- Two phases to close out: finished urgent response, and fully closed
The responding team holds a retrospective to analyze the incident, capture follow-up action items and lessons-learned, and write a formal report.
Real-time communication happens in Slack.
- If needed, the team can use a Google Hangout and/or Google Docs to share information that’s not appropriate for Slack or GitHub (PII, etc.).
- About a month after the retrospective, review the action items to see which ones are complete and if any merit further action.
For full details, read on.
An incident begins when someone becomes aware of a potential incident. We define “incident” broadly, following NIST SP 800-61, as “a violation or imminent threat of violation of computer security policies, acceptable use policies, or standard security practices” (6). This is a deliberately broad definition, designed to encompass any scenario that might threaten the security of Login.gov.
When a person outside the Login.gov team (the reporter) notices a Login.gov-related incident, they should begin reporting it by using the 18F incident response process (need link here), and then post about it in #login-situation. If they don’t get acknowledgment from the Login.gov team right away, they should escalate by contacting the Login.gov leads directly until they receive acknowledgment of their report.
The first person to notice an incident, they should begin reporting it by using the this incident response process and posting about it in #login-situation (including notifying the Login.gov leads).
The first participant becomes the initial SL and carries out the next steps in the response. The SL’s responsibility is coordination, not necessarily investigation. The SL’s primary role is to guide the process. The first responder may remain SL throughout the process, or they may hand off SL duties later in the process.
The SL makes sure that the Login.gov incident response process is followed, including supporting the reporter if the reporter already started it, or starting it if nobody has started it yet.
Comms at the Initiate phase
Note that at this point the issue’s status is “investigating” — we haven’t confirmed that it’s really an issue yet. So, we should actually refer to this as just an “event” at this point; it doesn’t become an “incident” until we’ve confirmed it.
At this phase, communications should follow these steps (and any additional steps listed at :
- SL notifies #login channel that an incident is underway in #login-situation, brief summary
- The SL should inform GSA of the investigation by emailing email@example.com, firstname.lastname@example.org with a description of the incident, via a single email to all three addresses WITHIN one (1) hour of the finding.
- GSA-IR team responds:
- Does not yet meet threshold for reportable incident
- Continue investigating and keep us in the loop
- Real-time chat should happen in #login-situation.
- Create an issue in the identity-security-private GitHub repository.
- Create a google docs
- If incident is an outage SL updates the Login.gov Statuspage via the Statuspage Admin Interface (View Sample Message)
- Login.gov Agency Partners: send out an incident summary to LOGIN-PARTNERS@listserv.gsa.gov. Partner list: https://drive.google.com/drive/u/0/folders/0B4yIa0Upv1JJSkJOSmdsLWVOVmM)
The SL is responsible for keeping this issue up-to-date as investigation and remediation progresses. Everyone involved in the issue (responders) should leave notes as comments on the issue.
- The SL may start a Google Hangout and/or create Google Docs so that responders can share sensitive information not suitable for sharing in GitHub or Slack.
The next step is to assess the issue. We need to answer two questions:
- Is this an incident (i.e., did the thing we suspect happen actually happen?)
- If so, how severe is it? (This will determine how our response proceeds.)
To answer these questions, the SL should form a response team by Direct Messaging (DM) people in Slack. The response team should work to confirm the issue and assess its impact.
If the issue turns out to be a false alarm, the SL should update the ticket, setting the status to “false alarm”, and closing the issue.
If the issue is valid, the team should assess its impact and determine an initial severity following the incident severity guide below. (Note that the severity can change over the lifespan of the incident, so it’s OK to determine the initial severity fairly quickly.)
Once this is done, the SL should update the ticket, noting:
- ` ** Status: “confirmed” `
- ` ** Severity: High/Med/Low `
- ` ** Any new/changed responders `
At this point, the SL should write an initial situation report (“sitrep”) confirming the incident, summarizing what’s going on, identifying the SL, and linking to the issue. Here’s an example sitrep:
Subject: [sitrep] The chickens have escaped
- ` ** Severity: high `
- ` ** SL: Farmer Jane `
- ` ** Responders: Spot the Dog, Farmer Dave `
We’ve confirmed reports of escaped chickens. Looks like a fox may have tunneled into the run. Dave is working to fix the fence, Spot is tracking the fox.
This sitrep should be:
- Posted in #login-situation
- Emailed to GSA IR (email@example.com)
- Send (email or Slack) to external stakeholders, if applicable and relevant
Comms at the Assess phase
Updates and real-time chat should continue as above (updates on the GitHub issue, chat in Slack or Google Hangouts, and update to open Statuspage incident if applicable).
At this point, we’re trying to fix the issue! Remediation will be very situation-specific, so specific steps are hard to suggest. However, a few guidelines to follow during this process:
The SL’s responsibility is coordination, communication, and information-collection. The remediation team will be focused on resolving the issue, so it’s up to the SL to make sure that we properly track what happened, how we’re fixing it, who’s doing what, etc. Ideally, the notes kept by the SL should be sufficient for an outside investigator to independently follow the work of the response team and validate the team’s work.
Similarly, the team will develop a list of remediation steps. The SL is likewise responsible for tracking those, making sure they’re assigned and followed-up, and verifying them as they’re completed. These may be tracked in the central GitHub issue as well. The SL should distinguish between immediate concerns which should be completed before the incident is considered resolved and long-term improvements/hardening which can be deferred to the Retrospective.
The response team should aim to adopt a containment strategy: if machines are compromised, they should avoid destroying or shutting them down if possible (this can hamper forensics). For AWS instances, you can leave the instance running and instead reconfigure the Security Group for the instance to drop all ingress and egress traffic until forensics can be performed. (GSA IR team May be involved as well at this stage).
Remediation may require service disruption. If it does, the team should proceed in a different way depending on the severity:
For High-severity incidents, the team should take action immediately, even if this causes disruption. A notification about the disruption should be sent out as soon as possible, but the team needs no permission to take action at this level.
For Medium-severity incidents, the team should notify the Login.gov leads of the planned action, and help them assess the relative risk of disruption vs. security. If the leads are unavailable via Slack, they should be contacted using the phone numbers in their Slack profiles. The team should reach a collaborative decision on action, with a bias towards disruption. If they can’t be reached within 1 hour, the team may take action without them.
For Low-severity issues, the team should notify as above, and not take action until a mutually-agreed-on course of action has been determined.
Remediation can sometimes take a long time. If the issue progresses for more than 3 hours without being resolved, the SL should plan for a long remediation.
Determine if remediation efforts should be “business hours” or “24⁄7.” This will depend on the severity of the issue, and whether breaches are ongoing.
For 24⁄7 responses, the SL should begin shift-planning. Generally, responders should only work 3 hour shifts, so the SL should begin scheduling shifts and sending people “home” to preserve their ability to function.
The SL also needs to get into rotation — again, 3 hour shifts are about the maximum suggested. So, the SL should likely hand off duties at this point.
Once the incident is no longer active — i.e. the breach has been contained, the issue has been fixed, etc. — the SL should spin down the incident. There may still be longer-term remediation needed, and possibly more investigation, but as long as the incident is no longer active these activities can proceed at the regular pace of business. To close out an incident, the SL should:
- Set the status of the incident to “resolved.”
- Send a final sitrep to stakeholders.
- Thank everyone involved for their service!
Comms at the Remediate phase
Updates and real-time chat should continue as above (updates on the GitHub issue, chat in Slack or Google Hangouts).
The SL should continue to post updated sitreps on a regular cadence (the section on severities, below, suggests cadences for each level). These sitreps should be sent to Slack, to GSA-IT and US-CERT via email, and to any other stakeholders identified throughout the process (e.g. clients).
The final step in handling a security incident is figuring out what we learned. The SL (or one of the SLs if there were multiple, or a designated other party) should lead a retrospective and develop an incident report.
The report should contain a timeline of the incident, details about how the incident progressed, and information about the vulnerabilities that led to the incident. A cause analysis is an important part of this report; the team should use tools such as Infinite Hows and Five Whys to try to dig into causes, how future incidents could be prevented, how responses could be better in the future, etc.
The report should also contain some basic response metrics:
Discovery method (how did we become aware of the issue?)
- Time to discovery (how long did it take from when the incident started until we became aware of it?)
- Time to containment (how long did it take from when we became aware until the issue was contained?)
- Threat actions (which specific actions – e.g. phishing, password attacks, etc) – were taken by the actor)?
This report should be posted as a final comment on the GitHub issue, which can then be closed. If appropriate, this should also be posted at postmortems (omitting any sensitive information).
Severity ratings drive the actions of the response team. Below are the severities ratings we use, some examples of incidents that might fall into that bucket, and some guidelines for SLs and response teams about how to treat each class of incident.
Note the severities may (and often will) change during the lifecycle of the incident. That’s normal.
High-sev incidents successfully compromise the confidentiality/integrity of Personally Identifiable Information (PII), impact the availability of services for a large number of customers, or have significant financial impact. Examples include:
Confirmed breach of PII
Successful root-level compromise of production systems
Financial malware (ie. CryptoLocker) targeting 18F staff
Denial of Service attacks resulting in severe outages
Guidelines for addressing High-sev issues:
Work will likely be 24⁄7 (e.g. work until the issue is contained).
Responders are empowered to take any step needed to contain the issue, up to and including complete service degradation.
Sitreps should be sent every hour, or more.
Medium-sev incidents represent attempts (possibly un- or not-yet-successful) at breaching PII, or those with limited availability/financial impact. Examples include:
Suspected PII breach
Targeted (but as-of-yet) attempts to compromise production systems
Spam/phishing attacks targeting 18F staff
DoS attacks resulting in limited service degradation
Guidelines for addressing Medium-sev issues:
Response should be business-hours.
Responders should attempt to consult stakeholders before causing downtime, but may proceed without them if they can’t be contacted in a reasonable time-frame.
Sitreps should be sent approximately twice a day.
Low-sev incidents don’t affect PII, and have no availability or financial impact. Examples include:
Attempted compromise of non-important systems (staging/dm/qa/demo, etc.)
Incidents involving specific employees
DoS attacks with no noticeable customer impact
Guidelines for addressing Low-sev issues:
Response should be business-hours.
Responders should avoid service degradation unless stakeholders agree.
Sitreps should be sent approximately daily.
Frequently Asked Questions and Expectations
Who does the Incident Response Guide apply to?
Anyone with access to production, including:
- All SREs
- All SecOps
- All developers
- All partnerships technical resources
- Representatives from the following teams:
- User Support
What is an incident?
- Anything that presents an immediate risk to Confidentiality, Integrity, or Availability of public Login.gov services
- If it smells like an incident, declare an incident
- False positives are normal and useful to keep us sharp
- Long delays while deciding if something is an incident are not acceptable
- When in doubt, call it out! (If someone asks if it is an incident, it is probably an incident)
When do you declare an incident?
- Failing to report an incident in a timely manner after discovery jeopardizes Login.gov - FISMA requires timely reporting and declaring an incident and following our policy ensures we do not fail to meet that requirement
When do you engage GSA-IR?
- Always - (Max of 60 minutes) GSA Incident Response is here to support our mission and protect our users, our agency, our program, and you!
- Even if it is low priority/no big deal? - That is not a determination that can be made alone, and GSA-IR will constructively challenge assumptions to ensure it is indeed no big deal
How should I prioritize assisting with an incident vs. my other work?
- If you are on call (primary or secondary) you are expected to join situations immediately
- Incidents are always highest priority! If you are asked to join, please check in
- Development, planning, and meeting activities should be paused during participation in an incident
- When the Situation Lead acknowledges you are not needed for incident response activities you can resume normal work
- Engineering and Platform leads are happy to answer questions about prioritization and your need to drop everything and participate
When should I report into the situation room?
- Business hours:
- The current AppDev Primary and Secondary on-call
- The current DevOps Primary and Secondary on-call
- Additional Responders as needed - The Situation Lead can call in anyone defined in Who does the Incident Response Guide apply to?
- After hours - DevOps On Call and Application On Call - Additional resources will be brought in via OpsGenie notification
What is expected when participating?
- Situation Lead (SL) - Once declared, leads the process until the incident is resolved OR Situation Lead role is handed off
- Ensures process is followed
- Stays in situation room
- Delegates activities - Does NOT act as Technical Lead
- Ensures communication and mitigation are being addressed by others
- Makes sure everything that needs doing is being done
- Checks in with participants for status updates
- Makes sure that people have something to do, and if not, communicates clearly that they can drop
- Provide clear updates to the team
- If unfilled or someone needs to rotate out, volunteers to serve in one of the defined roles:
- Technical lead (TL): Leads technical investigation and mitigation
- Comms lead (CL): Coordinates communication outside of #login-situation, within GSA, and if needed, with partners and the public
- Scribe (S): Relays information discussed in war room (hangout) to #login-situation and aids SL in recording incident (Just notes, not a transcript)
- If assigned a specific role or task, sees it through until it is complete OR handed off to another participant and confirmed by the SL
- Be ready to answer SL/others when questions arise
- Scribe your activities and artifacts in #login-situation
What if I can’t participate?
- We are humans, it is OK! All the same reasons for not working at a given time apply to incidents.
- If asked to join, let the Situation Lead know you cannot at this time.
- If you are participating but need to drop off:
- Ensure any role or activity you are responsible for has been handed off to another participant
- Let the Situation Lead know you are dropping off
- Wait for confirmation from the Situation Lead before dropping
Why do I need to participate in contingency planning exercises (wargames) and IR Fire Drills?
- Resources - It takes a full team to respond, and every participant is important!
- Readiness - Only through practice will we be and remain ready to meet the challenges Login.gov faces
- Refinement - Only through participating fully can we refine our processes to improve response and efficiency
- Required - GSA authorizes Login.gov to operate, in part on the understanding that we will adhere to GSA IT - IT Security Procedural Guide: Incident Response
- Regulation (Law) - Per FISMA we must follow NIST 800-61r2 - Computer Security Incident Handling Guide