Security Incident Response

Comprehensive guide to handling security incidents and cyber threats
Security Advanced 35 min read

Overview

This comprehensive guide covers essential security incident response procedures, from initial detection to post-incident analysis. Learn how to establish incident response teams, implement detection systems, and follow industry-standard frameworks like NIST and SANS for effective cybersecurity incident management.

Quick Reference

  • Incident Types: Malware, DDoS, data breaches, insider threats
  • Response Phases: Preparation, identification, containment, eradication, recovery
  • Tools: SIEM, EDR, forensics tools, communication systems
  • Frameworks: NIST, SANS, ISO 27035

1. Incident Response Fundamentals

1.1 Incident Response Lifecycle

Understanding the six phases of incident response according to NIST framework.

NIST Incident Response Phases:

  • Preparation: Establish policies, procedures, and capabilities
  • Detection and Analysis: Identify and analyze security incidents
  • Containment: Limit the scope and impact of incidents
  • Eradication: Remove threats and vulnerabilities
  • Recovery: Restore systems and services
  • Lessons Learned: Document and improve response capabilities

1.2 Incident Classification

Classifying security incidents by severity and impact.

Incident Severity Levels:

  • Critical (P1): Active data breach, system compromise, service outage
  • High (P2): Potential data exposure, significant security risk
  • Medium (P3): Security policy violations, suspicious activity
  • Low (P4): Minor security events, false positives

2. Incident Response Team Structure

2.1 Team Roles and Responsibilities

Defining roles and responsibilities for effective incident response.

Core Team Roles:

  • Incident Commander: Overall coordination and decision making
  • Security Analyst: Technical analysis and investigation
  • Forensics Specialist: Evidence collection and analysis
  • Communications Lead: Internal and external communications
  • Legal Counsel: Legal and regulatory compliance
  • IT Operations: System restoration and recovery

2.2 Escalation Procedures

Establishing clear escalation procedures for different incident types.

Escalation Matrix:

# Escalation procedures
P1 (Critical):
- Immediate notification to Incident Commander
- Escalate to CISO within 15 minutes
- Escalate to CEO within 30 minutes
- External notification within 1 hour

P2 (High):
- Notify Incident Commander within 30 minutes
- Escalate to CISO within 1 hour
- Escalate to CEO within 2 hours

P3 (Medium):
- Notify Security Team within 1 hour
- Escalate to Incident Commander within 4 hours

P4 (Low):
- Log and monitor
- Escalate if pattern emerges

3. Detection and Analysis

3.1 Threat Detection Systems

Implementing comprehensive threat detection capabilities.

Detection Tools and Techniques:

  • SIEM (Security Information and Event Management): Centralized log analysis
  • EDR (Endpoint Detection and Response): Endpoint monitoring and response
  • Network Monitoring: Traffic analysis and anomaly detection
  • Threat Intelligence: External threat feeds and indicators
  • User Behavior Analytics: Insider threat detection

3.2 Incident Analysis Procedures

Systematic approach to analyzing security incidents.

Analysis Checklist:

  • ✓ Gather initial information and context
  • ✓ Identify affected systems and data
  • ✓ Determine attack vector and methods
  • ✓ Assess scope and impact
  • ✓ Collect and preserve evidence
  • ✓ Document findings and timeline

Incident Analysis Script:

#!/bin/bash
# incident_analysis.sh

INCIDENT_ID="$1"
ANALYST="$2"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Create incident directory
mkdir -p /incidents/$INCIDENT_ID
cd /incidents/$INCIDENT_ID

# Log initial information
echo "Incident ID: $INCIDENT_ID" > incident_log.txt
echo "Analyst: $ANALYST" >> incident_log.txt
echo "Timestamp: $TIMESTAMP" >> incident_log.txt
echo "Status: Under Investigation" >> incident_log.txt

# Collect system information
uname -a > system_info.txt
ps aux > running_processes.txt
netstat -tulpn > network_connections.txt
ss -tulpn >> network_connections.txt

# Collect log files
cp /var/log/auth.log auth_log.txt
cp /var/log/syslog syslog.txt
cp /var/log/messages messages.txt

# Collect network information
ip addr show > network_interfaces.txt
route -n > routing_table.txt
arp -a > arp_table.txt

# Collect file system information
find /tmp -type f -mtime -1 > recent_tmp_files.txt
find /var/log -name "*.log" -mtime -1 > recent_logs.txt

# Create analysis report
echo "Initial Analysis Report" > analysis_report.txt
echo "======================" >> analysis_report.txt
echo "Incident ID: $INCIDENT_ID" >> analysis_report.txt
echo "Analysis Date: $(date)" >> analysis_report.txt
echo "Analyst: $ANALYST" >> analysis_report.txt
echo "" >> analysis_report.txt
echo "Initial Findings:" >> analysis_report.txt
echo "- System compromised: [TBD]" >> analysis_report.txt
echo "- Attack vector: [TBD]" >> analysis_report.txt
echo "- Data affected: [TBD]" >> analysis_report.txt
echo "- Impact level: [TBD]" >> analysis_report.txt

echo "Initial analysis completed for incident $INCIDENT_ID"

4. Containment Strategies

4.1 Immediate Containment

Rapid containment measures to limit incident impact.

Containment Actions:

  • Network Isolation: Disconnect affected systems from network
  • Account Lockout: Disable compromised user accounts
  • Service Shutdown: Stop affected services and applications
  • Access Revocation: Revoke compromised access credentials
  • System Quarantine: Isolate affected systems for analysis

4.2 Long-term Containment

Sustained containment measures while maintaining business operations.

Containment Script:

#!/bin/bash
# containment_actions.sh

INCIDENT_ID="$1"
AFFECTED_SYSTEM="$2"
ACTION="$3"

case $ACTION in
    "isolate")
        # Isolate system from network
        iptables -A INPUT -s $AFFECTED_SYSTEM -j DROP
        iptables -A OUTPUT -d $AFFECTED_SYSTEM -j DROP
        echo "System $AFFECTED_SYSTEM isolated from network"
        ;;
    "lockout")
        # Lock out user accounts
        usermod -L $AFFECTED_SYSTEM
        echo "User account $AFFECTED_SYSTEM locked"
        ;;
    "shutdown")
        # Shutdown affected services
        systemctl stop $AFFECTED_SYSTEM
        systemctl disable $AFFECTED_SYSTEM
        echo "Service $AFFECTED_SYSTEM shutdown"
        ;;
    "quarantine")
        # Quarantine system
        systemctl stop networking
        echo "System quarantined - networking disabled"
        ;;
    "revoke")
        # Revoke access credentials
        # Remove from sudoers
        sed -i "/$AFFECTED_SYSTEM/d" /etc/sudoers
        # Remove SSH keys
        rm -f /home/$AFFECTED_SYSTEM/.ssh/authorized_keys
        echo "Access credentials revoked for $AFFECTED_SYSTEM"
        ;;
esac

# Log containment action
echo "$(date): $ACTION performed on $AFFECTED_SYSTEM for incident $INCIDENT_ID" >> /incidents/$INCIDENT_ID/containment_log.txt

5. Eradication and Recovery

5.1 Threat Eradication

Removing threats and vulnerabilities from affected systems.

Eradication Steps:

  • Malware Removal: Identify and remove malicious software
  • Vulnerability Patching: Apply security patches and updates
  • Configuration Hardening: Strengthen security configurations
  • Access Review: Review and update access controls
  • System Rebuild: Rebuild compromised systems from clean images

5.2 System Recovery

Restoring systems and services to normal operations.

Recovery Checklist:

  • ✓ Verify threat eradication is complete
  • ✓ Apply all necessary security patches
  • ✓ Restore systems from clean backups
  • ✓ Update security configurations
  • ✓ Test system functionality
  • ✓ Monitor for signs of re-infection
  • ✓ Gradually restore network connectivity
  • ✓ Validate business processes

Recovery Script:

#!/bin/bash
# system_recovery.sh

INCIDENT_ID="$1"
SYSTEM_NAME="$2"

echo "Starting recovery process for $SYSTEM_NAME (Incident: $INCIDENT_ID)"

# 1. Verify system is clean
echo "Verifying system is clean..."
clamscan -r / --infected --remove
rkhunter --check --skip-keypress

# 2. Apply security updates
echo "Applying security updates..."
apt update
apt upgrade -y
apt autoremove -y

# 3. Restore from backup
echo "Restoring from clean backup..."
# This would be specific to your backup solution
# Example: restore from clean snapshot

# 4. Update security configurations
echo "Updating security configurations..."
# Harden SSH
sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart ssh

# 5. Enable monitoring
echo "Enabling enhanced monitoring..."
# Enable additional logging
echo "*.info /var/log/security.log" >> /etc/rsyslog.conf
systemctl restart rsyslog

# 6. Test system functionality
echo "Testing system functionality..."
systemctl status
netstat -tulpn
ps aux

echo "Recovery process completed for $SYSTEM_NAME"

6. Communication and Reporting

6.1 Internal Communication

Managing internal communications during security incidents.

Communication Templates:

# Internal Incident Notification Template
Subject: Security Incident Alert - [INCIDENT_ID] - [SEVERITY]

Incident Summary:
- Incident ID: [INCIDENT_ID]
- Severity: [SEVERITY]
- Detection Time: [TIMESTAMP]
- Affected Systems: [SYSTEMS]
- Initial Assessment: [ASSESSMENT]

Immediate Actions Taken:
- [ACTION_1]
- [ACTION_2]
- [ACTION_3]

Next Steps:
- [NEXT_STEP_1]
- [NEXT_STEP_2]

Contact Information:
- Incident Commander: [NAME] - [PHONE] - [EMAIL]
- Security Team: [PHONE] - [EMAIL]

This is an automated notification. Please do not reply to this email.

6.2 External Reporting

Managing external communications and regulatory reporting.

External Notification Checklist:

  • ✓ Legal team consultation
  • ✓ Regulatory compliance review
  • ✓ Customer notification preparation
  • ✓ Media response strategy
  • ✓ Law enforcement notification (if required)
  • ✓ Insurance company notification

7. Post-Incident Analysis

7.1 Lessons Learned Process

Conducting thorough post-incident analysis and improvement planning.

Post-Incident Review Questions:

  • What was the root cause of the incident?
  • How was the incident detected?
  • What containment measures were effective?
  • What could have been done better?
  • What preventive measures should be implemented?
  • How can response procedures be improved?

7.2 Improvement Planning

Developing and implementing improvements based on incident lessons learned.

Improvement Action Items:

  • Technical Improvements: Enhanced monitoring, better detection tools
  • Process Improvements: Updated procedures, better training
  • Policy Updates: Revised security policies and guidelines
  • Training Programs: Enhanced security awareness training
  • Tool Upgrades: New security tools and technologies

Post-Incident Report Template:

# Post-Incident Report Template

## Executive Summary
- Incident ID: [ID]
- Date Range: [START] to [END]
- Severity: [SEVERITY]
- Impact: [IMPACT]
- Root Cause: [ROOT_CAUSE]

## Incident Timeline
- Detection: [TIME]
- Response: [TIME]
- Containment: [TIME]
- Eradication: [TIME]
- Recovery: [TIME]

## Technical Details
- Attack Vector: [VECTOR]
- Affected Systems: [SYSTEMS]
- Data Compromised: [DATA]
- Tools Used: [TOOLS]

## Response Actions
- Immediate Actions: [ACTIONS]
- Containment Measures: [MEASURES]
- Recovery Steps: [STEPS]

## Lessons Learned
- What Went Well: [POSITIVES]
- What Could Be Improved: [IMPROVEMENTS]
- Root Cause Analysis: [ANALYSIS]

## Recommendations
- Short-term (0-30 days): [SHORT_TERM]
- Medium-term (1-6 months): [MEDIUM_TERM]
- Long-term (6+ months): [LONG_TERM]

## Action Items
- [ ] [ACTION_1] - [OWNER] - [DUE_DATE]
- [ ] [ACTION_2] - [OWNER] - [DUE_DATE]
- [ ] [ACTION_3] - [OWNER] - [DUE_DATE]

8. Incident Response Tools

8.1 SIEM Configuration

Setting up SIEM for effective incident detection and response.

SIEM Alert Rules:

# Splunk SIEM Alert Rules
# Failed login attempts
index=security sourcetype=auth_logs "Failed password" | 
stats count by src_ip, user | 
where count > 5 | 
eval severity=if(count>10, "high", "medium")

# Privilege escalation
index=security sourcetype=auth_logs "sudo" | 
search user=* | 
stats count by src_ip, user, command | 
where count > 3

# Network anomalies
index=network sourcetype=firewall_logs | 
stats sum(bytes) by src_ip, dest_ip | 
where bytes > 1000000000 | 
eval severity="high"

# File integrity monitoring
index=security sourcetype=file_monitor | 
search action=modified | 
stats count by file_path, user | 
where count > 1

8.2 Forensics Tools

Essential tools for digital forensics and incident analysis.

Forensics Tool Setup:

#!/bin/bash
# forensics_tools_install.sh

# Install Volatility (memory forensics)
pip3 install volatility3

# Install Autopsy (disk forensics)
wget https://github.com/sleuthkit/autopsy/releases/download/autopsy-4.19.0/autopsy-4.19.0_64bit.run
chmod +x autopsy-4.19.0_64bit.run
./autopsy-4.19.0_64bit.run

# Install Wireshark (network forensics)
apt install -y wireshark

# Install Sleuth Kit
apt install -y sleuthkit

# Install YARA (malware detection)
apt install -y yara

# Install ClamAV (antivirus)
apt install -y clamav clamav-daemon

# Install rkhunter (rootkit detection)
apt install -y rkhunter

# Install chkrootkit (rootkit detection)
apt install -y chkrootkit

Download the Complete Guide

Get the full PDF version with additional incident response procedures, templates, and tool configurations.

Download PDF