Security Incident Response
Overview
This comprehensive guide covers essential security incident response procedures, from initial detection to post-incident analysis. Learn how to establish incident response teams, implement detection systems, and follow industry-standard frameworks like NIST and SANS for effective cybersecurity incident management.
Quick Reference
- Incident Types: Malware, DDoS, data breaches, insider threats
- Response Phases: Preparation, identification, containment, eradication, recovery
- Tools: SIEM, EDR, forensics tools, communication systems
- Frameworks: NIST, SANS, ISO 27035
1. Incident Response Fundamentals
1.1 Incident Response Lifecycle
Understanding the six phases of incident response according to NIST framework.
NIST Incident Response Phases:
- Preparation: Establish policies, procedures, and capabilities
- Detection and Analysis: Identify and analyze security incidents
- Containment: Limit the scope and impact of incidents
- Eradication: Remove threats and vulnerabilities
- Recovery: Restore systems and services
- Lessons Learned: Document and improve response capabilities
1.2 Incident Classification
Classifying security incidents by severity and impact.
Incident Severity Levels:
- Critical (P1): Active data breach, system compromise, service outage
- High (P2): Potential data exposure, significant security risk
- Medium (P3): Security policy violations, suspicious activity
- Low (P4): Minor security events, false positives
2. Incident Response Team Structure
2.1 Team Roles and Responsibilities
Defining roles and responsibilities for effective incident response.
Core Team Roles:
- Incident Commander: Overall coordination and decision making
- Security Analyst: Technical analysis and investigation
- Forensics Specialist: Evidence collection and analysis
- Communications Lead: Internal and external communications
- Legal Counsel: Legal and regulatory compliance
- IT Operations: System restoration and recovery
2.2 Escalation Procedures
Establishing clear escalation procedures for different incident types.
Escalation Matrix:
# Escalation procedures
P1 (Critical):
- Immediate notification to Incident Commander
- Escalate to CISO within 15 minutes
- Escalate to CEO within 30 minutes
- External notification within 1 hour
P2 (High):
- Notify Incident Commander within 30 minutes
- Escalate to CISO within 1 hour
- Escalate to CEO within 2 hours
P3 (Medium):
- Notify Security Team within 1 hour
- Escalate to Incident Commander within 4 hours
P4 (Low):
- Log and monitor
- Escalate if pattern emerges
3. Detection and Analysis
3.1 Threat Detection Systems
Implementing comprehensive threat detection capabilities.
Detection Tools and Techniques:
- SIEM (Security Information and Event Management): Centralized log analysis
- EDR (Endpoint Detection and Response): Endpoint monitoring and response
- Network Monitoring: Traffic analysis and anomaly detection
- Threat Intelligence: External threat feeds and indicators
- User Behavior Analytics: Insider threat detection
3.2 Incident Analysis Procedures
Systematic approach to analyzing security incidents.
Analysis Checklist:
- ✓ Gather initial information and context
- ✓ Identify affected systems and data
- ✓ Determine attack vector and methods
- ✓ Assess scope and impact
- ✓ Collect and preserve evidence
- ✓ Document findings and timeline
Incident Analysis Script:
#!/bin/bash
# incident_analysis.sh
INCIDENT_ID="$1"
ANALYST="$2"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
# Create incident directory
mkdir -p /incidents/$INCIDENT_ID
cd /incidents/$INCIDENT_ID
# Log initial information
echo "Incident ID: $INCIDENT_ID" > incident_log.txt
echo "Analyst: $ANALYST" >> incident_log.txt
echo "Timestamp: $TIMESTAMP" >> incident_log.txt
echo "Status: Under Investigation" >> incident_log.txt
# Collect system information
uname -a > system_info.txt
ps aux > running_processes.txt
netstat -tulpn > network_connections.txt
ss -tulpn >> network_connections.txt
# Collect log files
cp /var/log/auth.log auth_log.txt
cp /var/log/syslog syslog.txt
cp /var/log/messages messages.txt
# Collect network information
ip addr show > network_interfaces.txt
route -n > routing_table.txt
arp -a > arp_table.txt
# Collect file system information
find /tmp -type f -mtime -1 > recent_tmp_files.txt
find /var/log -name "*.log" -mtime -1 > recent_logs.txt
# Create analysis report
echo "Initial Analysis Report" > analysis_report.txt
echo "======================" >> analysis_report.txt
echo "Incident ID: $INCIDENT_ID" >> analysis_report.txt
echo "Analysis Date: $(date)" >> analysis_report.txt
echo "Analyst: $ANALYST" >> analysis_report.txt
echo "" >> analysis_report.txt
echo "Initial Findings:" >> analysis_report.txt
echo "- System compromised: [TBD]" >> analysis_report.txt
echo "- Attack vector: [TBD]" >> analysis_report.txt
echo "- Data affected: [TBD]" >> analysis_report.txt
echo "- Impact level: [TBD]" >> analysis_report.txt
echo "Initial analysis completed for incident $INCIDENT_ID"
4. Containment Strategies
4.1 Immediate Containment
Rapid containment measures to limit incident impact.
Containment Actions:
- Network Isolation: Disconnect affected systems from network
- Account Lockout: Disable compromised user accounts
- Service Shutdown: Stop affected services and applications
- Access Revocation: Revoke compromised access credentials
- System Quarantine: Isolate affected systems for analysis
4.2 Long-term Containment
Sustained containment measures while maintaining business operations.
Containment Script:
#!/bin/bash
# containment_actions.sh
INCIDENT_ID="$1"
AFFECTED_SYSTEM="$2"
ACTION="$3"
case $ACTION in
"isolate")
# Isolate system from network
iptables -A INPUT -s $AFFECTED_SYSTEM -j DROP
iptables -A OUTPUT -d $AFFECTED_SYSTEM -j DROP
echo "System $AFFECTED_SYSTEM isolated from network"
;;
"lockout")
# Lock out user accounts
usermod -L $AFFECTED_SYSTEM
echo "User account $AFFECTED_SYSTEM locked"
;;
"shutdown")
# Shutdown affected services
systemctl stop $AFFECTED_SYSTEM
systemctl disable $AFFECTED_SYSTEM
echo "Service $AFFECTED_SYSTEM shutdown"
;;
"quarantine")
# Quarantine system
systemctl stop networking
echo "System quarantined - networking disabled"
;;
"revoke")
# Revoke access credentials
# Remove from sudoers
sed -i "/$AFFECTED_SYSTEM/d" /etc/sudoers
# Remove SSH keys
rm -f /home/$AFFECTED_SYSTEM/.ssh/authorized_keys
echo "Access credentials revoked for $AFFECTED_SYSTEM"
;;
esac
# Log containment action
echo "$(date): $ACTION performed on $AFFECTED_SYSTEM for incident $INCIDENT_ID" >> /incidents/$INCIDENT_ID/containment_log.txt
5. Eradication and Recovery
5.1 Threat Eradication
Removing threats and vulnerabilities from affected systems.
Eradication Steps:
- Malware Removal: Identify and remove malicious software
- Vulnerability Patching: Apply security patches and updates
- Configuration Hardening: Strengthen security configurations
- Access Review: Review and update access controls
- System Rebuild: Rebuild compromised systems from clean images
5.2 System Recovery
Restoring systems and services to normal operations.
Recovery Checklist:
- ✓ Verify threat eradication is complete
- ✓ Apply all necessary security patches
- ✓ Restore systems from clean backups
- ✓ Update security configurations
- ✓ Test system functionality
- ✓ Monitor for signs of re-infection
- ✓ Gradually restore network connectivity
- ✓ Validate business processes
Recovery Script:
#!/bin/bash
# system_recovery.sh
INCIDENT_ID="$1"
SYSTEM_NAME="$2"
echo "Starting recovery process for $SYSTEM_NAME (Incident: $INCIDENT_ID)"
# 1. Verify system is clean
echo "Verifying system is clean..."
clamscan -r / --infected --remove
rkhunter --check --skip-keypress
# 2. Apply security updates
echo "Applying security updates..."
apt update
apt upgrade -y
apt autoremove -y
# 3. Restore from backup
echo "Restoring from clean backup..."
# This would be specific to your backup solution
# Example: restore from clean snapshot
# 4. Update security configurations
echo "Updating security configurations..."
# Harden SSH
sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart ssh
# 5. Enable monitoring
echo "Enabling enhanced monitoring..."
# Enable additional logging
echo "*.info /var/log/security.log" >> /etc/rsyslog.conf
systemctl restart rsyslog
# 6. Test system functionality
echo "Testing system functionality..."
systemctl status
netstat -tulpn
ps aux
echo "Recovery process completed for $SYSTEM_NAME"
6. Communication and Reporting
6.1 Internal Communication
Managing internal communications during security incidents.
Communication Templates:
# Internal Incident Notification Template
Subject: Security Incident Alert - [INCIDENT_ID] - [SEVERITY]
Incident Summary:
- Incident ID: [INCIDENT_ID]
- Severity: [SEVERITY]
- Detection Time: [TIMESTAMP]
- Affected Systems: [SYSTEMS]
- Initial Assessment: [ASSESSMENT]
Immediate Actions Taken:
- [ACTION_1]
- [ACTION_2]
- [ACTION_3]
Next Steps:
- [NEXT_STEP_1]
- [NEXT_STEP_2]
Contact Information:
- Incident Commander: [NAME] - [PHONE] - [EMAIL]
- Security Team: [PHONE] - [EMAIL]
This is an automated notification. Please do not reply to this email.
6.2 External Reporting
Managing external communications and regulatory reporting.
External Notification Checklist:
- ✓ Legal team consultation
- ✓ Regulatory compliance review
- ✓ Customer notification preparation
- ✓ Media response strategy
- ✓ Law enforcement notification (if required)
- ✓ Insurance company notification
7. Post-Incident Analysis
7.1 Lessons Learned Process
Conducting thorough post-incident analysis and improvement planning.
Post-Incident Review Questions:
- What was the root cause of the incident?
- How was the incident detected?
- What containment measures were effective?
- What could have been done better?
- What preventive measures should be implemented?
- How can response procedures be improved?
7.2 Improvement Planning
Developing and implementing improvements based on incident lessons learned.
Improvement Action Items:
- Technical Improvements: Enhanced monitoring, better detection tools
- Process Improvements: Updated procedures, better training
- Policy Updates: Revised security policies and guidelines
- Training Programs: Enhanced security awareness training
- Tool Upgrades: New security tools and technologies
Post-Incident Report Template:
# Post-Incident Report Template
## Executive Summary
- Incident ID: [ID]
- Date Range: [START] to [END]
- Severity: [SEVERITY]
- Impact: [IMPACT]
- Root Cause: [ROOT_CAUSE]
## Incident Timeline
- Detection: [TIME]
- Response: [TIME]
- Containment: [TIME]
- Eradication: [TIME]
- Recovery: [TIME]
## Technical Details
- Attack Vector: [VECTOR]
- Affected Systems: [SYSTEMS]
- Data Compromised: [DATA]
- Tools Used: [TOOLS]
## Response Actions
- Immediate Actions: [ACTIONS]
- Containment Measures: [MEASURES]
- Recovery Steps: [STEPS]
## Lessons Learned
- What Went Well: [POSITIVES]
- What Could Be Improved: [IMPROVEMENTS]
- Root Cause Analysis: [ANALYSIS]
## Recommendations
- Short-term (0-30 days): [SHORT_TERM]
- Medium-term (1-6 months): [MEDIUM_TERM]
- Long-term (6+ months): [LONG_TERM]
## Action Items
- [ ] [ACTION_1] - [OWNER] - [DUE_DATE]
- [ ] [ACTION_2] - [OWNER] - [DUE_DATE]
- [ ] [ACTION_3] - [OWNER] - [DUE_DATE]
8. Incident Response Tools
8.1 SIEM Configuration
Setting up SIEM for effective incident detection and response.
SIEM Alert Rules:
# Splunk SIEM Alert Rules
# Failed login attempts
index=security sourcetype=auth_logs "Failed password" |
stats count by src_ip, user |
where count > 5 |
eval severity=if(count>10, "high", "medium")
# Privilege escalation
index=security sourcetype=auth_logs "sudo" |
search user=* |
stats count by src_ip, user, command |
where count > 3
# Network anomalies
index=network sourcetype=firewall_logs |
stats sum(bytes) by src_ip, dest_ip |
where bytes > 1000000000 |
eval severity="high"
# File integrity monitoring
index=security sourcetype=file_monitor |
search action=modified |
stats count by file_path, user |
where count > 1
8.2 Forensics Tools
Essential tools for digital forensics and incident analysis.
Forensics Tool Setup:
#!/bin/bash
# forensics_tools_install.sh
# Install Volatility (memory forensics)
pip3 install volatility3
# Install Autopsy (disk forensics)
wget https://github.com/sleuthkit/autopsy/releases/download/autopsy-4.19.0/autopsy-4.19.0_64bit.run
chmod +x autopsy-4.19.0_64bit.run
./autopsy-4.19.0_64bit.run
# Install Wireshark (network forensics)
apt install -y wireshark
# Install Sleuth Kit
apt install -y sleuthkit
# Install YARA (malware detection)
apt install -y yara
# Install ClamAV (antivirus)
apt install -y clamav clamav-daemon
# Install rkhunter (rootkit detection)
apt install -y rkhunter
# Install chkrootkit (rootkit detection)
apt install -y chkrootkit
Download the Complete Guide
Get the full PDF version with additional incident response procedures, templates, and tool configurations.
Download PDF