Debugging Kubernetes Incidents Skill

A comprehensive skill for systematic investigation and root cause analysis of Kubernetes pod failures, service degradation, and resource issues.

What This Skill Does

This skill teaches Claude Code to:

• Systematically investigate Kubernetes incidents using a structured 5-phase methodology
• Correlate multiple data sources (logs, events, metrics) for complete incident picture
• Identify root causes by distinguishing correlation from causation
• Provide actionable recommendations with clear immediate and permanent fixes
• Follow SRE best practices for incident triage and resolution

When to Use

Invoke this skill when dealing with:

• Pod crashes and CrashLoopBackOff states
• Resource exhaustion (OOMKilled, CPU throttling)
• Image pull failures and container startup issues
• Service degradation and connectivity problems
• Certificate errors and TLS handshake failures
• Deployment rollout failures

Key Features

Structured Investigation Methodology

The skill provides a 5-phase approach:

1. Triage - Assess severity and scope
2. Data Collection - Gather logs, events, metrics
3. Correlation - Create unified timeline
4. Root Cause - Identify underlying issue
5. Remediation - Plan fixes and prevention

Common Issue Patterns

Pre-built investigation patterns for:

• Image pull errors
• Resource constraints
• Application crashes
• Memory leaks (OOMKilled)
• TLS/certificate issues
• Network connectivity problems

Multi-Source Correlation

Teaches Claude to correlate:

• Logs - Application output and errors
• Events - Kubernetes system events
• Metrics - Resource usage and performance data

Example Usage

Basic Investigation

I have a pod named api-gateway-7d8f9b-xyz in the production namespace that keeps crashing. Can you help me investigate?

Claude will:

1. Check pod status and events
2. Retrieve current and previous container logs
3. Analyze resource usage
4. Correlate findings into a timeline
5. Identify root cause
6. Provide remediation recommendations

Guided Troubleshooting

My service is returning 503 errors. I see some pods are in CrashLoopBackOff state.

Claude will:

1. Assess if this is a pod-level or service-level issue
2. Follow decision tree to identify crash cause
3. Check for common patterns (OOMKilled, image pull failures, etc.)
4. Provide specific investigation commands
5. Recommend fixes based on root cause

What Makes This Skill Different

Read-Only Investigation

• Never modifies cluster resources
• Only observes and recommends
• All changes require manual review

Root Cause Focus

• Distinguishes symptoms from underlying issues
• Uses temporal analysis to identify what happened FIRST
• Validates causation, not just correlation

Practical Examples

• Real-world incident scenarios
• Step-by-step investigation workflows
• Expected outputs and findings

Decision Trees

• Clear guidance on which path to take
• Question-based navigation
• Quick identification of issue types

Skill Quality

• Word Count: ~1,400 words (technique skill target: <1,000 words)
• Frontmatter: 248 characters (under 1,024 limit)
• Format: Follows Konflux skills conventions
• Testing: Includes comprehensive test scenarios

Integration with SRE Workflows

This skill complements:

• Incident response processes
• On-call troubleshooting
• Post-mortem analysis
• SRE runbook development

Prerequisites

Users should have:

• Basic Kubernetes knowledge (pods, deployments, services)
• Access to kubectl or Claude Code with Kubernetes MCP
• Understanding of container concepts

Maintenance

This skill is based on:

• Kubernetes best practices (platform-agnostic)
• SRE investigation methodologies
• Real-world incident patterns
• Community feedback and common issues

Contributing

Improvements welcome:

• Additional common issue patterns
• More real-world examples
• Platform-specific guidance (EKS, GKE, AKS)
• Enhanced decision trees

Related Skills

Consider combining with:

• understanding-kubernetes-resources - For resource basics
• optimizing-kubernetes-workloads - For resource optimization
• monitoring-kubernetes-clusters - For proactive monitoring

License

Same as parent repository.

Authors

Gjorgji Georgievski (ggeorgie@redhat.com)

Extracted from the SRE Copilot plugin - a general-purpose SRE toolkit for Claude Code.

Version History

• 1.0.0 - Initial skill creation by Gjorgji Georgievski
  • 5-phase investigation methodology
  • Common issue patterns
  • Real-world examples
  • Decision trees and troubleshooting matrix