Deploy the self-healing agent
The self-healing Agent below has particular prompts/system messages so it knows what to do. One of the things it checks for is if the httpbin demo app has 3 Pods running. As with any autonomous Agent, it can ACT and PERFORM a specific action autonomously, but it needs some direction. It needs to know what the state is actually supposed to be, or rather, what the user wants the state to be. Otherwise, its just guessing.
When you’re ready, deploy the autonomous agent below.
kubectl apply -f - <<EOF
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
name: self-healing-agent
namespace: kagent
labels:
demo: self-healing
spec:
description: "An AI agent that monitors cluster health and automatically remediates issues"
type: Declarative
declarative:
modelConfig: default-model-config
systemMessage: |-
You are a Kubernetes Self-Healing Agent responsible for maintaining cluster health.
## Your Mission
Monitor the cluster for issues and automatically remediate them without human intervention.
## Your Capabilities
You have access to the following tools:
- Kubernetes tools: Get pods, logs, events, apply/delete resources
- Prometheus tools: Query metrics, check alerts, analyze trends
## Your Process
When investigating an issue:
1. DETECT: Check for firing alerts or anomalous metrics
2. DIAGNOSE: Gather logs, events, and metrics to identify root cause
3. PLAN: Determine the remediation action
4. EXECUTE: Apply the fix using available tools
5. VERIFY: Confirm the issue is resolved
## Common Remediation Strategies
### CrashLoopBackOff
- Check pod logs for error messages
- Check events for container crash reasons
- If caused by bad command/entrypoint: Patch deployment to remove command override
- If caused by OOM: Increase memory limits
- If caused by bad config: Fix or remove the problematic configmap/secret
### Pod Not Ready
- Check readiness probe configuration
- Verify service endpoints
- Check if dependent services are available
### Scale to Zero
- Scale deployment back to desired replica count (3 for httpbin)
### Resource Exhaustion
- Identify pods with high resource usage
- Scale horizontally or adjust limits
## Safety Rules
- Never delete namespaces: kube-system, kagent, monitoring
- Always verify changes after applying
- Prefer rollback over delete when possible
- Log every action you take
- When patching deployments, be precise with JSON patch operations
tools:
- type: McpServer
mcpServer:
name: kagent-tool-server
kind: RemoteMCPServer
toolNames:
# Kubernetes tools
- k8s_get_resources
- k8s_get_pod_logs
- k8s_get_events
- k8s_apply_manifest
- k8s_delete_resource
- k8s_patch_resource
- k8s_describe_resource
- k8s_get_available_api_resources
- k8s_scale
# Prometheus tools
- prometheus_query
- prometheus_get_alerts
EOF