Basic Usage Guide¶

This guide explains how to use MANA day to day once it is installed and your pipeline is sending webhooks: how to see what it did, how to run health checks and rollbacks by hand, and how to read its logs.

kubectl -n orchestrator port-forward svc/orchestrator-service 8000:8000 &
export MANA=http://localhost:8000

In-cluster, use http://orchestrator-service.orchestrator.svc:8000.

How MANA Works¶

MANA is event-driven, not a monitoring loop:

1. Receive. Your pipeline posts a webhook when a deployment starts, succeeds or fails.
2. Record. On success, the version is stored as the rollback target. On failure, the failure count for that deployment ID is incremented.
3. Decide. Retry (while the count is within MAX_RETRIES), roll back (once it exceeds the limits and a previous version exists), or alert.
4. Act. A retry asks GitHub to re-run the workflow after a backoff. A rollback patches the Deployment back to the last good image tag.
5. Verify. A rollback is only reported successful after all replicas are ready and each pod's health endpoint answers.
6. Log. Every step is written to the audit log and counted in Prometheus metrics.

MANA only acts on deployments named in webhooks. It does not discover or watch Deployments by namespace or label.

Seeing What MANA Did¶

Look up one deployment¶

State is stored per deployment ID (the ID your pipeline puts in the webhook), for 24 hours:

curl -s $MANA/deployment/<deployment-id> | jq

{
  "deployment_id": "deploy-123",
  "namespace": "production",
  "app_name": "sample-app",
  "version": "v1a2b3c4-1760000000",
  "status": "rolled_back",
  "retry_count": 4,
  "failure_type": "health_check_failure",
  "previous_version": "v0f9e8d7-1759990000",
  "timestamp": 1760000123.45,
  "metadata": { "workflow_run_id": "9876543210" }
}

A 404 means the ID is unknown or older than 24 hours.

List recent deployments¶

curl -s "$MANA/deployments/recent" | jq                         # last 10
curl -s "$MANA/deployments/recent?namespace=production&limit=20" | jq

Results are newest first and include deployment_id, namespace, app_name, version, status, retry_count and timestamp.

Deployment statuses¶

Statuses are lowercase in the API.

Check the live Deployment in Kubernetes¶

For replica counts and conditions, ask Kubernetes directly:

kubectl -n production get deployment sample-app
kubectl -n production rollout history deployment/sample-app
kubectl -n production get pods -l app=sample-app -o wide

GET /metrics/deployment/{namespace}/{name} is documented in the API Reference but is currently shadowed by the Prometheus endpoint, so use kubectl for this.

Manual Operations¶

Run a health check¶

Runs the same verification MANA uses after a rollback (all replicas ready, then each pod's /health). It blocks until the check passes, fails or times out.

curl -s -X POST $MANA/health-check -H 'Content-Type: application/json' \
  -d '{"namespace":"production","deployment_name":"sample-app","timeout":60}' | jq

{
  "deployment_name": "sample-app",
  "namespace": "production",
  "healthy": true,
  "check_duration_seconds": 3.4
}

Roll back to a specific version¶

curl -s -X POST $MANA/rollback -H 'Content-Type: application/json' \
  -d '{"namespace":"production","deployment_name":"sample-app","target_version":"v0f9e8d7-1759990000"}' | jq

{
  "status": "success",
  "deployment_id": "manual-rollback-6c1f0c0e-6f58-4a8b-9d5f-0d7a3e1b2c44",
  "message": "Rolled back to version v0f9e8d7-1759990000"
}

The image tag of the application container (named like the Deployment, or the only container) is replaced, the version labels are updated, and health is verified. If the rollback is not healthy the call returns HTTP 500. Manual rollbacks are audited and counted (rollbacks_total{reason="manual"}) but do not create a stored deployment record. Use this when you want MANA's verification and audit trail instead of a plain kubectl rollout undo.

Simulate a pipeline event¶

Useful for testing and demos (see the Quick Start):

curl -s -X POST $MANA/webhook/deployment -H 'Content-Type: application/json' -d '{
  "deployment_id": "test-1", "namespace": "staging", "app_name": "sample-app",
  "version": "1.2.3", "status": "failed", "failure_type": "health_check_failure",
  "metadata": {"workflow_run_id": "1"}
}' | jq

Webhooks have real effects: enough failures for the same ID will patch the live Deployment. Test in a non-production namespace.

Things there is no endpoint for¶

• Marking a deployment successful. Send a success webhook instead; it also records the version as the rollback target.
• Triggering a retry directly. Retries are triggered by failure webhooks. To re-run a pipeline by hand, use GitHub (Actions, Re-run jobs).
• Listing monitored namespaces. There is no such concept; MANA acts on what the webhooks name.

Understanding Health Checks¶

Verification has two layers:

1. Kubernetes readiness. ready_replicas must equal the Deployment's replicas. Until then MANA keeps waiting (polling every 10 s).
2. Application health. MANA calls GET http://<pod-ip>:<HEALTH_CHECK_PORT><HEALTH_CHECK_PATH> on each running pod. A pod is healthy if it returns HTTP 200 with JSON containing "status": "healthy". The Deployment passes if at least one pod is healthy.

If replicas are ready but the application check fails 3 consecutive times, the verification fails early. Otherwise it gives up after 300 s (or the timeout you pass to POST /health-check). Details: Data Flow.

Test your application's endpoint the same way MANA does:

POD_IP=$(kubectl -n production get pod -l app=sample-app -o jsonpath='{.items[0].status.podIP}')
kubectl -n orchestrator exec deploy/orchestrator -- \
  python -c "import requests; r=requests.get('http://$POD_IP:5000/health', timeout=5); print(r.status_code, r.text)"

Retries and Backoff¶

The delay before retry n is min(INITIAL_BACKOFF × BACKOFF_MULTIPLIER^(n-1), MAX_BACKOFF). With the defaults (10 s, ×2, cap 300 s):

There is no jitter. With the default MAX_RETRIES=3, only the first three delays are used before a rollback becomes possible. Tune with Orchestrator Configuration.

Reading the Audit Log¶

Audit events go to the orchestrator's standard output.

kubectl -n orchestrator logs -l app=orchestrator --tail=200
kubectl -n orchestrator logs -l app=orchestrator --since=1h
kubectl -n orchestrator logs -l app=orchestrator -f

# Only audit lines
kubectl -n orchestrator logs -l app=orchestrator | grep "AUDIT:"

# Everything about one deployment
kubectl -n orchestrator logs -l app=orchestrator | grep "<deployment-id>"

With the default log format, an audit line looks like:

2026-02-05 14:30:00,123 - self-healing-orchestrator.audit - INFO - AUDIT: rollback_completed

The structured fields (deployment ID, namespace, version, success, details) are attached to the log record as audit_event. To see them as JSON in your log pipeline, use a JSON log formatter; see Monitoring.

Audit actions¶

Inspecting Redis¶

kubectl -n orchestrator exec deploy/redis -- redis-cli keys 'deployment:*'
kubectl -n orchestrator exec deploy/redis -- redis-cli get 'last_successful:production:sample-app'
kubectl -n orchestrator exec deploy/redis -- redis-cli ttl 'deployment:<deployment-id>'

last_successful:<namespace>:<app> is the rollback target; it has no expiry. If a rollback keeps answering "No previous version", this key is probably missing.

Best Practices¶

1. Send a success webhook for every good deployment so a rollback target always exists.
2. Keep deployment IDs stable across retries of the same logical deployment, or the failure count resets (CI/CD Integration).
3. Monitor MANA itself. Alert on OrchestratorDown, pod restarts, and /health reporting degraded.
4. Watch the key metrics: deployments_total{status="failed"}, rollbacks_total, retries_total. See Monitoring.
5. Alert on manual_intervention_required in your log pipeline; it means automation gave up.
6. Rehearse failures regularly in staging with the failure recipes.
7. Review the audit trail periodically for recurring failure patterns.
8. Restrict access to the API; it has no authentication (Security).

Troubleshooting Usage Issues¶

More in the Troubleshooting Guide.

Next Steps¶

1. API Reference
2. Configuration Guide
3. Monitoring
4. CI/CD Integration
5. Architecture Overview
6. Contributing