Architecture Overview¶

MANA (Self-Healing CI/CD Pipeline) is a Kubernetes-native orchestration service that reacts to deployment events from a CI/CD pipeline and automatically recovers from failures by retrying the pipeline and, when that does not help, rolling the Deployment back to its last known-good version.

This page explains the design at a high level. Details are in Components and Data Flow.

Design Principles¶

• Event-driven. The pipeline reports deployment events by webhook. The orchestrator reacts to them; it does not poll the cluster for new Deployments.
• Stateful decisions. Whether to retry or roll back depends on history (how many times this deployment has failed and what the last good version was), so state lives in Redis.
• Verify, do not assume. A rollback is only reported as successful after Kubernetes shows all replicas ready and the application's own /health endpoint answers.
• Auditable. Every decision and action produces a structured audit event.
• Simple, replaceable pieces. One small FastAPI service, Redis, and the Kubernetes and GitHub APIs. No custom resource definitions or operators.

High-Level Architecture¶

Core Components¶

How a Deployment Is Handled¶

1. The pipeline deploys a new version and posts POST /webhook/deployment with status of in_progress, success or failed.
2. On success, the orchestrator records the version as the new rollback target.
3. On failure, it increments the retry count for that deployment ID and applies the rule:
4. up to MAX_RETRIES failures: schedule a GitHub Actions re-run after an exponential backoff;
5. after that, if a previous version is known and the failed version is the one running: patch the Deployment back to the last good version and verify health;
6. if neither is possible (no previous version, rollback unhealthy): raise an alert (manual_intervention_required).
7. Every step is written to the audit log, and Prometheus counters are updated.

The step-by-step version, with the decision table, is in Data Flow.

Retry path¶

Rollback path¶

Design Decisions¶

Why webhooks instead of watching the cluster? The pipeline already knows precisely when a deployment starts and how it failed, and it holds the context needed to retry (the workflow run). Watching would duplicate that and still could not re-run the pipeline.

Why re-run the pipeline for retries? A retry should repeat exactly what the pipeline did (build, push, deploy). Asking GitHub to re-run the workflow reuses all of that logic and keeps CI history accurate.

Why roll back by patching the image tag? It is explicit and auditable: the orchestrator sets the container image to the last version that reported success, rather than relying on Kubernetes revision history (kubectl rollout undo), which can point at a version that was never verified.

Why Redis? Retry counts and rollback targets need a small, fast, shared store with built-in expiry. Deployment records expire after 24 hours; rollback targets (last_successful:*) do not.

Security Architecture¶

The orchestrator can modify Deployments, so its permissions and exposure matter. This section summarises what the code and manifests implement today; the full model, known gaps and a hardening checklist are on the Security page.

Implemented

• RBAC: a ClusterRole limited to get/list/watch/update/patch on Deployments and ReplicaSets, get/list/watch on Pods, and create/patch on Events. It cannot read Secrets.
• Container hardening: non-root user (UID 1000), allowPrivilegeEscalation: false, all Linux capabilities dropped, CPU and memory limits.
• Secrets: the GitHub token comes from a Kubernetes Secret, never from an image or ConfigMap.
• Network exposure: Services are ClusterIP; nothing is exposed outside the cluster by default.
• Input validation: request bodies are validated by Pydantic models; configuration is range-checked at startup.
• Audit trail: structured events for every recovery action.

Not implemented (be aware)

• The API and webhook endpoint have no authentication; restrict access at the network layer.
• Redis has no password or TLS support in the client.
• The RBAC role is cluster-wide.

Scalability and Performance¶

• Replicas. The manifests run 2 orchestrator replicas that share state through Redis. There is no leader election, so if the same webhook reaches both replicas, both can act on it. Use one replica if you need strict single-actor behaviour.
• Synchronous rollback. A rollback runs inside the webhook request and waits for health verification (up to 300 s by default). Size your pipeline's HTTP timeout accordingly.
• Background retries. Retry waits run as in-memory background tasks in the replica that received the webhook.
• Listing deployments. GET /deployments/recent scans all deployment:* keys; it is meant for dashboards and debugging.
• Footprint. The orchestrator requests 100m CPU and 256Mi memory per replica.

Failure Modes and Recovery¶

Technology Stack¶

Extending MANA¶

MANA has no plugin framework; extension means changing the code. The natural extension points are:

• New failure types: add a value to FailureType in orchestrator/src/orchestrator.py and send it from your pipeline.
• New decision rules: edit handle_deployment_failure(); add a test for each branch.
• Other CI systems: the webhook half is CI-agnostic; implement an equivalent of trigger_github_workflow_rerun() for your system's re-run API.
• Notifications: alerts are routed by Alertmanager; add receivers in monitoring/alertmanager/alertmanager.yml. The orchestrator itself only logs manual_intervention_required.
• Health strategies: adjust check_application_health() for different response formats.
• Other storage backends: Redis is used directly through redis_client; replacing it means abstracting those calls.

See Contributing for conventions.

Known Limitations¶

Behaviours that differ from the intended design (re-run semantics, a shadowed metrics route, and some alerts without matching metrics) are listed with explanations on the Data Flow page.