Configuration Guide¶
MANA is configured through environment variables. There is no configuration file, no ConfigMap-based loader and no Helm values file: the orchestrator and the sample app read the environment once at startup, validate it, and fail fast if a value is invalid.
This page is the starting point. It shows where to set values, summarises what you can tune, and gives ready-made profiles. Detailed pages:
- Orchestrator Configuration: every orchestrator setting explained
- Sample App Configuration: settings and failure recipes for the demo app
- Environment Variables: flat lookup of all variables, secrets and script inputs
- Configuration Reference: defaults per deployment method, Terraform variables, ports, resources
How Configuration Works¶
- Load.
OrchestratorConfig.from_env()reads variables with defaults. Types are converted withint()/float(), so a non-numeric value fails immediately. - Validate.
validate()checks ranges (for exampleMAX_RETRIES >= 1,MAX_BACKOFF > INITIAL_BACKOFF). - Start. The orchestrator then connects to Redis and loads the Kubernetes config. Any failure aborts startup, and Kubernetes restarts the pod until it is fixed.
- Apply changes by restarting. Values are not reloaded at runtime. After changing them, restart the pods.
GET /configshows what the running process uses.
Where to Set Values¶
| Deployment method | Where | Apply with |
|---|---|---|
| Kubernetes manifests | env: list in orchestrator/k8s/deployment.yaml |
kubectl apply -f orchestrator/k8s/ && kubectl rollout restart deployment/orchestrator -n orchestrator |
| Quick change on a running cluster | kubectl set env |
kubectl set env deployment/orchestrator MAX_RETRIES=2 -n orchestrator (triggers a rollout) |
| Terraform | max_retries, replicas, resources module variables |
terraform apply (other variables are not exposed yet) |
| Docker Compose | environment: under services.orchestrator in docker-compose.yml |
docker compose up -d |
| Local process | orchestrator/.env (copy from .env.example) loaded into your shell |
restart uvicorn |
Note
Changes made with kubectl set env are overwritten the next time you kubectl apply the manifest or run Terraform. Update the source of truth too.
What You Can Tune¶
Orchestrator¶
| Area | Variables | What it controls |
|---|---|---|
| Redis | REDIS_HOST, REDIS_PORT |
Where state is stored |
| Retry and rollback | MAX_RETRIES |
How many failures are retried before rollback |
| Backoff | INITIAL_BACKOFF, BACKOFF_MULTIPLIER, MAX_BACKOFF |
Delay before each retry |
| Health checks | HEALTH_CHECK_PORT, HEALTH_CHECK_PATH, HEALTH_CHECK_TIMEOUT |
How the application's health endpoint is called |
| GitHub | GITHUB_TOKEN, GITHUB_REPO |
Whether retries can trigger workflow re-runs |
| Logging | LOG_LEVEL, ENABLE_AUDIT_LOGGING |
Parsed but not yet applied (logging is fixed at INFO; audit logging is always on) |
| Service | PORT |
Listen port when run with python -m src.api |
Fixed behaviours (not configurable today)¶
| Behaviour | Value |
|---|---|
| Deployment state TTL in Redis | 24 hours |
| Consecutive application-health failures tolerated | 3 |
| Overall health verification timeout | 300 s (the manual POST /health-check accepts a timeout) |
| Redis connect and socket timeout | 5 s |
| Poll interval while waiting for readiness | 10 s |
Sample app¶
APP_VERSION, ENVIRONMENT, PORT, DEBUG, FAILURE_RATE, STARTUP_TIME, plus parsed-but-unused WORKERS, TIMEOUT, KEEPALIVE, ENABLE_METRICS, METRICS_PORT. See Sample App Configuration.
Understanding the Key Settings¶
Retries versus rollback. Retries are decided first: while the failure count for a deployment ID is <= MAX_RETRIES, MANA schedules a re-run. Once it is higher, MANA rolls back if a previous version is known and the failed version is the one running. With the default (3) that means three retries and a rollback on the fourth failure. MAX_RETRIES is the only setting that decides when rollback starts; a leftover ROLLBACK_THRESHOLD is ignored and logged as deprecated. See Data Flow.
Backoff. The delay before retry n is min(INITIAL_BACKOFF × BACKOFF_MULTIPLIER^(n-1), MAX_BACKOFF): 10, 20, 40 seconds by default. There is no jitter.
Time to recovery. Retries cost backoff time plus the time for each pipeline re-run. If your goal is a short MTTR, lower MAX_RETRIES and INITIAL_BACKOFF; if failures are usually flaky infrastructure, raise them.
Health checks. Verification needs both Kubernetes readiness and an HTTP 200 with {"status": "healthy"} from the configured path on the configured port. If your application listens elsewhere or answers differently, change HEALTH_CHECK_PORT / HEALTH_CHECK_PATH or adapt the app.
GitHub integration. GITHUB_TOKEN and GITHUB_REPO are required only for retries. Without them rollbacks still work.
Profiles¶
Copy the block that fits and adjust. All variables are optional because every one has a default.
Fast feedback, short waits.
MAX_RETRIES=2
INITIAL_BACKOFF=2.0
MAX_BACKOFF=30.0
BACKOFF_MULTIPLIER=2.0
HEALTH_CHECK_TIMEOUT=3
Mirrors production behaviour, a little faster.
MAX_RETRIES=3
INITIAL_BACKOFF=5.0
MAX_BACKOFF=60.0
BACKOFF_MULTIPLIER=2.0
HEALTH_CHECK_TIMEOUT=5
Patient but bounded.
MAX_RETRIES=3
INITIAL_BACKOFF=10.0
MAX_BACKOFF=120.0
BACKOFF_MULTIPLIER=2.0
HEALTH_CHECK_TIMEOUT=5
One retry, then roll back.
MAX_RETRIES=1
INITIAL_BACKOFF=5.0
MAX_BACKOFF=30.0
Apply a profile to a running cluster:
kubectl set env deployment/orchestrator -n orchestrator \
MAX_RETRIES=2 INITIAL_BACKOFF=5.0
kubectl rollout status deployment/orchestrator -n orchestrator
curl -s $MANA/config | jq
Secrets¶
Only one setting is sensitive: GITHUB_TOKEN.
- Store it in a Kubernetes Secret and reference it with
secretKeyRef(the shipped manifest expects Secretgithub-token, keytoken). Never put it in a ConfigMap, an image, or a committed file. - Use a fine-grained token limited to one repository with Actions: Read and write.
- Rotate it regularly:
kubectl create secret generic github-token --from-literal=token="$NEW_TOKEN" \
-n orchestrator --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/orchestrator -n orchestrator
For production-grade secret management (Sealed Secrets, External Secrets Operator, Vault) see Security.
Validation Rules¶
| Setting | Rule |
|---|---|
MAX_RETRIES |
integer, at least 1 |
INITIAL_BACKOFF |
number, greater than 0 |
MAX_BACKOFF |
number, greater than INITIAL_BACKOFF |
BACKOFF_MULTIPLIER |
number, greater than 1.0 |
HEALTH_CHECK_TIMEOUT |
integer, greater than 0 |
HEALTH_CHECK_PORT |
integer, 1 to 65535 |
Error messages and fixes are listed in Orchestrator Configuration. Validate before applying by running the orchestrator's config check locally:
cd orchestrator
MAX_RETRIES=0 python -c "from src.config import OrchestratorConfig as C; C.from_env().validate()"
# ValueError: max_retries must be >= 1
Best Practices¶
- Keep manifests as the source of truth. Avoid long-lived
kubectl set envoverrides. - Change one thing at a time and confirm with
GET /config. - Match the health settings to your app before trusting rollbacks: a wrong path or port makes every verification fail.
- Make sure a rollback target exists. Send a
successwebhook for each good deployment, and give pods aversionlabel. - Document non-default values and why you chose them.
- Test in staging with the failure recipes in Sample App Configuration.
- Review the audit log after tuning to see how decisions changed (Monitoring).
Troubleshooting Configuration¶
| Symptom | Likely cause | Check |
|---|---|---|
Pod CrashLoopBackOff right after a change |
Invalid value | kubectl logs -n orchestrator -l app=orchestrator shows the ValueError |
| Change has no effect | Pods not restarted, or the manifest was re-applied over a kubectl set env |
curl $MANA/config; kubectl get deploy orchestrator -n orchestrator -o yaml |
github_integration: false |
GITHUB_TOKEN or GITHUB_REPO missing (common with Terraform) |
curl $MANA/health; add the variables |
| Every health check fails | Wrong HEALTH_CHECK_PORT/PATH, app returns something other than {"status":"healthy"}, or a NetworkPolicy blocks the orchestrator |
kubectl exec -n orchestrator deploy/orchestrator -- python -c "import requests;print(requests.get('http://<pod-ip>:5000/health').text)" |
LOG_LEVEL=DEBUG shows nothing extra |
Logging level is not applied yet | Known limitation |
| A variable from another document does nothing | It is not implemented | See below |
Settings you may see mentioned elsewhere¶
Earlier drafts of the documentation described options such as leader election, webhook authentication and secrets, Redis password/TLS, state TTL and cleanup intervals, jitter, namespace or label selectors for monitoring, and GitHub tuning variables. None of these exist in the code today, and setting them has no effect. In particular, MANA does not watch namespaces by label: it acts on the deployments named in webhook payloads. The complete list of supported variables is in Environment Variables.