Orchestrator Configuration¶

The orchestrator is configured entirely through environment variables, read once at startup by orchestrator/src/config.py. This page explains what each setting does, how the settings interact, and how to set them in each deployment method. For a flat list of every variable see Environment Variables; for the cross-component defaults table see the Configuration Reference.

This page documents the variables that the code actually reads. The broader Configuration Guide describes some additional options (leader election, webhook authentication, Redis TLS, and so on) that are not implemented in the current code. If a variable is not listed here or in Environment Variables, the orchestrator ignores it.

How Configuration Is Loaded¶

1. get_config() builds an OrchestratorConfig with OrchestratorConfig.from_env() the first time it is called and caches it.
2. validate() checks the values and raises ValueError on the first violation.
3. SelfHealingOrchestrator.__init__ validates again, connects to Redis (PING) and loads the Kubernetes config. An invalid value raises ValueError; an unreachable Redis or an unloadable Kubernetes config raises RuntimeError. In every case the process exits, so Kubernetes restarts the pod until the problem is fixed.

Changing a variable requires restarting the orchestrator pods. GET /config shows the values currently in effect (see the API Reference).

Settings¶

Redis¶

The client uses a 5-second connect and socket timeout. There is no password or TLS option in the current code, so use a network-isolated Redis (see Security).

Retry and rollback policy¶

The delay before retry n is:

delay(n) = min(INITIAL_BACKOFF × BACKOFF_MULTIPLIER^(n-1), MAX_BACKOFF)

Choosing values. The decision rule is retry while failure_count <= MAX_RETRIES, otherwise roll back if a previous version exists and the failed version is the one running (see Data Flow):

• MAX_RETRIES=3 (default): three retries, rollback on the fourth failure.
• MAX_RETRIES=1: one retry, rollback on the second failure. Good for fast-moving staging environments.
• MAX_RETRIES=0 is rejected by validation, so at least one retry is always attempted before a rollback. There is currently no "roll back immediately" mode.
• ROLLBACK_THRESHOLD was removed. If it is still set, the orchestrator logs a deprecation warning and ignores it.

The worst-case time spent retrying before a rollback is the sum of the backoff delays plus the time each re-run takes. With the defaults that is 70 s of backoff alone. If your target MTTR is under a minute, lower INITIAL_BACKOFF and MAX_RETRIES.

Health checks¶

The application health check is only the second half of verification. First, Kubernetes must report ready_replicas == replicas. The orchestrator then calls http://<pod-ip>:<HEALTH_CHECK_PORT><HEALTH_CHECK_PATH> on every running pod and expects HTTP 200 with a JSON body containing "status": "healthy".

Two related values are not environment variables:

• The overall verification timeout is 300 s by default (check_deployment_health(timeout=300)); the manual POST /health-check endpoint accepts a timeout field.
• The number of consecutive application-health failures tolerated before giving up is fixed at 3 (max_app_health_failures).

If your application returns a different health payload, either make it return {"status": "healthy"} or adapt check_application_health().

GitHub integration¶

Both are read directly by the orchestrator (not through OrchestratorConfig). If either is missing the orchestrator starts, logs a warning, and any attempted retry fails with an error log. GET / and GET /health report github_integration: false in that case.

The token needs permission to re-run workflows: Actions: Read and write on a fine-grained token scoped to the repository, or the repo (and workflow) scopes on a classic token. Store it in a Kubernetes Secret, never in a ConfigMap or image:

kubectl create secret generic github-token \
  --from-literal=token="$GITHUB_TOKEN" \
  -n orchestrator

orchestrator/k8s/deployment.yaml already reads it as secretKeyRef: {name: github-token, key: token}.

Logging and audit¶

Both values are loaded and validated into the config object, but the logging setup in orchestrator.py and api.py uses a fixed INFO level, and the audit logger is always on. Changing these variables currently has no effect on output.

Service¶

Setting Values by Deployment Method¶

cd orchestrator
cp .env.example .env          # edit values, never commit .env
set -a; source .env; set +a
uvicorn src.api:app --reload --port 8000

kubectl apply -f orchestrator/k8s/ -n orchestrator
kubectl rollout restart deployment/orchestrator -n orchestrator

Presets¶

Copy-paste starting points. Adjust to your risk tolerance.

# Development: fast feedback
MAX_RETRIES=2
INITIAL_BACKOFF=2.0
MAX_BACKOFF=30.0
BACKOFF_MULTIPLIER=2.0
HEALTH_CHECK_TIMEOUT=3

# Production: patient, but bounded
MAX_RETRIES=3
INITIAL_BACKOFF=10.0
MAX_BACKOFF=120.0
BACKOFF_MULTIPLIER=2.0
HEALTH_CHECK_TIMEOUT=5

Validation Errors¶

validate() raises ValueError with one of these messages. The pod will crash-loop until the value is corrected; check kubectl logs -n orchestrator -l app=orchestrator.

Non-numeric values for numeric variables fail earlier with Python's own ValueError: invalid literal for int().

Resources and Permissions¶

Both replicas serve webhooks and share state through Redis. There is no leader election, so two replicas can both act on the same deployment ID if the CI system delivers a webhook twice. Use a single replica if you need strict single-actor behaviour.