> ## Documentation Index
> Fetch the complete documentation index at: https://agno-v2-codex-homepage-20260719-015142.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Kubernetes Reference
> Commands, customization, environment variables, and troubleshooting for the Kubernetes template.
The scripts install the Helm release `agentos` into the `agentos` namespace. Override with `AGENTOS_RELEASE` and `AGENTOS_NAMESPACE`. Start with [Deploy AgentOS on Kubernetes](/deploy/templates/helm/deploy) for prerequisites and the first deployment.
## Manage
| Task | Command |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Roll to a new image tag | `IMAGE_TAG=v2 ./scripts/k8s/redeploy.sh` after the release points at your registry |
| Restart pods in place | `./scripts/k8s/redeploy.sh` |
| Sync supported env variables | `./scripts/k8s/env-sync.sh` (updates nonempty values from a fixed allowlist that includes `RUNTIME_ENV`; defaults to `.env.production`, or pass `.env`) |
| Tail logs | Run the override-aware `kubectl logs` command printed by `up.sh` (default: `kubectl logs deploy/agentos -n agentos -f`) |
| Port-forward the API | Run the override-aware command printed by `up.sh` (default: `kubectl port-forward svc/agentos 8000:8000 -n agentos`) |
| Roll back a release | `helm rollback "${AGENTOS_RELEASE:-agentos}" -n "${AGENTOS_NAMESPACE:-agentos}"` |
| Tear down | `./scripts/k8s/down.sh` (add `--yes` to skip the confirmation) |
## Production auth
Token-Based Authorization protects AgentOS routes by default in production. Startup requires `JWT_VERIFICATION_KEY` or a readable JWKS file at the pod path in `JWT_JWKS_FILE`; otherwise the process exits.
Token-Based Auth gives you three things:
1. **Protected API access.** Requests to protected AgentOS routes require a valid token. `/`, `/health`, `/info`, `/docs`, `/redoc`, `/openapi.json`, and `/docs/oauth2-redirect` remain public.
2. **Per-request identity.** Middleware validates the token and exposes its `user_id`, optional `session_id`, scopes, and claims to the request.
3. **Scope-based permissions.** Token scopes control access to AgentOS routes and resources.
The templates do not enable per-user data isolation. To scope non-admin session, memory, trace, and run access to the JWT subject, pass `authorization_config=AuthorizationConfig(user_isolation=True)` to `AgentOS`. See [User Isolation](/agent-os/security/authorization/user-isolation).
To disable JWT authentication, set `authorization=False` in `app/main.py`, clear `secrets.jwtVerificationKey` and `jwtJwksFile` in the Helm release, then build and roll out the new image. `authorization=False` disables AgentOS scope enforcement. Configured JWT values continue to enable JWT validation. MCP OAuth remains active when `MCP_CONNECT_SECRET` is set. Only do this when another layer protects the service.
## Customize
Ask your coding agent to run `/create-new-agent`, or do it by hand. Create `agents/my_agent.py`:
```python theme={null}
from agno.agent import Agent
from app.settings import default_model
from db import get_postgres_db
INSTRUCTIONS = """\
What the agent does, which tools it uses, the rules to follow when answering.
"""
my_agent = Agent(
id="my-agent",
name="My Agent",
model=default_model(),
db=get_postgres_db(),
instructions=INSTRUCTIONS,
enable_agentic_memory=True,
add_datetime_to_context=True,
add_history_to_context=True,
num_history_runs=5,
)
```
Register it in `app/main.py`:
```python theme={null}
from agents.my_agent import my_agent
agent_os = AgentOS(
...
agents=[agent_builder, platform_manager, web_search, my_agent],
)
```
Local containers hot-reload on save. For production, build and push a new image tag, then run `IMAGE_TAG= ./scripts/k8s/redeploy.sh`. If the release still runs the official image, point it at your registry first: `IMAGE_REPOSITORY=/agentos IMAGE_TAG= ./scripts/k8s/up.sh`.
`app/settings.py` defines `default_model()`, used by every agent. Change it in one place:
```python theme={null}
from agno.models.anthropic import Claude
def default_model():
return Claude(id="claude-sonnet-5")
```
Add `anthropic` to `pyproject.toml`, set `ANTHROPIC_API_KEY` in `.env` for local runs, and regenerate pins:
```bash theme={null}
./scripts/generate_requirements.sh
```
Rebuild locally with `docker compose up -d --build`. For production, export `ANTHROPIC_API_KEY`, build and push a new tag, create a Kubernetes Secret, and add it through `extraEnv` in a small values file:
```bash theme={null}
docker build -t /agentos:v2 . && docker push /agentos:v2
kubectl create secret generic agentos-anthropic \
--from-literal=ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
-n "${AGENTOS_NAMESPACE:-agentos}" \
--dry-run=client -o yaml | kubectl apply -f -
```
```yaml anthropic-values.yaml theme={null}
image:
repository: /agentos
tag: v2
extraEnv:
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
name: agentos-anthropic
key: ANTHROPIC_API_KEY
```
```bash theme={null}
helm upgrade "${AGENTOS_RELEASE:-agentos}" charts/agentos \
-n "${AGENTOS_NAMESPACE:-agentos}" --reuse-values \
-f anthropic-values.yaml
```
`env-sync.sh` does not deliver arbitrary provider keys or the referenced JWKS file. Provide new provider keys through a Kubernetes Secret as shown above. Provide a JWKS file through a custom image or chart volume.
Agno ships 100+ toolkits. See [Toolkits](/tools/toolkits/overview).
```python theme={null}
from agno.tools.slack import SlackTools
my_agent = Agent(
...
tools=[SlackTools()],
)
```
1. Edit `pyproject.toml`.
2. Regenerate pins: `./scripts/generate_requirements.sh` (add `upgrade` to refresh every pin).
3. Rebuild locally with `docker compose up -d --build`, or build and push a new tag and roll to it with `IMAGE_TAG= ./scripts/k8s/redeploy.sh`.
Set both variables in your env file:
```bash theme={null}
SLACK_BOT_TOKEN=xoxb-...
SLACK_SIGNING_SECRET=...
```
Sync with `./scripts/k8s/env-sync.sh`. The interface activates automatically and routes messages to Agent Builder; change the `agent=` argument in `app/main.py` to point at another agent. See [Slack setup](/agent-os/interfaces/slack/setup).
The deployment check runs daily by default (`ENABLE_DEPLOY_CHECK=True`); it is deterministic and free. Scheduled evals are off by default (`ENABLE_SCHEDULED_EVALS=False`) because they use model calls. Both workflows stay runnable on demand regardless.
In the cluster these are chart values. Set them via `extraEnv` and `helm upgrade`; `env-sync.sh` does not sync `ENABLE_DEPLOY_CHECK`, `ENABLE_SCHEDULED_EVALS`, or `EVALS_*`.
## Format, validate, and run evals
The format, validate, and eval scripts run on the host and need a venv. Set it up once:
```bash theme={null}
./scripts/venv_setup.sh
source .venv/bin/activate
```
| Task | Command |
| ------------------- | ----------------------------- |
| Format | `./scripts/format.sh` |
| Lint and type-check | `./scripts/validate.sh` |
| Run smoke evals | `python -m evals --tag smoke` |
`./scripts/mcp_check.sh` runs inside the container, so it needs no venv.
## Environment variables
| Variable | Required | Default | Description |
| ------------------------------------------------------------- | ---------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OPENAI_API_KEY` | Yes | - | Models and embeddings. |
| `RUNTIME_ENV` | No | `prd` | `dev` sets `authorization=False`, which disables AgentOS scope enforcement. Configured JWT values continue to enable JWT validation. Compose sets `dev` locally; keep `prd` on a production cluster. |
| `JWT_VERIFICATION_KEY` | Production | - | Public key from os.agno.com. Quote the value so the multi-line PEM parses as one variable. |
| `JWT_JWKS_FILE` | Production | - | Path inside the pod to a JWKS file. `up.sh` and `env-sync.sh` set only the `jwtJwksFile` path. The current chart does not mount the file. |
| `MCP_CONNECT_SECRET` | No | - | OAuth consent secret (16+ chars) for connecting claude.ai and ChatGPT to `/mcp`. `up.sh` generates it into `.env.production` when the deploy has a public URL (`INGRESS_HOST` or `AGENTOS_URL`); set it by hand otherwise. |
| `AGENTOS_MCP_SIGNING_KEY` | No | generated | Optional high-entropy signing-key material (32+ chars) for OAuth tokens. Unset, a strong key is generated and persisted in the database. Rotating it invalidates outstanding tokens. |
| `AGENTOS_URL` | No | `http://agentos:8000` | Scheduler base URL. The chart resolves an explicit value first, then the ingress URL, then the release service URL. Set it only for a custom domain or tunnel. When `MCP_CONNECT_SECRET` is set, OAuth metadata uses this URL as its public origin. |
| `ENABLE_DEPLOY_CHECK` | No | `True` | Daily deployment-check cron. |
| `ENABLE_SCHEDULED_EVALS` | No | `False` | Daily run-evals cron. Uses model calls. |
| `EVALS_TAG` | No | `smoke` | Eval tag the run-evals workflow runs. |
| `EVALS_CASE_TIMEOUT_SECONDS` | No | `90` | Per-case timeout for run-evals runs. |
| `EVALS_SUITE_TIMEOUT_SECONDS` | No | `900` | Whole-suite timeout for run-evals runs. |
| `PARALLEL_API_KEY` | No | - | WebSearch uses the Parallel SDK when set, keyless MCP otherwise. |
| `SLACK_BOT_TOKEN` | No | - | Set with the signing secret to enable Slack. |
| `SLACK_SIGNING_SECRET` | No | - | Set with the bot token to enable Slack. |
| `DB_HOST` / `DB_PORT` / `DB_USER` / `DB_PASS` / `DB_DATABASE` | No | matches compose | Postgres connection. `up.sh` generates `DB_PASS` once and saves it to your env file. |
| `DB_DRIVER` | No | `postgresql+psycopg` | SQLAlchemy driver. |
| `AGNO_DEBUG` | No | `False` | Verbose Agno logs. Compose sets it for dev. |
| `WAIT_FOR_DB` | No | `True` in Helm | If `True`, the entrypoint blocks on the database before starting. The Helm chart and Compose set it to `True`. |
| `AGENTOS_NAMESPACE` | No | `agentos` | Namespace the k8s scripts target. |
| `AGENTOS_RELEASE` | No | `agentos` | Helm release name the k8s scripts target. |
| `IMAGE_REPOSITORY` | No | `agnohq/agentos` | Image the chart deploys. Read by `up.sh`. |
| `IMAGE_TAG` | No | `latest` | Image tag. `up.sh` installs it; `redeploy.sh` rolls the release to it. |
| `IMAGE_PULL_POLICY` | No | `IfNotPresent` | Set `Never` for images loaded into kind. Read by `up.sh`. |
| `INGRESS_HOST` | No | - | Publishes the API behind your ingress controller at this host. Read by `up.sh`. |
| `INGRESS_CLASS` | No | - | Ingress class name, for example `nginx`. Read by `up.sh`. |
## Troubleshooting
Install [kubectl](https://kubernetes.io/docs/tasks/tools/) and [Helm](https://helm.sh/docs/intro/install/) 3+. The scripts check for both before doing anything.
`up.sh` deploys into your current kubectl context and verifies it can reach the cluster first. Point kubectl at the target cluster and confirm `kubectl get namespace` works, then rerun.
Expected. Mint the key at [os.agno.com](https://os.agno.com): connect your OS (**Connect OS** → **Live**, enter your AgentOS URL), then turn on **Token-Based Authorization (JWT)** under **Settings** → **OS & Security** and paste the full PEM. To add a PEM later, set `JWT_VERIFICATION_KEY` and run `./scripts/k8s/env-sync.sh`. To use JWKS, first provide the file through a custom image or chart volume, then set `JWT_JWKS_FILE` to its pod path and sync.
AgentOS scope enforcement is on whenever `RUNTIME_ENV` is not `dev`. Set `JWT_VERIFICATION_KEY` and sync. `JWT_JWKS_FILE` works only when a custom image or chart mount already provides a readable file at that pod path. To disable JWT, set `authorization=False` in `app/main.py`, clear `secrets.jwtVerificationKey` and `jwtJwksFile` in the Helm release, and roll out your image. MCP OAuth remains active when `MCP_CONNECT_SECRET` is set.
The cluster can't pull the image. Confirm the tag was pushed and the cluster has access to your registry; for private registries, set `imagePullSecrets` in `charts/agentos/values.yaml`. On kind, `kind load docker-image` the tag and deploy with `IMAGE_PULL_POLICY=Never`.
The Postgres volume reads its password only on first initialization, so a lost or regenerated `DB_PASS` locks the app out of an existing volume. Restore the `DB_PASS` that `up.sh` saved to your env file and sync, fix the database in place with `ALTER USER`, or delete the PVC to reinitialize. Deleting the PVC deletes all data.
`AGENTOS_URL` resolves automatically: explicit value, then ingress URL, then in-cluster service DNS. If you set it by hand, make sure the pod can reach that URL, then run `./scripts/k8s/env-sync.sh`.
`up.sh` generates `MCP_CONNECT_SECRET` only when the deploy has a public URL (`INGRESS_HOST` or an explicit `AGENTOS_URL`). Deployed without one? Set `MCP_CONNECT_SECRET` and a public `AGENTOS_URL` in `.env.production` and run `./scripts/k8s/env-sync.sh`.