Secrets, logging, monitoring, cost guards

Production AI without observability is gambling. This lesson is the safety net.

A pilot who never reads the instruments will eventually crash. Logs and dashboards are the instruments.

Five non-negotiable production habits:

1. Secrets in env vars or secret manager. Never in code.
2. Structured logs for every LLM call: timestamp, user_id, prompt_hash, model, tokens, latency, status.
3. Cost dashboard: provider dashboard + your own.
4. Hard spending caps in provider dashboards.
5. Rate limit per user/IP to prevent abuse.

For Python, log with logging and ship to Logtail, Datadog, or Grafana Cloud (free tiers exist).

Wrap LLM calls:

import logging, time, hashlib, json
log = logging.getLogger("ai")

def call_llm(messages, user_id):
    started = time.time()
    h = hashlib.sha256(json.dumps(messages, sort_keys=True).encode()).hexdigest()[:10]
    try:
        r = client.chat.completions.create(model="gpt-4o-mini", messages=messages)
        log.info({
            "user_id": user_id, "prompt_hash": h, "model": "gpt-4o-mini",
            "tokens": r.usage.total_tokens, "latency_ms": int((time.time()-started)*1000),
            "status": "ok",
        })
        return r
    except Exception as e:
        log.error({"user_id": user_id, "prompt_hash": h, "status": "error", "err": str(e)})
        raise

Cost guard via simple per-user counter in Redis, capping daily token spend.