> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hollerith.monarcha.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Running Hollerith in production

> Run a scheduled job

```python theme={null}
context_id = store.get("churn-nightly:context_id")
clf = Hollerith.from_fitted_context(context_id)
preds = clf.predict(batch, idempotency_key=key)
```

A scheduled job is three decisions: where the context id lives between runs, what you do when
it is gone, and what you retry. Everything below follows from those.

## The shape of a scheduled job

Fit when the training data changes. Predict on a schedule, against the saved context.

```python theme={null}
from hollerith import Hollerith, HollerithError, NotFoundError

JOB = "churn-nightly"


def fit_and_store(train, run_day):
    clf = Hollerith().fit(
        train,
        target="churned",
        idempotency_key=f"{JOB}:fit:{run_day}",
    )
    store.put(f"{JOB}:context_id", clf.fitted_context_id_)
    store.put(f"{JOB}:expires_at", clf.fitted_context_expires_at_)
    return clf
```

`fitted_context_id_` is a raw document id with no prefix. Store it as an opaque string beside
`fitted_context_expires_at_`, so a scheduler can read the deadline without a network call.

## Handling expiry

A context lives 7 days from the `fit` that created it. Predicting against it does not extend
the clock, so anything scheduled less often than weekly will find it gone.

```python theme={null}
def client_for(train, run_day):
    context_id = store.get(f"{JOB}:context_id")
    if context_id is not None:
        try:
            return Hollerith.from_fitted_context(context_id)
        except NotFoundError:
            pass                      # expired, or never ours
    return fit_and_store(train, run_day)
```

A context can also lapse between the resume and the predict. Catch `NotFoundError` there too,
re-fit, and retry against the new context.

```python theme={null}
def score(train, run_day, batch, shard):
    clf = client_for(train, run_day)
    try:
        return clf.predict(batch, idempotency_key=key_for(clf, run_day, shard))
    except NotFoundError:
        clf = fit_and_store(train, run_day)
        return clf.predict(batch, idempotency_key=key_for(clf, run_day, shard))
```

Two codes arrive here: `fitted_context_expired` when yours lapsed, `fitted_context_not_found`
when the id was never yours. Both are `NotFoundError`, and re-fitting is the fix for both.

## Idempotency

Pass `idempotency_key` on every submit. A replay under the same key returns the original job
instead of queueing a second one, so a retried fit creates no second context and bills nothing.

```python theme={null}
def key_for(clf, run_day, shard):
    return f"{JOB}:predict:{run_day}:{clf.fitted_context_id_}:{shard:03d}"
```

Put the context id in the key. The server fingerprints a key against the job's kind, task,
target column, input hash, row counts, engine version and fitted context — reuse it with any
of those changed and you get `idempotency_conflict` (409) rather than a replay.

* **After a re-fit the key has to change.** The context is part of the fingerprint, so the old
  key against the new context conflicts. Deriving the key from `fitted_context_id_` handles it.
* **A replay is not a cached result.** It returns the same job, and scored predictions are kept
  1 hour — a replay after that gives you the job and `training_ref_expired` on its result.

## Concurrency buys you nothing

One worker drains one queue, oldest job first. Parallel submission adds three failure modes
and no throughput.

* **Nothing finishes sooner.** Ten concurrent submits finish no earlier than ten sequential
  ones, because the queue is FIFO and one job runs at a time.
* **Quota is reserved at submit** and released at completion, so a burst can hit
  `quota_exceeded` before any of it has run. A job that stays queued holds its reservation
  until the 00:00 UTC reset.
* **Byte-identical payloads share one staged object.** Two concurrent runs of the same
  deterministic batch race, and the first to finish schedules that object for deletion out
  from under the second.

Drain one queue from one worker, in order, with an idempotency key on every submit.

## What to retry, and what not to

Exactly four codes are retryable: `rate_limited`, `worker_warming_up`, `worker_unavailable`
and `cache_unavailable`. `quota_exceeded` is a 429 that is not one of them, so branch on
`retryable` rather than on the status code.

```python theme={null}
import time

def with_retry(call, attempts=4, base=2.0):
    for n in range(attempts):
        try:
            return call()
        except HollerithError as err:
            if not err.retryable or n == attempts - 1:
                raise
            time.sleep(base * 2**n)
```

The SDK has no backoff layer. It absorbs a retryable 503 while polling a job it has already
submitted, at a fixed `poll_interval`, and nothing else. The loop above is yours to write.

## Timeouts

Blocking calls stop waiting after 900 seconds and raise `TimeoutError`. The timeout ends your
wait, not the job — it keeps running, finishes, and bills.

```python theme={null}
handle = clf.submit(batch, idempotency_key=key)
store.put(f"{JOB}:pending_job", handle.id)
result = handle.wait(timeout=600.0)
preds = result.predictions
```

Submit first and keep the handle, so a timeout costs a wait and not the job id. If the process
itself dies, resubmit with the same key inside the hour: the payload uploads again, the job
replays, and you get the original result.

## What to log

* **The request id on every error.** `err.request_id` in the SDK, `requestId` on the wire.
  A bare `invalid_request` response carries none, and neither does an error the SDK raises
  before any request.
* **`fitted_context_id_` and `fitted_context_expires_at_`** on every fit. Without the id you
  cannot resume, and a re-fit is the only way back.
* **`handle.job.engine_version` and `handle.job.duration_ms`** on every terminal job. The
  engine version is what explains a prediction that moved.
* **`evaluation_.value` across runs.** One metric comes back per task, and its drift is the
  only accuracy signal you get.

`evaluate()` is a separate billed job that re-uploads the labeled frame, and a client resumed
from a context id cannot call it. Run it in the process that fits.

## When to re-fit

Predictions assume the rows you score look like the rows you fit on. A new product line or a
pricing change breaks that, and nothing warns you.

Re-fitting is one call and bills your training rows once. Put it on a schedule rather than
building drift detection you do not need yet.

## Checklist

* A stored `fitted_context_id_`, with `fitted_context_expires_at_` beside it.
* `NotFoundError` caught around both the resume and the predict, with a re-fit behind it.
* `idempotency_key` on every submit, derived from the run day and the context id.
* One worker, one queue, submissions in order.
* Retries branching on `err.retryable`, with backoff you wrote.
* `submit()` plus `handle.wait()` wherever a timeout would otherwise lose the job.
* Request id, context id, engine version and `evaluation_` in your logs.
* A scheduled re-fit, whether or not anything looks wrong.

## Next

* [The fitted context](/concepts/fitted-context) — expiry, resuming, and losing the fast path
* [Errors](/reference/errors) — all 26 codes, and which four are retryable
* [Limits and quotas](/reference/limits) — the daily reset and the 900-second timeout
* [Usage and billing](/account/usage-and-billing) — what each job kind bills
* [Troubleshooting](/help/troubleshooting) — a symptom, its cause, and the fix