Running Hollerith in production

context_id = store.get("churn-nightly:context_id")
clf = Hollerith.from_fitted_context(context_id)
preds = clf.predict(batch, idempotency_key=key)

A scheduled job is three decisions: where the context id lives between runs, what you do when it is gone, and what you retry. Everything below follows from those.

The shape of a scheduled job

Fit when the training data changes. Predict on a schedule, against the saved context.

from hollerith import Hollerith, HollerithError, NotFoundError

JOB = "churn-nightly"


def fit_and_store(train, run_day):
    clf = Hollerith().fit(
        train,
        target="churned",
        idempotency_key=f"{JOB}:fit:{run_day}",
    )
    store.put(f"{JOB}:context_id", clf.fitted_context_id_)
    store.put(f"{JOB}:expires_at", clf.fitted_context_expires_at_)
    return clf

fitted_context_id_ is a raw document id with no prefix. Store it as an opaque string beside fitted_context_expires_at_, so a scheduler can read the deadline without a network call.

Handling expiry

A context lives 7 days from the fit that created it. Predicting against it does not extend the clock, so anything scheduled less often than weekly will find it gone.

def client_for(train, run_day):
    context_id = store.get(f"{JOB}:context_id")
    if context_id is not None:
        try:
            return Hollerith.from_fitted_context(context_id)
        except NotFoundError:
            pass                      # expired, or never ours
    return fit_and_store(train, run_day)

A context can also lapse between the resume and the predict. Catch NotFoundError there too, re-fit, and retry against the new context.

def score(train, run_day, batch, shard):
    clf = client_for(train, run_day)
    try:
        return clf.predict(batch, idempotency_key=key_for(clf, run_day, shard))
    except NotFoundError:
        clf = fit_and_store(train, run_day)
        return clf.predict(batch, idempotency_key=key_for(clf, run_day, shard))

Two codes arrive here: fitted_context_expired when yours lapsed, fitted_context_not_found when the id was never yours. Both are NotFoundError, and re-fitting is the fix for both.

Idempotency

Pass idempotency_key on every submit. A replay under the same key returns the original job instead of queueing a second one, so a retried fit creates no second context and bills nothing.

def key_for(clf, run_day, shard):
    return f"{JOB}:predict:{run_day}:{clf.fitted_context_id_}:{shard:03d}"

Put the context id in the key. The server fingerprints a key against the job’s kind, task, target column, input hash, row counts, engine version and fitted context — reuse it with any of those changed and you get idempotency_conflict (409) rather than a replay.

After a re-fit the key has to change. The context is part of the fingerprint, so the old key against the new context conflicts. Deriving the key from fitted_context_id_ handles it.
A replay is not a cached result. It returns the same job, and scored predictions are kept 1 hour — a replay after that gives you the job and training_ref_expired on its result.

Concurrency buys you nothing

One worker drains one queue, oldest job first. Parallel submission adds three failure modes and no throughput.

Nothing finishes sooner. Ten concurrent submits finish no earlier than ten sequential ones, because the queue is FIFO and one job runs at a time.
Quota is reserved at submit and released at completion, so a burst can hit quota_exceeded before any of it has run. A job that stays queued holds its reservation until the 00:00 UTC reset.
Byte-identical payloads share one staged object. Two concurrent runs of the same deterministic batch race, and the first to finish schedules that object for deletion out from under the second.

Drain one queue from one worker, in order, with an idempotency key on every submit.

What to retry, and what not to

Exactly four codes are retryable: rate_limited, worker_warming_up, worker_unavailable and cache_unavailable. quota_exceeded is a 429 that is not one of them, so branch on retryable rather than on the status code.

import time

def with_retry(call, attempts=4, base=2.0):
    for n in range(attempts):
        try:
            return call()
        except HollerithError as err:
            if not err.retryable or n == attempts - 1:
                raise
            time.sleep(base * 2**n)

The SDK has no backoff layer. It absorbs a retryable 503 while polling a job it has already submitted, at a fixed poll_interval, and nothing else. The loop above is yours to write.

Timeouts

Blocking calls stop waiting after 900 seconds and raise TimeoutError. The timeout ends your wait, not the job — it keeps running, finishes, and bills.

handle = clf.submit(batch, idempotency_key=key)
store.put(f"{JOB}:pending_job", handle.id)
result = handle.wait(timeout=600.0)
preds = result.predictions

Submit first and keep the handle, so a timeout costs a wait and not the job id. If the process itself dies, resubmit with the same key inside the hour: the payload uploads again, the job replays, and you get the original result.

What to log

The request id on every error. err.request_id in the SDK, requestId on the wire. A bare invalid_request response carries none, and neither does an error the SDK raises before any request.
fitted_context_id_ and fitted_context_expires_at_ on every fit. Without the id you cannot resume, and a re-fit is the only way back.
handle.job.engine_version and handle.job.duration_ms on every terminal job. The engine version is what explains a prediction that moved.
evaluation_.value across runs. One metric comes back per task, and its drift is the only accuracy signal you get.

evaluate() is a separate billed job that re-uploads the labeled frame, and a client resumed from a context id cannot call it. Run it in the process that fits.

When to re-fit

Predictions assume the rows you score look like the rows you fit on. A new product line or a pricing change breaks that, and nothing warns you. Re-fitting is one call and bills your training rows once. Put it on a schedule rather than building drift detection you do not need yet.

Checklist

A stored fitted_context_id_, with fitted_context_expires_at_ beside it.
NotFoundError caught around both the resume and the predict, with a re-fit behind it.
idempotency_key on every submit, derived from the run day and the context id.
One worker, one queue, submissions in order.
Retries branching on err.retryable, with backoff you wrote.
submit() plus handle.wait() wherever a timeout would otherwise lose the job.
Request id, context id, engine version and evaluation_ in your logs.
A scheduled re-fit, whether or not anything looks wrong.

The fitted context — expiry, resuming, and losing the fast path
Errors — all 26 codes, and which four are retryable
Limits and quotas — the daily reset and the 900-second timeout
Usage and billing — what each job kind bills
Troubleshooting — a symptom, its cause, and the fix

Get started

Concepts

Guides

Reference

Account

Help

Running Hollerith in production

The shape of a scheduled job

Handling expiry

Idempotency

Concurrency buys you nothing

What to retry, and what not to

Timeouts

What to log

When to re-fit

Checklist

Next

​The shape of a scheduled job

​Handling expiry

​Idempotency

​Concurrency buys you nothing

​What to retry, and what not to

​Timeouts

​What to log

​When to re-fit

​Checklist

​Next

The shape of a scheduled job

Handling expiry

Idempotency

Concurrency buys you nothing

What to retry, and what not to

Timeouts

What to log

When to re-fit

Checklist

Next