Skip to main content
A call spends its time in four places, and each is driven by something different. Find the one that dominates before you change anything.
  • Upload — how many bytes your table gzips down to.
  • Queue and warm-up — what is ahead of you, and whether the worker is up.
  • Forward pass — how much table the model reads.
  • Result fetch — how many rows come back, and whether they fit inline.

​
Upload

The SDK serializes your payload to JSON, gzips it, hashes the compressed bytes, and PUTs them to object storage. At or below 64 MiB compressed that is one PUT; above it, multipart.
  • The lever is bytes, not rows. Drop columns you do not use, and stop carrying float64 where you do not need it.
  • Nothing checks the ceiling locally. A payload over 5,000,000,000 bytes is serialized and gzipped in full, then rejected at presign with payload_too_large. You pay the compression pass, not the transfer.

​
Queue and warm-up

One worker drains the queue, oldest job first, one at a time. Your queue wait is the sum of the jobs ahead of yours, so a single large fit delays every small predict behind it.
  • Whether the worker stays up is a deployment setting. A persistent worker pays the model load on a rollout or restart; a scale-to-zero one pays it whenever the queue was empty.
  • An idle worker re-polls. The interval is a deployment setting — 60 seconds in production — so an arrival into an empty queue can wait that long to be claimed.
While a poll returns a retryable outage the SDK keeps waiting rather than raising, and calls on_warming each time. The hook receives the ServiceUnavailableError, so a lambda: ... taking no argument raises TypeError inside your own callback.

​
Forward pass

This is the stage that scales with how much table the model reads.
  • fit reads your whole training table.
  • Inline predict reads the training rows and the scored rows in one pass.
  • Context-backed predict reads only the rows you sent.
The relationship is worse than linear in rows: doubling the rows more than doubles the pass. Columns behave the same way, which is what the cell budget is protecting. No latency figure is published. The one measurement on file was taken on a GPU tier other than the live fleet, and cannot be quoted as a guarantee. Time your own table at the size you intend to run, and time it again when that size changes.

​
Result fetch

A small result comes back inline. Above 512 KiB the worker gzips it to object storage instead, and the API returns a presigned resultUrl and a content hash.
  • The large-result path costs a second round trip. The SDK follows the URL, gunzips it, and verifies the hash for you.
  • Ask for probabilities and quantiles when you will use them. Both multiply what has to come back.

​
The fitted context is the main lever

Everything above is dominated by one choice: whether the training table is re-read on every call. Fit once, and each later predict skips re-uploading it and skips re-reading it. The worker also caches loaded contexts, so repeated predicts against the same one skip the artifact download. The three ways you silently lose it are in The fitted context.

​
Bounding the wait

poll_interval defaults to 1.0 seconds. The loop sleeps before its first status check, so every blocking call costs at least one interval.
timeout defaults to 900.0 seconds and ends your wait, not the job. On expiry you get a TimeoutError while the job keeps running, finishes, and bills.

​
What this shape is for

One worker drains one queue, and there is no SLA on any of it. Nothing above is a guarantee about when your call returns. That makes Hollerith a fit for batch and near-line scoring. A synchronous request path that blocks a person on predict is the case it does not fit — put your own queue in front of it.

​
Next