Skip to main content
evaluate=True scores the table you fit on and stores the result. clf.evaluate() does the same thing later, on demand.

​
What you get back

evaluation_ is an EvaluationResult frozen dataclass — not a dict, not a string. Read the fields off it.
The attribute does not exist until an evaluation has run. After a plain fit, hasattr(clf, "evaluation_") is False.

​
Exactly one metric

Classification returns accuracy. Regression returns RMSE. Those are the only two metric names the worker produces. There is no AUC, no F1, no R² and no MAE. If you need one of those, hold rows back yourself and score predict or predict_proba output against them.

​
How the method is chosen

Row count decides it, server-side. You cannot override the choice.
  • 10,000 rows or fewer — k-fold, starting at 5 folds and trimmed toward 2 if the run would exceed the evaluation compute budget.
  • Above 10,000 rows — a single holdout split. folds is None.
  • Fold membership is positional. With 5 folds, the first fold is every fifth row rather than a contiguous block.

​
The holdout split is unshuffled

The holdout is the last 20% of rows in file order. Training takes the first 80%, and nothing shuffles. The split is deterministic, so it reproduces exactly. It is also blind to how your file is sorted. A table sorted by its target produces a metric that means nothing. The model fits on the classes that happen to come first and is scored on ones it never saw. Shuffle once before you fit, with a seed you record.

​
What evaluate costs

fit(..., evaluate=True) is a second billed job. It re-uploads the whole frame, label column included, after fit has already uploaded it once. Two behaviours to know before you wire this into anything:
  • fit(evaluate=True, wait=False) drops the evaluation. The early return skips it. No error, no warning, no evaluation_.
  • A client from from_fitted_context() cannot evaluate. It raises RuntimeError("call fit() before predict()") — the labeled rows are not in that process.

​
Comparing Hollerith with what you run today

This page carries no accuracy numbers of our own. A score on somebody else’s table tells you nothing about yours. A tuned gradient boosting pipeline — XGBoost, LightGBM, CatBoost — may still win on a stable table that someone already maintains. evaluate exists to settle that question, not to win it.

​
Hold the data fixed

Same rows, same features, same split, for every model in the comparison. Build the split once and reuse that object — splitting separately per model hands each one different data. Fit any preprocessor on the training split only. An encoder or scaler fitted on the full frame leaks the test set into training and flatters whichever model consumed it. Fix every seed, and report the seed next to the number.

​
Measure more than one thing

A single metric does not decide anything on its own. Put fit time and inference time in the same table as accuracy. The costs sit in different places. A model you retrain nightly on your own hardware is a different proposition from one you pay for per call — see Latency.

​
Know your noise floor

A holdout metric is an estimate, and estimates have spread. Two models separated by less than that spread are tied, not ranked. Repeat the split under several seeds and look at how far the metric moves. On small tables it moves a lot.

​
Match the budget on both sides

If you tune the baseline, say so. Give both sides comparable effort, or state plainly which one got more. Hollerith has no hyperparameters, so the effort is not symmetric. Report the wall-clock and engineering time each side consumed.

​
Split the way you will use it

A random split assumes the rows you score look like the rows you fit on. If you will predict next month from this month, use a time-based split. If you will score customers the model has never seen, group the split by customer. A random split on grouped data reports a number you will not see again in production.

​
One dataset is an anecdote

Run at least three tables before making a general claim, including one about your own stack. A result that holds direction across tables is worth acting on. A single win is worth running a second time.

​
Checklist

Before you believe any comparison, yours or ours:
  • Every model saw the same rows, the same features and the same split object.
  • Seeds are fixed and recorded next to the result.
  • Preprocessors were fit on the training split only.
  • The test set was scored once, at the end.
  • The metric matches what an error actually costs you.
  • Fit time and inference time are in the table, not only accuracy.
  • The gap between models is wider than the spread across seeds.
  • Tuning budget was matched on both sides, and stated either way.
  • The split is time-based or grouped where that is how the model will be used.
  • More than one dataset stands behind any general claim.

​
Next