What to check before you deploy a model.
Use held-out examples, inspect the mistakes that matter, and compare available baselines before deciding whether a trained prediction model is ready to use.
A completed training run gives you a candidate model. Whether it helps is a separate question. Answer it with examples the model did not train on and criteria that match your intended use.
Keep an evaluation set aside
Held-out examples let you inspect performance on cases outside the training set. Check that duplicate or closely related records have not ended up on both sides of the split. For tasks that change over time, consider whether the test reflects the period in which the model will be used.
Read the mistakes
An overall result can hide important differences. Inspect per-question results and examples where the model was wrong. In a sales task, wrongly prioritizing an unlikely deal and missing a promising deal can have different costs.
Record which errors matter for your workflow. Consider whether some results need human review or more context before they are useful.
Compare with a starting point
Where baseline comparisons are available, use them to ask whether training helped. Compare on the same examples and with the same criteria. An improvement on a small sample is a reason to investigate further, not a guarantee about future cases.
Do not read a model’s probability as a measured accuracy rate. If you plan to use a threshold, inspect what falls on either side and how changing it affects the mistakes you care about.
Decide what is ready
Tuned Predictions separates evaluation from deployment. Review the candidate’s results before choosing a version to use. The illustrative numbers on this website are not benchmarks for your data.
Read more about the workflow and product transparency.