A Bayesian model will always give you a number. The harder question is whether that number deserves to be acted on. TriaCaf answers it the same way on every run: five verification layers, the last of which is a person on your team.
This post walks through each layer, what it checks, and what happens when a check does not pass. Nothing here is a future plan; it is how runs work in the product today.
1. Execution: did the fit run on the data it names?
The first layer is deliberately boring. It confirms that the model fit completed, and that it ran against the dataset the run says it used. If a fit crashes, times out or is aborted, the run stops here and no estimate is shown. A result that cannot say which data produced it is not a result.
2. Diagnostics: did the sampler actually converge?
Most TriaCaf packs fit their models with Markov chain Monte Carlo (MCMC). MCMC can produce confident-looking posteriors from chains that never settled, so the fits behind every reported estimate are checked against standard convergence gates:
- R-hat at most 1.01. R-hat compares the chains with each other; values near 1 mean they agree on the same distribution.
- Bulk effective sample size (ESS) at least 400. Correlated draws carry less information than their count suggests; ESS is the honest count.
- Divergences at most 0.5% of draws. Divergent transitions are the sampler telling you part of the posterior was hard to explore.
A specification that misses one of these gates is flagged as a warning on the result, so you see it rather than discover it later. A specification with R-hat above 1.1 is treated as not converged at all: the run fails and no estimate is reported.
Some packs do not need MCMC. Customer lifetime value uses conjugate models with exact posteriors, and parts of A/B testing and anomaly detection are closed-form. Those are labelled as exact rather than passed through sampler checks that do not apply to them. Multi-touch attribution and the geo test designer are deterministic, and say so.
3. Predictive: does the model reproduce the data it was fit on?
A converged model can still be the wrong model. The predictive layer asks whether it describes the data:
- Marketing mix models get a posterior predictive check, reported as R², MAPE and the share of observations that fall inside the 95% predictive interval.
- Synthetic control has a pre-period fit gate: if the synthetic unit cannot track the treated unit before the intervention, the run fails rather than reporting an effect.
- Most other packs check that their specifications agree on the quantity being predicted.
Two packs report this layer as skipped, with the reason shown. Causal inference does not have a pre-period predictive check yet; it is in development. The geo test designer plans an experiment rather than estimating an effect, so there is nothing to predict.
4. Sensitivity: does the answer survive a different set of assumptions?
Every pack that estimates an effect fits two or three model specifications, not one. For marketing mix modeling, for example, the same data is fit with different adstock lags; for churn, with pooled, segmented and covariate-adjusted survival models. The sensitivity layer compares the specifications and produces an agreement score.
For most packs, that score drives the verdict at the top of the Decision Brief: at least 80% agreement reads as high confidence, 50 to 79% as moderate, and below 50% as low, with a plain sentence telling the reader whether to act, review or treat the result as provisional. Where a pack reports a sensitivity check instead of an agreement score, as marketing mix modeling does, the verdict follows the verification layers: a warning on any layer tells the reader to review before acting. Either way, disagreement between specifications is reported rather than averaged away.
5. Human: a person approves or rejects
When the first four layers are done, the run pauses. Someone on your team reviews the result in the TriaCaf portal and approves or rejects it. Only an approved run is marked complete. A rejected run fails, and a run nobody answers does not quietly complete on its own.
This layer is not a formality. The first four layers can tell you that a model is internally sound; only someone who knows the business can tell you whether the answer makes sense for it.
What you get at the end
Each layer's status travels with the result: into the portal, into the Decision Brief's verification strip, and into the run's model card, which is signed with Ed25519 so anyone with TriaCaf's public key can confirm it has not been altered. If you want to see what that looks like on paper, the next post walks through a Decision Brief section by section, or you can read more about the layers on the Platform page.