RESEARCH
How accurate is our risk model?
Every risk number Abacus Labs returns, from an account's volatility to the tracking error of a proposed trade, comes from one equity factor risk model. We tested it on 406 month-ends from 1993 to 2026, against five standard alternatives, on the kinds of portfolios our clients actually hold. Here is what we found, including where it still falls short.
What the model is
A factor risk model explains each stock's return with a small number of shared drivers, plus a part that belongs to the company alone. The shared drivers are the market, the company's industry, and characteristics such as size, momentum, volatility, value and profitability. Two stocks are correlated to the degree that they share these drivers.
Ours covers every US-listed stock and ETF, about 11,800 securities, with up to 52 industries and 13 style factors. It is updated every trading day. It belongs to the same class as the Barra and Axioma models, built from their public methodology notes, and we test it the way the academic literature recommends.
Three design choices carry most of the weight.
- Point in time. Every input is dated by when the market could first have known it. Companies that later went bankrupt or were acquired stay in the history, with the return their holders actually received. A quarterly filing counts from the first close after it was published, not from its period end.
- Calibrated for optimized portfolios. Risk models are usually judged on how well they forecast an index. We judge ours on the portfolios an optimizer builds, because those are the ones our clients hold.
- Honest about thin data. A stock that listed sixty days ago has sixty days of history. Instead of treating missing days as zero, which makes a new stock look safe, we borrow an estimate from seasoned companies of the same size and industry and blend it out as history builds.
How we measure accuracy
The standard test is the bias statistic. For each portfolio and each date, divide the return the portfolio earned by the risk the model predicted. If the model is right, that ratio has a standard deviation of one across dates. A reading of 1.2 means realized risk was 20 percent higher than forecast. A reading of 0.8 means the model was too cautious. With 406 dates, anything between about 0.93 and 1.07 cannot be told apart from perfect.
We scored the model at horizons from one month to one year, across several hundred portfolio types: random baskets, the major indexes, index trackers of 25 to 250 stocks, optimized portfolios, single stocks and ETFs. We scored five common alternatives on the same dates: a sample covariance matrix, Ledoit-Wolf shrinkage, a constant-correlation model, a single-factor market model, and our own earlier statistical model.
Random portfolios cannot tell models apart
Most published accuracy tests use random or index portfolios. On those, almost any model with correlations looks good: all six scored between 0.98 and 1.11 on random 100-stock baskets. That test cannot tell a raw sample covariance from a sixty-factor model.
The test that separates them is a portfolio an optimizer built. An optimizer seeks out the stocks whose risk the model underestimates, so every flaw in the model shows up in its answer. On the minimum-variance portfolio of the 1,000 largest stocks, the sample covariance under-predicted risk by a factor of 1.93, and the single-factor model by 3.34. Our model read 0.98, inside the band.
This matters because an optimizer drives every Abacus Labs recommendation. When an agent asks to cut a concentrated position while keeping tracking error under two percent, the model is both the objective and the constraint. A model that under-predicts optimized risk by half would make every option look twice as safe as it is.
Tracking error you can plan on
For direct indexing and tax-loss harvesting, the number that matters is tracking error against a benchmark. We built optimized S&P 500 trackers of 25 to 250 stocks every month and compared predicted with realized tracking error.
Realized tracking error runs 10 to 40 percent above the forecast, the same gap the literature documents for any optimized portfolio. We publish it rather than hide it. In practice, read a forecast of 1 percent as about 1.0 to 1.4 percent realized.
Built to survive scrutiny
A risk model is only as good as its data discipline. Four checks run with every build.
- A truncation test. We delete every row after the model date from the raw inputs, rebuild, and check that the model is bit-for-bit identical. Any leak of future information fails the build.
- An availability audit over all 1,764 weekly models since 1993, confirming each used only data published before its date.
- Survivorship. About 27,000 delisted securities stay in the history with their terminal returns.
- Coverage. 100 percent of listed US stocks and ETFs at every year end from 1995 to 2025.
We also wrote down the rule for adopting a new model before we built it. The current model met it on every family that matters: optimized portfolios improved by nine to twelve times the tolerance. One random-portfolio family missed by 0.001, inside sampling noise. We adopted it anyway and published why.
Where it is not good enough yet
No risk model is calibrated in the first weeks of a crisis, and ours is no exception. Risk models learn from the past, and a crisis is the moment the past stops being a guide.
A second open problem is larger trackers. For 100- and 250-stock trackers the bias reads 1.15 to 1.30: their active risk is small, so the part the factors miss is a large share of it. We are testing two fixes from the literature, nonlinear shrinkage and a factor aligned with the optimizer's own holdings. We publish these numbers with every model version.
The production model
The research above covers 33 years of history. The model Abacus Labs serves is rebuilt every morning from licensed exchange data with the same method. On 2021 to 2026, data neither version was tuned on, it matched the research model on every test family and was better calibrated on minimum-variance portfolios. Each daily build runs a calibration check before it is published.
Could an AI do this instead?
We tested whether frontier language models could improve the model. In one experiment they tuned the estimator's settings. In another they read each month's output and edited the risk numbers directly, for 130 months with the date hidden. The edits cut out-of-sample error by 19 percent. A one-line mechanical rule fed the same information cut it by 15 percent, and head to head the two were a coin flip. Language models are useful analysts of a risk model's diagnostics and poor estimators of its numbers. The estimator stays mechanical. The language model's job in Abacus Labs is the conversation, not the covariance.
What it means in practice
An agent using Abacus Labs never sees a covariance matrix. It sees what the matrix implies for one account: volatility and tracking error before and after each proposed trade, exposures in plain words, and the number of stocks needed to stay close to a benchmark. Behind those numbers is a model tested on the portfolios that matter, with its weaknesses written down.