A threshold is not a property of a model. It is a property of a model and a year.
Leaving a score cut-off where the development sample put it cost Brechin 2,713,298 in 2020 and 2021. The optimum had moved from 550 points to 580. At 580 the book earns 2,563,882; at the inherited 550 it loses 149,416. The card was not retrained, no coefficient moved, and its ranking held: area under the curve fell only 0.016816. The price of risk moved instead, and the threshold did not follow it.
Worked in full in Machine Learning for Finance by Julian R. Sterling, with every figure reproduced in a free workbook.See the book on Amazon →
Brechin Credit's threshold was chosen on 6,000 applications decided between 2015 and 2019, where profit peaked at 550 points. The identical card — no coefficient touched, no bin boundary moved, no points table rewritten — was then run over the 2,400 applications of 2020 and 2021. Every threshold from 480 to 550 loses money. The peak has moved to 580.
| Cut-off | Approved, per cent | Bad rate, per cent | Profit |
|---|---|---|---|
| 540 | 86.54 | 9.58 | −2,367,019 |
| 550 | 76.00 | 8.33 | −149,416 |
| 560 | 63.29 | 6.98 | 1,620,356 |
| 570 | 48.17 | 5.80 | 2,391,803 |
| 580 | 32.50 | 4.36 | 2,563,882 |
| 590 | 19.17 | 3.70 | 1,770,681 |
At the inherited 550 the book approves 76.00 per cent and loses 149,416. At 580 it approves 32.50 per cent and earns 2,563,882, or 1,068.28 per application. The gap between the policy that was running and the policy that was available is 2,713,298.
It sits in one slab of applicants. Between 550 and 580 there are 1,044 applications, of which 118 defaulted and 926 performed. Two prices settle the rest. An accepted case that performs is worth 6,982, being 185,000 of exposure at a margin of 4.12 per cent less 640 of origination cost. One that defaults costs 84,767 more than that.
| Slab from 550 to 580 points | Amount |
|---|---|
| Applications accepted | 1,044 |
| Value at 6,982 each | 7,289,208 |
| Defaults | 118 |
| Cost at 84,767 each | 10,002,506 |
| Value of writing the slab | −2,713,298 |
The ratio is the form worth carrying. That slab delivers 926 performing loans for 118 defaults, or 7.8475 to one. Brechin needs 84,767 divided by 6,982, or 12.1408 performing loans for every default, to break even — equivalently, a pool is worth writing while it defaults at less than 8.2367 per cent. In the development years the equivalent slab, from 600 down to 550, ran at 19.9529 to one and was worth 11,751,467. Same lender, same card, same four prices, and the marginal business went from paying nearly twenty to one to paying under eight.
The band between 550 and 560 defaulted at 7.1672 per cent in development and at 15.0820 per cent out of time, 2.1043 times as high. Identical points, identical bins, identical model. The applicants sitting between 550 and 560 in 2021 were not the applicants who sat there in 2017.
The ranking held. Area under the curve falls from 0.735329 to 0.718513, a loss of 0.016816; Gini falls from 0.470659 to 0.437027 and KS from 0.341154 to 0.324311. The bad rate column of the new grid falls without a single reversal in seventeen rows. The card sorts the new applications into the right order.
The decision moved much further than the measurement did. The cut-off moved thirty points. On a scale where twenty points doubles the odds, thirty points is one and a half doublings, a factor of 2.8284, so the odds demanded of a marginal applicant nearly trebled while the model's ability to sort them barely changed. Part of the move is a level error rather than a ranking error: correcting the intercept by 0.573620 in log-odds moves every applicant 16.55 points, so a policy written as decline below 550 has to be rewritten as 566.55 merely to mean what it meant. That accounts for 16.55 of the thirty. The remaining 13.45 points is what a uniform shift cannot fix, because the gaps are not uniform — 9.612 points in the worst band and far less at the top, so the deterioration is concentrated exactly where the lending decision is made.
The population moved as well as the level. The stability index between the two samples is 0.090517, inside most institutions' amber threshold. At a fixed 550 the approval rate falls on its own from 83.50 per cent to 76.00 per cent, because a fixed line declines more people when more people are below it. That is 7.50 points of automatic tightening. The optimum required 51.00 points of it.
Ask three things of any cut-off in front of a committee: what population it was optimised on, over what period, and what it earns on the most recent population available. Here those questions produce 17,348,284 and a loss, from a card nobody touched.
Then price the proposal against the alternatives on either side of it, in money, over the whole grid rather than at the single row being recommended. Out of time, 550 against 580 is 2,713,298. A threshold is not a property of a model. It is a property of a model and a year.
It comes from the prices, not the model. A performing loan is worth 6,982 and a default costs 84,767 more than that, so a pool is worth writing while it defaults at less than 6,982 divided by 84,767, or 8.2367 per cent. Equivalently, 12.1408 performing loans pay for one default.
On Brechin's development sample, moving from 600 points to 550 raises the bad rate from 1.69 per cent to 4.15 per cent and adds 11,751,467 of profit. The slab in between delivers 19.9529 performing loans for each default, against a break-even ratio of 12.1408.
No. Area under the curve fell only 0.016816, from 0.735329 to 0.718513, while the profit-maximising cut-off moved thirty points and a policy left at 550 went from earning 17,348,284 to losing 149,416. The trigger is the marginal slab's bad rate against 8.2367 per cent.
This article is one calculation from Machine Learning for Finance. The book takes the same case from first principles to the decision, chapter by chapter, and every figure it prints is a live formula in the free companion workbooks.
Get the book on Amazon →Free companion files
Also on Amazon UK · Amazon Germany · Amazon France · Amazon Canada
Reading guide: corporate finance, valuation and markets → · All 1706 articles →
If this book helped — or didn’t — a few lines on Amazon are worth more than they look: they are what the next reader goes on. Write a review. The workbook stays free either way.