Validate specific assumptions by reviewing historical behaviors. Run specific tests to demonstrate that the model is handling certain key assumptions.
I can think of several analogies to use for the “check-up”, the most obvious being medical. There’s also car maintenance, home HVAC maintenance, etc. All could be used as analogies for the check-up part of model back-testing. Back in my post on “Inspection” I talked about the need to review standard assumptions or perform the “sniff test”. The check-up is similar, but much more involved.
You perform a brief inspection of your health everyday, for instance are you tired? have a headache? something hurt? etc. But you head to the doctor for a check-up. There you’re likely to have a variety of more in-depth tests run (reflexes, blood work, etc.) - much more than you could or would do yourself.
When you do a check-up on your model you should look at two things.
1) Reviewing historical behavior
This is probably the most straightforward of the check-ups. You validate or substantiate certain modeling assumptions by looking a historical behavior.
The most obvious example of this is the core deposit beta or pricing assumption. This modeling parameter represents the rate sensitivity of core deposits given movements in market rates. If the Fed moves rates (up or down) how much will you change the administered rates on your core accounts? The answer is that it depends on the current environment, but we can also look and get a historical perspective. How sensitive has your core pricing been to rate changes in the past?
Since 2001 there have been three distinct periods over which the Fed made significant and multiple rate movements. Two periods of significantly falling rates, and one period of significantly rising rates. When you are considering appropriate beta factors for modeling it can help to see how the bank's rates reacted to changes in the past. Typically a graph like the one above will show that the bank's rates lagged changes by the Fed.
While the data does provide some insight to core rate behavior, we can't necessarily use it to "compute" a beta factor. Core rate changes are influenced by a variety of other factors in addition to market rates. These other factors include (but are not limited to) current core rate levels, current deposit ($) levels, loan demand or loan growth pressure, expectation of deposit disintermediation, etc. When you perform the check-up ask, “are the current beta factors used by the model consistent with history, why or why not?”
You can review historical data to validate or substantiate many different assumptions, here are a few more common ones:
Again keep in mind that historical behavior is not necessarily indicative of future behavior. Just look at the miserable job that the industry’s done in predicting the credit quality problems we’ve experienced recently. Commonly used “best guess” industry prepayment speeds turned out to be quite wrong too. But history is always a good place to start. You need some perspective.
2) Specific tests
The other part of the check-up involves running specific tests. Each test is designed to demonstrate that the tool is indeed “modeling it correctly”. To continue the check-up of the core deposit beta factors we would rerun the earnings simulation using more volatile - then less volatile factors to see if the results are appropriately impacted. You need to be a little careful here however. Don’t go crazy with running too many different or overly complicated tests. You can over do it. Just like in medicine you can’t test for everything (unless you’re Dr. House).
One popular test we run these days, and a good illustration of the “check-up”, is what we commonly refer to as the “floor test”. That’s not an official name, it’s just we our analysts call it. Given the unusually low interest rate environment of today, many banks have been adding floors to the loans in their portfolio. In fact a reasonably large portion of existing loans are currently “at their floor”.
If you know your bank’s loan portfolio has a substantial number of floors, there’s an easy way to check to make sure the model is representing them properly. First take a look at the earnings-at-risk line item for the loan portfolio in question.
In this example the change in income from commercial loans changes by –10% rates down versus a +12% change rates-up. The smaller change given rates down seems to indicate that floors are being modeled, but it’s by no means obvious. The base forecast used includes many assumptions that could be getting in the way of observing the floor behavior. First we’re modeling prepayments. Second the forecast includes assumptions for adding new balances to the portfolio.
In order to isolate the behavior of the floors we can change the model to eliminate prepayment and new dollar assumptions. We may call this a “run-off” scenario. In the Run-off scenario it’s much easier to see that floors are in-fact being modeled. There’s little change in the amount of modeled income given a rate shock down. The change we do see is from the few variable rate loans that don’t have floors.
There are many different specific tests you could design to make it easier to observe specific behaviors. Here are a few common ones that use the same “run-off” forecast as the starting point:
- Bond portfolio calls (or lack of calls)
- FHLB convertible advances
- Early withdrawal of CDs
To extend the medical analogy, you should probably perform a “check-up” test or two on a regular basis (possibly every other year). Remember that if you don’t observe the expected behavior it doesn’t necessarily mean that the model “is broken”. It could mean that the data being fed into the model has incorrect or missing information (see Inspection).
- Further information about core deposit beta factors: -
(Here’s a link back to the start of this series of posts on back-testing.)
