Several weeks ago the Maryland chapter of FMS asked if I would be interested in presenting to their group. They wanted me to cover a hot topic in asset/liability management and modeling. While several issues like credit quality, liquidity, capital, and interest rate risk (and oh yea, the bad economy in general) top the list, there’s a specific issue that I think has been bugging our clients for several years. The issue is model back-testing.
Ever since OCC Bulletin 2000-16 (Guidelines for Model Validation) was released the pressure has intensified to back-test the models we use to measure interest rate risk (and liquidity risk). All of these communications from the examiners mention model back-testing in one way or another:
- March 2010 – Joint Policy Statement on Funding & Liquidity Risk Management
- January 2010 - Advisory on Interest Rate Risk Management
- December 2006 - Commercial Real Estate Lending Joint Guidance
- May 2002 – OCC Unsafe and Unsound Investment Portfolio Practices
- May 2000 – OCC Guidelines for Model Validation
However, given the amount of communication, there’s a surprising lack of clarity about just what a back-test is. Don’t get me wrong I think most folks have a notion of what “running a back-test” means. But if asked to tackle the back-test problem they’re not sure where to start. (side note: The industry can’t even agree on the appropriate word(s) to use. Is it “backtesting” or “back-testing”? Popular media seems to use the one word. I’ve seen many other places use the hyphenated word. I’ll stick to “back-testing” since I also talk about stress-testing quite a bit. The hyphenated usage seems to make more sense.)
For instance if I asked you to think about some words or phrases when you hear “run a back-test” what comes to mind? Here are just a few words we hear from clients when they call and ask us for help with back-testing a model:
- Validation, Review, Independent, 3rd Party, Variance, Audit, Accuracy, Verify, Test, Right/Wrong, Inputs/Outputs, Assumptions, Stress-test, etc.
It probably wouldn’t shock you to learn that I’ve seen this sentence regarding back-testing. It uses eight of the word listed above:
A bank should audit the model by obtaining an independent third party validation or review which periodically back-tests the model inputs, assumptions, and outputs.
That sounds nice and official doesn’t it? Well ok, I made it up. I didn’t actually find this sentence in anything I’ve read, but that sounds like examiner and audit speak. The problem is that it really doesn’t give you a place to start. How do you go about “running a back-test”? What is the first step?
Over the next several weeks I’m going to run a series of posts that will be a practical guide to model back-testing. In fact that was the title of my presentation to the Maryland chapter of the FMS this week. My 50 minute presentation seemed to be well received. I’d welcome your comments on each of the upcoming posts. I’ll start here – Seven ways to back-test your model.
Comments