Manage your bank, not your model

Projection accuracy

Every year at this time we’re inundated with articles that either offer predictions for the year to come, or a look back a past predictions.  The look back is often especially entertaining to data geeks like me.  More often than not the predictions are wrong (some more wrong than others). The fact is we’re generally terrible at predicting the future.  However, every so often someone will get it right and we’ll believe for a moment that this person has found the “right” model to successfully predict future events.  That moment in the sun usually doesn’t last very long.

In the world of bank risk management the fascination with modeling exposures is no different. I’ve written about this frequently over the past few years including my series on “Seven ways to back-test your model”. There is a constant quest to back-test and improve the predictive capabilities of our models.  Recently the standard-bearers for this quest have been regulators, accountants, and auditors.  They have intensified their focus on back-testing and their attention has turned to the Comparison back-test in particular. 

The comparison back-test can be a fantastic way to learn more about the usefulness of a risk model.  Sadly however most reviews of such comparisons boil down to weather the model was “right” or “wrong”.  That’s too bad because most of time (if not all the time) the model is wrong and there usually isn’t a way to change it so that it will be “right”.  The best we can do is learn and improve, but we have to toss aside this notion that we’ll somehow fix our risk models and get the “correct” answer.

In our quarterly A/L BENCHMARKS report we deliver to clients a short report that shows the comparison back-test.  We’re often asked, “how close the should the forecast be?” Reviewers want to know if there is some sort of tolerance limit that will allow us to judge the forecast to be “right” or “wrong”.  My first response to such questions has always been – no. The forecast will almost always be wrong.  The value of the comparison back-test lies in the process, not in calculating the final difference.  However I have to admit that I am intrigued by differences though.  How far off are the forecasts that clients submit?  Each quarter they are using either a static or dynamic forecast to model their interest rate risk sensitivity…how accurate is the forecast?  I know they’ll be off, but by how much?  I’m most curious about differences when market conditions are changing versus when they are rather stable.  After all we’d like our models to be the most accurate when times are changing.

I collected the forecast projection data from over 200 of our bank clients.  Each quarter we use either a static (flat balance sheet) or dynamic forecast to run their earnings stress-test.  For each bank I compared their projected Total Assets amount (from one year ago) to the actual Total Assets amount then calculated the difference as a percentage of the projected amount.  Overall the differences are smaller than you might expect, but remember this is a projection of Total Assets.  It is probably the easiest overall target to hit compared to more volatile performance measures like margin, net income, and other specific balance sheet categories like total loans or total deposits (even the mix within these categories would be more difficult to project as accurately.)

Here is a quick overview of each environment.  Note that when I talk about the “average” projection difference I’m referring to the median which is technically not the average, but for this analysis using the median makes things a little clearer.

When rates were rising
In the second quarter of 2005 the average Fed Funds rate was 2.94%.  By the second quarter of 2006 the average was 4.91%, an increase of nearly +200bp.  On average banks using a static forecast were 3.20% too low on their projection.  The typical range for static forecast projections was from –1.4% up to 9.81%.  Banks using a dynamic forecast were an average of 2.06% off with a typical range of –1.75% to 6.80%. 

When rates were falling
Between the second quarter of 2007 and the second quarter of 2008 the average Fed Funds rate fell by –316bp from 5.25% down to 2.09%.  On average projections were slightly farther off.  Those banks using a static forecast were 4.70% too low, with a typical range between 0.09% and 11.45%.  The average projection difference for banks using a dynamic forecast was 2.34% with typical differences ranging from –3.01% up to 7.15%.

When rates were stable
From second quarter of 2011 up to the second quarter of 2012 while long-term rates dropped quite a bit, short-term rates remained quite stable (and historically low.)  The average Fed Funds rate was at or below 0.25%.  The average projection difference for banks using a static forecast was 2.41%.  For banks using a dynamic forecast the average difference was slightly closer at 1.73%.

What’s the takeaway?
As I mentioned before, we’re looking at projections of total assets which tend to be a little easier to predict.  For most banks even in the best of times Total Assets doesn’t grow by more than 10% to 15%.  I think the takeaway here is that if you are consistently running a comparison back-test on your model’s one-year projection, you should aim for a Total Assets difference of no more than 4% to 6%.  If you are outside of that range on a consistent basis there’s probably some important balance sheet activity that your model is missing.  Occasional quarters that show a difference of higher than that will happen, but don’t sweat them.  Every now and then there’s bound to be an unusual quarter or two.

The Fed’s SWAG

I don’t think anyone was surprised by the Fed’s message last week.  It looks like we’re in for more of the same.  Clearly though the general consensus has changed from, “tightening sometime in 2014,” to “tightening sometime in 2015,” if you choose to believe it.  While I do believe the FOMC participants are smart and well informed (and have good intentions), they’re still no better at predicting the future (especially two or three years into the future) than the rest of us are.

The most interesting thing to me is that at least one FOMC participant, possibly more, projects some pretty dramatic rate shocks in our future. Just using the graphs shown in the minutes it’s hard to tell exactly because the individual projections aren’t identified across each time period.  At a minimum however one projection has the target changing by +125bp by the end of 2013.  In 2014 things could get more volatile, again one projection could be as little as a +125bp change, or it could be as much as +275bp.

My advice for measuring your bank’s exposure to such potential interest rate changes?

• Stick to basic shock analysis using 100, 200, 300 and 400bp parallel changes.  Evaluating anything different will get you into the business of trying to predict rate changes.  And as we’ve seen from the Fed over the past eight months (see January, April, and September minutes), even these smart folks can’t nail down their prediction.

The cover-up

There’s a book called Interest Rate Risk Management that I’ve mentioned on this blog before.  In it the author dedicates an entire chapter to “Income Simulation.”  Essentially it’s an in-depth look at measuring IRR exposure using a future projection and then stress-testing or changing rates: i.e. earnings-at-risk.  Throughout the book the author does a great job of presenting not only the benefits of using such measurements but the drawbacks as well.  This chapter is no different.  In fact the last part of the chapter labeled, “Disadvantages of Income Simulation,” lists no less than six very specific problems with measuring earnings-at-risk.  The second item on the list is the following:

Assumptions can intentionally or inadvertently understate risk exposures
…income simulation permits rate risk managers to reflect potential balance sheet changes into their forecast.  This can be both a strength and a weakness…assumptions used in the simulation may actually mask some or all of the current rate risk exposure. (emphasis is mine)
Chapter 3, Interest Rate Risk Management by Leonard Matz, 2003, Sheshunoff Information Services

Actually the potential impact of future assumptions is at the core of the static versus dynamic forecast debate.  But “understating risk” and “masking current exposures”? What does this mean?  How much of an impact can it have?  More than you might think.

This was very clearly illustrated to us by a client’s recent forecast and IRR stress-test.  Normally this client provides a budget projection to use as their base-case forecast (i.e. a dynamic forecast).  It’s an aggressive balance sheet growth projection showing anywhere from 5 to 10% growth in various loan and deposit portfolios over the next year.  The stress-test shows the bank is exposed to rising rates in the short-term. The shock up shows a potential decrease of -5.32% in the bank's net interest earnings.  We also ran the stress-test using a static one-year forecast.  The static forecast projects no balance sheet growth but rather projects a flat “static” balance sheet (only volume rolling off is replaced).  After applying the rate shock-up we see a noticeably different result: a potential decrease of –9.65% in the bank’s net interest earnings; it’s almost double (see results below).

Many of you are probably looking at these results and saying, “so what?”  It doesn’t look like that big of a difference.  After all the latest interest rate risk peer data shows that the average Net-Interest Earnings at risk for banks $100M to $300M is around –8.2%.  Is the difference here that big of a deal?

Well, it’s pretty obvious in this example that a dynamic forecast projecting balance sheet growth can easily mask or cover-up potential risk.  That’s pretty significant especially given our current uncertain economic environment.  Which stress-test should the bank pay attention to?

What if I told you this bank’s policy limit is –10.0%? Both measurements are within the limits, but one is clearly much closer to the limit than the other.  What if I told you that while the bank’s commercial lenders were confident they would hit their targets (aren’t they always?), others on the management team weren’t so sure.  The growth definitely combats the pressure on margin, but what if it doesn’t materialize? 

In this case (as is the case with most of our clients) I think the static forecast is a much better stress-test.  It helps eliminate many of the assumptions that can potentially “cover-up” risk.

Skeptics on the Federal Reserve Board

Here’s a link to the video and minutes from the Open Board Meeting on June 7, 2012.  The purpose of the meeting is to discuss Basel III.   There is one question from Governor Raskin about the proposed regulatory requirements to use internal models to calculate risk-based capital:

…a dimension that strikes me, is a currently unavoidable deficiency in these proposed rules. In particular, I note that under the proposed rules we are permitting large, complex banks to rely on their internal models to ensure proper liquidity. We are proposing placing additional prudential requirements on these internal models, and these revisions, as I understand them, include requiring banks to take into account stressful conditions when modeling market risk capital requirements.

I applaud the intent of the staff to address some of the shortcomings inherent in a reliance on the internal models of banks, but I do remain skeptical. I think we should proceed with such a reliance cautiously. Or in other words, trust but verify. As a supervisory matter, I would like to see our examiners make sure that these internal risk models are keeping pace with the complexity that certain business lines or trades are presenting. If the internal risk models fall short then the activities must be flatly and out right prohibited. End of story.

My sense, by the way, is that our large and complex financial institutions have struggled with their ability to keep their internal risk models closely approximating and mirroring the risks that these institutions are embracing. Instead, we need assurances that the internal risk models are being updated regularly by the institution's chief risk officer and then examined regularly by the institution's examiners. To do this exercise correctly, the institution needs to assure that revenues and the risks generated by these revenues in various business lines are transparent so that stakeholders can provide input necessary to revise the internal risk models and assure their credibility.

So this is now the question. Here it is. Do we current feel comfortable from a supervisory perspective permitting banks to rely on their internal models for capital purposes? And to what extent will the proposed enhancements to these internal models address or not address our perception of any shortcomings in the bank's modeling processes?
p13, Open Board Meeting Minutes, June 7, 2012

The response to her question comes from Mr. Gibson (Division Director, Banking Supervision & Regulation):

...in the past, it's already our current practice to do that, and we expect that we will continue to do that in the future...We share the concern. The [Basel III] proposal moves us in the right direction, and it's definitely going to continue to be a supervisory focus.
p14, Open Board Meeting Minutes, June 7, 2012

I don’t know…I’m still very skeptical.

(hat tip to my colleagues at CSBS for linking me to these minutes.)

Tuned out

It’s no secret that bank examiners are pressuring community banks to step-up their interest rate risk measurement processes.  Armed with the new “requirements” (and I use that term loosely) that were outlined in the 2010 IRR Advisory, examiners are asking banks to run an ever increasing number of IRR stress-tests.  In additional to what I call the basic shock analyses, they want to see shocks with both static and dynamic forecasts.  They want not only 1-year projections but 2 and 3-year projections as well.  Rate ramps are also popular, and they throw in a request for a few non-parallel shifts just for good measure.

I’ve been warning anybody who will listen that this practice is slowly blinding community bankers.  With so many different stress-test scenarios and reports to comb through they’re beginning to ignore the information altogether.  I’m not saying they should, and I’m not trying to make excuses.  It’s just human nature.  The deluge of new IRR “requirements” is fueling some pretty serious information overload.

These requests for so many additional stress-tests reek of the “by the book” approach to examinations that the new Kansas City Fed Chief warned us about.  Bankers are starting to see the myriad of IRR stress-tests as just another item on the regulator’s checklist.  They are beginning to ignore the results altogether.

A telltale sign that this is happening is when the modeling requests themselves become, well (there’s no other word for it)…ridiculous.

We provide to all of our clients a 400bp shock analysis.  Or let me clarify, we provide a +400bp shock analysis (i.e. plus 400bp rates up stress-test).  We don’t currently run a –400bp shock (i.e. minus 400bp down).  In our current environment such a shock makes little sense.  As of June 30 the average 1-month rate was 0.05%, and the average 30-year rate was 2.70%.  In a –400bp shock down the entire curve would reach a floor of 0.00%.  That’s a pretty flat curve indeed!  So many modeling assumptions are tied to these rates: repricing spreads, new dollar rates, prepayment tables, early withdrawal tables, decay assumptions, discount rates, etc.  I can tell you with certainty that the model (ours or anybody else’s) would likely deliver some pretty strange results. I already don’t even try to analyze too deeply the minus 200bp and minus 300bp shocks for the same reason: the curve is basically flat…at 0.00%.

So, back to the ridiculous part.  Despite the fact that a minus 400bp stress-test will most likely produce strange results, we’re being asked to run them anyway.  “Our examiners are reviewing our IRR reports and they say we’re missing the –400bp shock  Can you please provide that?”  Since last fall easily a third of our customers have contacted us with this request.  I’m still caught off-guard by the question.  What is the value in looking at such a scenario?  A –400bp is such a classic case of garbage-in-garbage-out (GIGO) yet neither the client nor the examiner seems to care.  It is a clear illustration to me that folks are beginning to tune-out this information.  IRR stress-testing can be a very valuable management tool.  For many community banks however it’s becoming just another examination check-list item.

Just to Clarify

This morning I received this short email from a colleague:

Brad, I’m guessing you’ve seen this:

http://www.fdic.gov/news/news/press/2012/pr12054a.pdf

What they’re saying is, community banks don’t need to run CCAR/SCAP-type tests, but they’re not saying anything about other forms of stress testing, right? What was your take on this?

The press release is a “Statement to Clarify Supervisory Expectations for Stress Testing by Community Banks” from the Fed, FDIC, and OCC.  And that’s correct, they are NOT talking about the additional IRR stress-testing, or liquidity stress-testing, or credit stress-testing.

My take is that this press release is nothing short of a political/defensive move by the regulators to combat the recent noise the ICBA and ABA have been making about the deluge of regulation hurting community banks.  Everyone thinks that all the "new" requirements (which banks are complaining about) are bundled into Dodd-Frank Act…they're not. 

This press release is a gem (for regulators anyway) because they can say, "see? we're not burdening community banks with any of this Dodd-Frank stuff."  But still they can, for instance, continue "enforcing" all the additional stuff "required" by the 2010 IRR Advisory.

It's all buried in the last line of the press release:

"The agencies note that such existing guidance, including that covering interest rate risk management, commercial real estate concentrations, and funding and liquidity management (among others), continues to apply."

Sigh.

That was fast

After reading through the Fed’s latest economic projections (April 2012) I’ll concede that the general consensus among FOMC members didn’t change dramatically since January.  Most members still aren’t expecting a move until 2014.  However, we’ve already seen that some of the Fed’s past expectations have been completely off-the-mark.  These latest projections show a target fed funds rate that’s even more volatile than the projection released just three months ago.  Here are a few changes that I noticed:

• The final BIG change may come sooner
  In a recent post I talked about the dramatic +400bp shock to come – maybe.  That change still might occur, but at least two FOMC participants think it might occur sooner than they originally predicted.  The January predictions showed two policy firming holdouts in 2016.  April’s projections show all policy firming actions happening no later than 2015.
  .
• At least one member sees a +100bp change in the near future
  Back in January none of the members predicted a change of more than +75bp for 2012.  Now at least one participant predicts a change of +100bp by the end of 2012 (only 8 months to go!)
  .
• The change in 2014 might be dramatic
  Since the documentation doesn’t track individual member predictions it’s hard to put together a clear picture of the 2014 forecast.  But by the process of elimination you can see that at least one member thinks Fed Funds will change between +175bp and +250bp in 2014.  Analysts can puzzle over rate ramps if they want to, but +175bp to +250bp in one year sounds much more like a rate shock to me.

Will the general consensus be right?  I have no idea.  Even if they are it still looks like it could be a really bumpy ride.

Should institutions with non-complex balance sheets use earnings simulations < from the 2012 FAQ

Here’s the fourth question and answer text from the FAQ.  It’s one of the shortest parts of the entire document. I’ll share my comments after.

Question from the FAQ:
4) Should institutions with non-complex balance sheets use earnings simulations to measure risk to earnings?

Answer from the FAQ:
All institutions are encouraged to use earnings simulations. Advances in technology have made simulation modeling more accessible for all institutions. Financial regulators recognize that some institutions with non-complex balance sheets may have minimal levels of embedded options in both assets and liabilities, such as products discussed in response to question #3, and have few or no derivatives. In these limited cases, onsite financial regulators assess management’s alternative measurement processes to analyze the institution’s less-complex risk profile. Based on this assessment, regulators may determine that a less sophisticated measurement process may adequately measure earnings at risk.

What a cop-out.  Here was a great opportunity to banish the gap report as an interest rate risk measurement tool, but they didn’t take it.  The regulators could have easily said, “Yes, you must use an earnings simulation.  You can no longer just use a gap report to measure earnings sensitivity.”  I have no idea why they just didn’t say that.

Over and over regulatory publications on IRR use the term, “non-complex balance sheet” – this FAQ is no exception.  “Financial regulators recognize that some institutions with non-complex balance sheets may have minimal levels of embedded options in both assets and liabilities.”  Without getting too snarky I‘d like to know - which banks exactly?  Really? Are there noticeable numbers of banks out there that have little or no optionality on their balance sheet?  I don’t believe it for a second.*

It’s easy to make optionality sound complicated if you use examples like, “collateralized mortgage obligations, step-up notes, callable agency bonds, convertible Federal Home Loan Bank borrowings, and alternative certificates of deposit” (as discussed in question #3).  But step outside the investment and wholesale funding portfolios and you find optionality pretty much everywhere on the average bank’s balance sheet:

• Commercial loans refinance and restructure
• Residential loans prepay and refinance
• Consumer loans prepay (especially those with “same as cash” periods)
• Non-interest checking accounts rollover and decay
• Interest Bearing Checking and Money Markets reprice, rollover, and decay
• CD’s withdraw before maturity

Aren’t all of these products quite common on the “non-complex” bank’s balance sheet?  Optionality like this can’t be adequately captured in a gap report because a gap report is a single scenario, point in time measurement.  Before this post morphs into a rant against the gap report let me just refer you to my blog category “Doh!, the gap report”.

So for question #4 from the FAQ, should institutions with non-complex balance sheets use earnings simulations?  The answer should have simply been, YES!

---

* I don’t solicit many comments from readers of this blog because I’m usually deluged by spam.  In this case I’ll make an exception.  Please email me if you know of a bank that fits the profile of “non-complex balance sheet with minimal levels of embedded options.”  I’ll be surprised if I compile a list of more than 10 banks.  Send your emails to btolson@olsonresearch.com

The Fed’s stress-testing language sets the wrong expectation

It’s been several weeks since the Fed’s stress-test results were released.  According to some sources, “investors are still puzzling,” over the inputs, methodologies, and results.  I’m not very surprised by this.  I think it illustrates a big flaw in making the results of the bank stress-tests public.  I’ve written about this before (here and here). 

When most people hear “stress-test” they think it sounds very scientific and they expect variables to be concrete – like temperature, pressure or weight.  They expect inputs to be quantifiable and tangible.  The CCAR however uses many assumptions that are based on estimates.  This is why many people are “puzzling” over the information released.  For instance, one key parameter, a projected loss rate, is a modeling assumption based on an estimate.  Actual historical data may have been used to come up with the estimate, but it’s still an estimate. 

Suppose auto safety engineers just estimated the force and speed it would take for air bags to deploy. What if they just gathered the data from previous accidents, manufactured a new air bag and declared that, “based on past performance,” these new designs “should” deploy.  Wouldn’t you want them to actually stress-test a prototype?  You know, run a few thousand cars into a wall just to be sure?

I’m not suggesting this type of thing could even be done for bank stress-testing.  Nor am I suggesting bank stress-tests be as rigorous as auto-safety stress-tests.  But when the Fed uses terms like “stress-test”, “parameters”, and “pass” or “fail” they are using language that suggests a much more clearly defined scientific process.  However those knowledgeable about the CCAR will tell you – it’s just a model, not a prediction. 

The problem is the Fed’s choice of language has set the wrong expectation.

Four things I don’t like about the Fed’s most recent Stress-tests

I just finished another stop on my “Stress-testing Dos and Don’ts” road show.  This week I was in San Diego at the California DFI’s Annual Staff conference.  Inevitably during the Q&A I was asked for my opinion on the Fed’s latest round of stress-tests.  Here are four things I don’t like:

1) The metric under the microscope was a risk-based capital measurement and not just a plain leverage ratio. 
It seems to me that since 2008 there’s been enough debate about the appropriate risk-weights for various asset categories to cast some doubt on the usefulness of established risk-based capital levels.  If the Fed’s severe stress scenario is going to change a myriad of other intangible economic variables shouldn’t they consider the validity of the risk-weightings too?  Using a simpler equity to asset (un-risk-weighted) ratio would have been a better alternative.

2) The test appears to only measure credit risk with no apparent consideration given to interest rate risk.
At first this seemed odd because recently examiners have been pressuring banks of all sizes to run an unprecedented number of additional interest rate stress-tests (higher shocks, twists, steeper, flatter, ramps, etc.)  Ever since the 2010 IRR advisory the number of examiner motivated requests for additional stress-tests has skyrocketed.  The Supervisory Stress Scenario barely changes rates or the shape of the curve at all (see graph).  Then I remembered that it’s the Fed running these stress-tests.  According to their predictions rates aren’t going anywhere for a while.  They may be correct, they may be wrong.  But aren’t these stress-tests supposed to try and give us a glimpse of what might happen under circumstances we can’t really foresee?  I think it’s a little bold to predict things like 13% Unemployment, and a dive in the DOW down to 5668 without any noticeable changes in interest rates.  I know I would predict bigger changes in rates.  But then again, I’m no economist.  However I’m certain I’m just as bad at predicting things as they are.

3) We only know the macro level variables.
Perhaps I’m among only a few that care since it’s my job to run an A/L model for over 250 community banks.  Experience has shown me that some of the most critical estimates made in A/L modeling are (in no particular order) interest rates, prepayment speeds, credit spreads, default rates, future volume/mix changes, decay rates, beta factors, and early withdrawal rates - just to name a few.  Obviously someone can make the case that macro variables like Real GDP Growth, Disposable Income, Unemployment,  and the Commercial Real Estate Price Index impact these critical estimates – but how exactly?  There seems to be a HUGE black box between the macro parameters shared by the Fed and the more tangible parameters necessary to actually run a cash-flow model.  These black-box relationships are the things that need to be vetted and validated because last time I checked (in fact, every time I check) there is very little unanimous agreement among economists about what these macro parameters really tell us.

4) The “results” were made public. 
Don’t get me wrong, I’m a big believer in making sure the regulatory process is as transparent as possible.  But by comparison the regulators already share more data and financial information about banks than any other industry I can think of.  You could easily bury yourself with data by simply downloading it from the FDIC, Fed, FFIEC, or the SEC.   If investors, or depositors, or taxpayers (or whatever audience this stress-test was trying to speak to) really wanted to learn something about these 19 largest banks there is already a mound of data ready and waiting for them to dig into.  Unfortunately your average Joe won’t dig through data like that.  He’s far more likely to want to know which banks “passed” or “failed” some sort of test.  Which is exactly how these results are being interpreted by the average Joe and by the popular press.  Just look at the defensive responses coming from the banks that “failed”.  They are trying to educate everyone about how much more complicated the process is and that a pass/fail grade really isn’t reasonable (and I agree, it isn’t reasonable).

Sadly, I think the Fed did recognize the potential for these results to be misinterpreted.  Right in the introduction they say, “The Federal Reserve’s projections for the 19 BHCs under the Supervisory Stress Scenario should not be interpreted as expected or likely outcomes for these firms, but rather as possible results under hypothetical, highly adverse conditions.”  But I think there was concern about more than just a mere “chance” of misinterpretation.  Here are three similar statements that appear in the document (presented here in order, bolded and underlined for emphasis just like they were in the original document, not added by me):

• From page 3:
  …it is important to recall that the Federal Reserve’s stress scenario projections…are not forecasts… (no emphases in the original)
• From page 8:
  It is important to note that the Supervisory Stress Scenario is not a forecast… (bolding is original, not mine)
• Then finally from page 43:
  It is important to note that the Supervisory Stress Scenario is not a forecast… (bolding and underlining is original, not mine)

Nice try, but unless you’re a real data geek like me, or someone with an above average financial background all of this is just “blah, blah, blah”.  No amount of bolding or underlining is going to change that.  Sadly, the only thing most people want to know is, “who passed and who failed?”