Model Secrecy and Stress Tests

Following the 2008 financial crisis and the Dodd-Frank Act, bank stress tests have become a cornerstone of bank regulation. An important but unexplored issue is the optimal level of transparency of the supervisory stress tests models that are used to project bank capital when conducing the tests.[1] Until 2018, the Federal Reserve provided only a broad description of its stress test models, but in 2019, it implemented a new set of rules that provides more transparency. Under the new rules, the Fed will provide more information on certain equations and key variables, and it will also illustrate how its models work on hypothetical loan portfolios. Yet, even under these new rules, the Federal Reserve will not fully reveal its models.

An important reason for not revealing the models is to prevent banks from gaming the test—i.e., taking actions that enable them to pass the test without actually reducing risk. Indeed, in a speech from September 26, 2016, Former Fed Governor, Daniel Tarullo, said that “Full disclosure would permit firms to game the system – that is, to optimize portfolio characteristics based on the parameters of the model and take risks in areas not well-captured by the stress test just to minimize the estimated stress losses.” However, banks have complained about model secrecy, claiming that even their best efforts to prepare for a test could result in unexpected and costly failure. These claims cannot be ignored, particularly given evidence that regulatory uncertainty causes banks to reduce lending.

In a recent working paper, we present a theoretical framework that allows us to evaluate the costs and benefits of revealing the models to the banks before the test. Our setting has two main forces. Not revealing reduces gaming, but it can also induce banks to reduce investment in socially desirable assets.

In our setting, the bank has better capacity than the regulator to identify and measure risk, but there is a conflict of interest between the bank and the regulator: the bank wants to take more risk than is socially desirable. As a consequence, if the regulator discloses the stress test models, the bank “games the test” by overinvesting in assets for which it knows that the regulator’s models underestimate the risk. Essentially, when the regulator reveals a model that shows that it does not consider a certain asset as very risky, it gives a “green light” for the bank to invest, so the bank may end up investing even when it knows that the asset is harmful to financial stability. Not revealing the regulator’s model can mitigate this problem, but it opens the door to a new problem: The bank sometimes does not invest in risky assets even though it knows the assets are good from a social point of view. This underinvestment arises because although the bank knows that an investment is socially desirable, it may be worried that the regulator’s model does not measure risks accurately, which could result in the bank failing the test. Revealing the test mitigates this problem.

Hence, there is a tradeoff. One implication is that revealing is preferred to not revealing if the bank’s concerns about failing the test outweigh its appetite for the risky asset. Such concerns may arise because of potential market reaction to test failure, the cost of altering the bank’s capital plan, or the reputational cost for bank managers.

With our framework, we show that in some cases, the regulator can solve the underinvestment problem by making the test easier. Specifically, the regulator can reduce the threshold for passing the test. If it can do so, not revealing is preferred. However, this does not work if banks are very different from one another and the regulator must apply the same passing threshold to all banks. In this case, the regulator cannot calibrate the passing threshold to induce socially desirable investment by everyone, because making the test easier to alleviate concerns by banks that are too cautious (i.e., those with a high private cost of failing the test) could induce the “reckless” banks (those with a low cost of failing the test) to take excessive risks. The implication of this is that if banks are very different from one another and the regulator must apply the same passing threshold to everyone, revealing the test is preferred to not revealing.

We also analyze the more complicated scenario in which the regulator can reveal only partial information (e.g. whether estimated losses from an asset, according to the regulator’s model, are above or below some threshold). Our main result for this scenario is that in some cases, some disclosure is optimal even if the regulator can set the passing threshold optimally. That is, partial disclosure could be optimal even if the regulator is not obliged to apply the same threshold to all banks.

This result reflects the general idea that in some cases, it is optimal to combine more than one regulatory tool to achieve a desired outcome. In our setting, the regulator has two tools to induce the bank to reduce risk. First, it can make the test harder by increasing the passing threshold. Second, it can provide partial information. To see how partial disclosure works, suppose for example that the regulator promises to tell the bank whenever the model predicts a particularly high value for a specific asset, which essentially gives the bank a green light to invest. Then if the regulator does not give a green light, the bank infers that the regulator thinks the value is relatively low, and hence the bank does not invest in that asset. Hence, partial disclosure can help reduce risk. However, partial disclosure has a social cost: if the regulator reveals a very high asset value, the bank may end up taking excessive risk.

Given this social cost of partial disclosure, why not just use the first tool of adjusting the passing threshold, while disclosing nothing about the model? The reason is that the first tool of adjusting the passing threshold also has a social cost: if the regulator sets a high threshold for passing the test, then in many cases he ends up failing the bank even when he thinks the bank’s investment is valuable for society. We show that if the bank’s private cost for failing the test is high, it is optimal to set an easy test and disclose nothing about the model. Conversely, if the bank’s private cost for failing the test is low, it is optimal to set a hard test and combine it with partial disclosure. In the latter case, without any disclosure, the test would need to be even harder.

Finally, our model delivers additional policy implications. First, policy makers have suggested that if the Fed’s models were to be published, then in order to counteract gaming, the minimum capital requirement would need to materially increase. Our model suggests that this conclusion is only partially correct. In particular, for some cases, the optimal passing threshold when the model is revealed is lower than that when it is not revealed. This could happen, for example, if the bank’s cost of failing the test is low, so when not revealing the model, the regulator needs to set a very high passing threshold to reduce overly risky investment.

Second, policy makers have expressed the concern that disclosing the Fed’s models could increase correlations in asset holdings among banks subject to the stress tests (i.e., the largest banks), making the financial system more vulnerable to adverse financial shocks. An extension of our model would suggest that this concern is also valid if the Fed just illustrates how its models work on hypothetical loan portfolios, as it does under the new rules. In particular, the proposed hypothetical portfolios could serve as benchmark portfolios into which too many banks invest, leading to correlated investment. In other words, a bank will overinvest in portfolios, for which there is less uncertainty as to how the regulator will measure risk, and it will underinvest in idiosyncratic non-portfolio investments, for which it is unclear how the regulator’s models will work.


Yaron Leitner is a Visiting Associate Professor of Finance in the Olin Business School at Washington University in St. Louis.

Basil Williams is an Assistant Professor in the Department of Economics at New York University.

This post is adapted from their paper, “Model Secrecy and Stress Tests,” available on SSRN.


[1] A related issue, which has been more widely explored, is the transparency of the test results. See for example Goldstein and Leitner (2018) and Goldstein and Leitner (2020)

Leave a Reply

Leave a Reply

Your email address will not be published. Required fields are marked *