To paraphrase a statement by pro-gun lobbyists: “guns don’t kill people, people kill people”. Despite the provocative nature of this sentence, we can draw a parallel between it and model risk: that is, models are only dangerous in people’s hands.²² 2 Note that the developer is often the end user. A misplaced dependence or overreliance on a particular model – or the ego of the person conceiving it – may lead to dramatic failures.

The Federal Reserve (2011) and, more recently, the European Banking Authority (EBA) have officially addressed model risk, a topic that has been discussed by practitioners for a while but has only been dealt with partially. Indeed, in Article 1 of the EBA Regulatory Technical Standards (RTS) (European Banking Authority 2014a), the regulator includes model risk within the scope of operational risk, as can be seen in the following statement: “The competent authority shall verify that an institution has included legal risk, information and communication technology risks, as well as model risk, within the scope of operational risk with the exclusion of other kinds of risk.” The fact that model risk (Morini 2011) is included “within the scope of operational risk” is not surprising, considering that operational risk is defined as “the risk of direct or indirect loss resulting from inadequate or failed internal processes, people and systems or from external events”. Operational risk includes legal risks (as well as various boundary risks) but excludes reputational risk, and it is embedded in all banking products and activities (Basel Committee on Banking Supervision 2001b). Indeed, the failure of a model is included in the definition above, as model risk could arise from people, processes or inadequate systems or methodologies, as described in what follows.

Although authorities recognize model risk as a major problem, the way it is handled may differ from one regulator to the next. Indeed, while both the Federal Reserve (2011) and the EBA RTS (European Banking Authority 2014a) tend to cover a similar scope, the latter focuses on unexpected issues related to choosing the wrong model as well as unidentified limitations. Indeed, model risk arises when all of the characteristics, properties and assumptions of a model are unknown, not understood or voluntarily disregarded; in other words, when a model is used blindly. However, this approach is less devastating when compared with someone making an impulsive judgment call. We advocate that a scenario in which a simple model is used is still better than one in which no model is adopted. Therefore, a real and dramatic risk that any institution can face is misunderstanding the domain within which a model is valid. We will not hide behind a truism such as “all models are wrong but some are useful” (Box and Norman 1987). First, saying that a model is “wrong” is nothing more than a tautology, because a model is an approximation of reality: it is not reality itself. Second, all models are useful; even if a model is not applicable in a given scenario, it can still be useful in deducing that the phenomena to be captured are different from those for which the model has been designed. In other words, we can discover, using the process of elimination, that a model is simply not applicable in a given situation. Applying a model forcibly when we know or suspect that it is not applicable is wrong, because it introduces a bias in the perception of the pattern we are trying to highlight; this may prove misleading to users and the model’s outcomes may be fallacious.

Model risk is far more abstract than its description as a quantitative problem would suggest. It requires us to acknowledge that our quantitative methodologies, our understanding of the universe and our reduction of the gap between academic papers and their implementation in financial institutions are currently limited. This problem starts with mathematics itself. By analyzing the opposing views of foundationalists (mathematics has a permanent, secure foundation), absolutists (mathematical knowledge and truth are eternal, objective and absolute) and those espousing new schools of thought – such as fallibilists, humanists, relativists and social constructivists – we can observe that mathematics, as a representation of conceptual structures, is perpetually evolving. In addition, according to Paul Ernest (2004), mathematical knowledge is invented, not discovered. Although that statement may seem controversial, it is, in our opinion, sensible to consider that models are based on assumptions: depending on the underlying assumptions, any kind of model can be created, even if some might be absurd with respect to reality. Indeed, a clear distinction has to be made between theory and practice, that is, the theories of mathematics and their implementation (eg, a risk measure such as value-at-risk (VaR) or expected shortfall (ES) and its estimator). Consequently, even if a model is meaningless with respect to its implementation in reality, nothing prevents the theories of mathematics behind it from being written down. It is simply the responsibility of users and practitioners not to use such models if they are deemed inappropriate. In the meantime, we need to acknowledge the possibility that the theories of mathematics that have been created are representative of an absolute truth that has not yet materialized, has not been captured by data and, consequently, is not empirically verifiable. Going even further, Tegmark’s mathematical universe hypothesis (MUH) (Tegmark 2008) states that the physical universe is mathematics, ie, physical existence equals mathematical existence and vice versa.

Although the essence of this concept seems identical, regulators around the world have different ideas about how model risk should be managed or covered. For example, US and Polish regulators require model errors or uncertainty to be covered by financial institutions’ regulatory capital. The Federal Reserve defines model risk as follows (Federal Reserve 2011): “The use of models invariably presents model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. Model risk can lead to financial loss, poor business and strategic decision making, or damage to a bank’s reputation. Model risk occurs primarily for two reasons:

• •

  The model may have fundamental errors and may produce inaccurate outputs when viewed against the design objective and intended business uses. The mathematical calculation and quantification exercise underlying any model generally involves application of theory, choice of sample design and numerical routines, selection of inputs and estimation, and implementation in information systems. Errors can occur at any point from design through implementation. In addition, shortcuts, simplifications, or approximations used to manage complicated problems could compromise the integrity and reliability of outputs from those calculations. Finally, the quality of model outputs depends on the quality of input data and assumptions, and errors in inputs or incorrect assumptions will lead to inaccurate outputs.

• •

  The model may be used incorrectly or inappropriately. Even a fundamentally sound model producing accurate outputs consistent with the design objective of the model may exhibit high model risk if it is misapplied or misused. Models by their nature are simplifications of reality, and real-world events may prove those simplifications inappropriate. This is even more of a concern if a model is used outside the environment for which it was designed. Banks may do this intentionally as they apply existing models to new products or markets, or inadvertently as market conditions or customer behavior changes. Decision makers need to understand the limitations of a model to avoid using it in ways that are not consistent with the original intent. Limitations come in part from weaknesses in the model due to its various shortcomings, approximations, and uncertainties. Limitations are also a consequence of assumptions underlying a model that may restrict the scope to a limited set of specific circumstances and situations.”

Moreover, it states the following: “Another way in which banks may choose to be conservative is to hold an additional cushion of capital to protect against potential losses associated with model risk.” The European Banking Authority (2014a) is reducing the scope of model risk by acknowledging some kind of institutional “free will”, as follows: “As a specification of the paragraph 5(4), the following events, and the related losses, shall be excluded from the scope of operational risk: (a) events due to wrong selection of a model, made through a formalized corporate process; and (b) losses caused by a pricing model where the potential exposure to the model risk had been previously assessed.” Therefore, we can differentiate between expected model risk and unexpected model risk: the former should be managed and the latter should be offset, controlled, hedged or covered by a capital buffer. This differentiation is really interesting, as it recognizes the fact that the risk of a model failing is intrinsic to its use; this contrasts with US regulation, which tends to imply that a “good” model should not fail at all. The capture of a loss related to unexpected model failure in capital calculations should be an incentive for risk managers to understand the models they use, to test these abstractions and to ensure the models have been properly challenged and appropriately monitored. This piece of regulation implies that model risk cannot be fully offset. A model could fail for various reasons, and such risk is inherent in the use of any model. This acknowledgement seems to be a sensible way forward, as awareness is the first step toward sound risk management.

This paper is structured as follows. In the next section, we focus on the root causes of model failure and their impact on a model’s outcomes. In Section 3, we present solutions to manage these causes. In Section 4, we present our conclusions.

As described in Collins Dictionary,³³ 3 See http://www.collinsdictionary.com/dictionary/english/dogma. the term “dogma” comes from the Greek word “dokein”, which means opinion or belief. It is defined as follows:

1. (1)

   a religious doctrine or system of doctrines proclaimed by ecclesiastical authority as true; or

2. (2)

   a belief, principle, or doctrine or a code of beliefs, principles, or doctrines.

Besides, the first synonym provided is “blind faith”.

This definition has been extended to principles laid down by authorities whose opinions are considered to be incontrovertibly true, such as those of philosophical schools of thought and political parties. As a dogma is the basis and the cornerstone of an ideology, altering or failing to consider it would mechanically affect either a system’s paradigm or the ideology itself. Moreover, an ideology should always be considered a dangerous thing, as it is a utopic goal that can be used to justify the most vicious behavior exhibited to achieve it (Pinker 2012; Fiske and Rai 2014). Another famous definition of dogma was provided by Steve Jobs at the University of Stanford in 2005. He defined dogma as “living with the results of other people’s thinking” and, therefore, as a break from innovation and creativity.

The concept of dogma is intrinsically related to the concept of faith. We are not trying to be controversial or provocative when we say that faith may not be compatible with questioning, as it relies on a belief that is anchored deep inside one’s being. When a model becomes a dogma, people use it without questioning its legitimacy or soundness. Unfortunately, when most practitioners are blindly using the same models, their use quickly becomes so-called best practice. It is interesting to note that regulatory expectations (with respect to models) mechanically catalyze and spread dogmas. Relying on dogmas may even affect market microstructure itself (price formation, volatility, etc) (O’Hara 1998) and engender systemic risk.

The main problem with a dogma is that it does not accept questioning. It is profoundly anchored in our culture and the way we do and understand things, and it structures the way we think and approach problems. As an illustration, Euclidian axioms (Euclid 1956) may be considered dogmas as soon as they are accepted without questioning. Consequently, the knowledge that an alternative theory such as non-Euclidian geometry (Bonola 2007) exists leads to the use of Euclidian principles in inappropriate situations (eg, cosmology), which may lead to dramatic failures. As a case in point, we quote Nin (1972), who said: “When we blindly adopt a religion, a political system, a literary dogma, we become automatons.”

The proper definition of a paradigm shift, as stated by Thomas Kuhn (2012), is an alteration of the basic set of assumptions within “the ruling theory of science”. According to Kuhn, “[a] paradigm is what members of a scientific community, and they alone, share” (Kuhn 1979). It is noteworthy that Kuhn restricted his use of the word to “hard” sciences (ie, natural sciences as opposed to social sciences).

The main issue with a paradigm shift is that it may not be expected; consequently, a model may be rendered completely obsolete, leading to its use becoming significantly risky. It is interesting to note that self-fulfilling prophecies usually go hand in hand with paradigm shifts. Indeed, if everyone uses the same pricing strategy (eg, Black and Scholes for options, or the same discount factor for a range of other assets), then everyone engaged in the same business knows the others are using it too; as a result, prices tend to converge and products are evaluated in line with what the ruling pricing model says they should be. Consequently, when a paradigm shifts, the outcomes of the models that relied on the previous paradigm are not valid anymore with respect to the new paradigm. So, a paradigm shift may trigger the materialization of significant model risk. Research in many fields has been undertaken to detect paradigm shifts, as these may have a tremendous impact on our perceptions. To illustrate this, we can refer to medical science, a domain in which some research has proposed alternative ways of detecting cancers by predicting mutations (Ng et al 2012), or to seismology and its stress-forecasting approach, which advocates evaluating the likelihood of an earthquake as opposed to predicting it (Crampin et al 2008). A change in underlying paradigms may be considered a revolution.

While addressing paradigm shifts, it is necessary to discuss the difference between risk and uncertainty, as analyzed by Knight (1921). According to Knight, risk occurs when we do not know the outcome of a given situation but can accurately measure the odds. Uncertainty, on the other hand, occurs in situations where we cannot know all of the information we need to set accurate odds in the first place. The term acknowledges some fundamental degree of ignorance, a limit to knowledge and an essential unpredictability of future events. The concept of a black swan event, developed by Nassim Taleb (2007), is related to Knightian uncertainty.⁴⁴ 4 Please note that this theory is not unanimously accepted; here, it merely gives us the keys to understanding a particular situation. This statement is more or less valid for all of the theories discussed or presented in this paper. A black swan is an important and inherently unpredictable event that, once it has occurred, is rationalized with the benefit of hindsight. It is used to describe historical developments such as the widespread adoption of the personal computer, something entirely impossible to predict that nevertheless had world-changing effects. Another feature of the black swan theory is that appropriate preparation for such events is frequently hindered by the pretence of knowledge of all the risks. In other words, Knightian uncertainty is presumed not to exist in day-to-day affairs, often with disastrous consequences. These concepts are consistent with the idea of self-fulfilling prophecies going hand in hand with paradigm shifts, as stated in the previous paragraph.

While dealing with financial models, a paradigm shift that remains undetected may result in fallacious asset prices, risk measures or investment decisions. Some of the biggest operational risk losses due to model failures are because of a paradigm shift. However, as illustrated by Planck (1968), societies are reluctant to recognize paradigm shifts as they may not be fully understood and therefore may trigger a reaction of fear or rejection. According to Planck (1968): “A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.”

The Gaussian copula used to evaluate collateralized debt obligations (CDOs) failing because of a change in market conditions is a typical example of an alteration of a basic set of assumptions, as it meant that dependencies between tranches were no longer linear but characterized by upper tails. Translated into a paradigm shift, the ruling theory – which supported that dependencies were linear and that nonlinearity was just an anomaly the market would correct – became that dependencies were nonlinear and linear dependencies were just a particular case. Another example is the zero weight that was associated with G10 debt in the risk-weighted asset metric used to calculate Basel ratios, even though corporate debt was nonzero. This led to strange bond acquisition strategies (Lenarčič et al 2016; Acharya et al 2016). In that case, the regulation itself engendered the paradigm shift.

Erroneous assumptions or, more frequently, strong assumptions (even if models are very robust with respect to their assumptions) are the third potential issue that could lead to model failure. This is the issue we have seen with Gaussian yields, the market portfolio of the capital asset pricing model (CAPM) (Sharpe 1964; Roll 1977), the constant volatility in the Black–Scholes formula (Black and Scholes 1971) and the absence of arbitrage opportunity or market completeness (Flood 1991). Most assumptions are hardly sustainable and may not even have been proved right. However, the most destructive meta-hypothesis widely found in the infinite set model is the assumption of the stationarity of processes (Priestley 1983).

Let us recall, as a first step, the definition of a stationary process: this is a stochastic process whose joint probability distribution does not change over time. As a corollary, the mean and variance of this process, assuming they exist, neither follow a trend nor change over time. Obviously, if processes are stationary, they have usually only remained in this state for a short period of time.⁵⁵ 5 We acknowledge that the term “short” is relative; therefore, we define a time frame as that considered for the data used in the model. This is because by merely considering economic cycles, revolutions or human actions it is difficult to see this assumption as permanently reliable. For example, fraud data (eg, from credit card fraud) is not usually stationary, as every time fraud is observed a control is put in place, and therefore fraudsters’ behavior changes and adapts (Dal Pozzolo and Bontempi 2015). Consequently, past data is not necessarily informative in modeling, analyzing or predicting new fraudsters’ behavior. Unfortunately, the stationarity hypothesis (although it is not always inaccurate and therefore should not be discarded a priori) is the very assumption that leads practitioners to use constant volatility or, more generally, to use past data to predict the future.

Sometimes assumptions are made to ease the construction of a theoretical framework and to simplify calculations; however, is the resulting model usable in practice, or is it just a representation of academic progress? This paragraph is a rejoinder to the Ernest (2004) statement provided in the introduction: mathematics has not been discovered, it has been invented. Therefore, a mathematical model may not even be close to reality and may only be used to improve our understanding of some patterns. In some situations, what we learn from implementing a model is more important than its outcomes.

It is important to avoid misplaced reliance on a “one size fits all” kind of model. In this regard, we quote Maslow (1962), who said: “I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”

The term “domain of definition” is most commonly used to describe the set of values for which a function (map, transformation, etc) is defined. For example, a mathematical expression that is defined for real values has a domain $ℝ$. The set of acceptable values that is returned by the function is called the range. In a larger sense, the domain is nothing more than a delimited area, that is, a zone in which a model can behave as expected or in the most appropriate manner.

Popper (1963, 1992) provided us with the best theorization of the limit of the domain of application. Models, behaviors, patterns and interactions are handled, understood and tested for “corroboration” and their outcomes discussed with respect to certain assumptions. This domain of application is a general concept, and, for most models, it is usually highly limited. A model is only valid under some particular conditions. A failure in the model may be caused by the fact that the model is just not applicable in a particular situation. This does not mean that the model is irrelevant; it just means that the model is not suitable in a given context. We can illustrate this point with an absurd metaphor: no one would use a 12 mm spanner on a 5 mm bolt, as it simply would not work. Similarly, distributed methodologies such as the loss distribution approach (Frachot et al 2001) in operational risk capital calculation are only reliable as long as the data is not autocorrelated (Guégan and Hassani 2013b), which happens quite often, but not always.

This section can be summarized by quoting Einstein (2011), who said: “Occurrences in this domain are beyond the reach of exact prediction because of the variety of factors in operation, not because of any lack of order in nature.”

This is a typical Basel II category 7 type of operational risk, which involves either a model not being used in the right way or its outcome not being properly understood. The misuse may be dramatic, leading to conduct risk or a massive misevaluation in the books. Basel II category 7 is defined as follows (Basel Committee on Banking Supervision 2001a, 2002): “Execution delivery and process management: losses from failed transaction processing or process management, from relations with trade counterparties and vendors.” A lack of understanding about how to properly use a particular model engenders model risk. This risk is potentially induced by a disconnect between the model developer – that is, the person who thoroughly understands what the model is supposed to do and how it is supposed to behave – and the model owner, who simply uses it. This type of risk may result from using wrong, inappropriate or scanty inputs; from mistakes in the way the model has been implemented; or due to an absence of rigour when using the model.

The first of the processes we suggest financial institutions implement is the crash-test process (Popper 1992), through which a model is fully challenged and rigorously tested. Originally, crash tests were a way to evaluate the resistance of a vehicle, along with its components, system and passengers, by driving it at a certain speed into a wall. Instead of a wall, we need to build specific situations or scenarios in which a model might fail to understand its environment and limitations.

The components of the crash-test process can be summarized as follows:

1. (1)

   random data to fail the assumptions, eg, using generalized Pareto distributed data in a model that is designed for a lognormal distribution;

2. (2)

   preposterous parameterization, eg, using extreme probabilities of default and losses given default in a credit risk model;

3. (3)

   variance of the results in the case of nonstationary scenarios, eg, using nonstationary data in a model to understand that model’s behavior; and

4. (4)

   aliasing and propagation error analysis, etc, eg, evaluating the diffusion of an initial error in a model including convolution or forecasting.

A proper example can be found in operational risk modeling (Guégan and Hassani 2012), as using a lognormal distribution on censored data may lead to a negative location parameter (commonly denoted $\mu$), which implies that most modeled losses are between 0 and 1. In some categories, however, that may not make any sense. This crash-test process would help us understand the limitations of a designed model, its sensitivity to assumptions, its domain of application and the impact a paradigm shift might have on it.

The crash-test process forces us to question the way models are built. Most of the time, due to short deadlines (either driven by endogenous or exogenous factors), the initial models implemented are simple and therefore may combine some of the issues presented in Section 2 (although, fortunately, this not always the case). Pearl and Mackenzie (2018) suggest a way of structuring the construction of models, starting from the knowledge (in a broad sense) we have of the issue to be dealt with, ie, information, past data, environment, etc. A deeper analysis of the information available might be particularly useful in curating a first set of assumptions that simplifies reality. Then, Pearl and Mackenzie suggest focusing on causality, ie, what connects one process to another. Note that it is not necessarily easy to test for causality. Here, if the past information consists of data, we refer the reader to Granger (1969) or Sugihara et al (2012), who propose statistical tests to evaluate the cause-and-effect relationship between two time series variables (correlation does not imply causation). Once causality has been addressed, a model can be structured to address the targeted issue. The model creation process consists of a loop, where, if an approach does not permit answering the scientific question, assumptions have to be challenged and causality reevaluated until a satisfactory approach is obtained.

The crash-test process assumes the model under scrutiny has not failed before and therefore no information has been recorded regarding a failure. However, if the model has failed before, a forensic analysis might be particularly informative;⁶⁶ 6 The Institute for Pure and Applied Mathematics at UCLA hosted a workshop on “Forensic Analysis of Financial Data” in May 2015. therefore, the possibility of conducting such an analysis should not be discarded.

Conservatism can be engineered at various levels of model construction (Guégan and Hassani 2015). Here, we suggest solutions at four stages, namely distribution selection and the capture of intrinsic dynamics, correlations and risk measures. Here, the term “dynamics” refers to an evolution of the data (and, consequently, of the model and its parameters over time), not to dynamical systems precisely (as in chaos theory or bifurcation theory), which might be highly unstable.

Regarding the distribution used, we have several choices for capturing skewness, leptokurtosis and extreme events (tails) – in other words, for capturing asymmetric shocks and fat tails – as follows:

• •

  the generalized hyperbolic distribution (see Barndorff-Nielsen and Halgreen, 1977);

• •

  the extreme value distribution that describes the limit distributions of normalized maxima (see Resnick, 1987);

• •

  the generalized Pareto distribution, which appears as the limit distribution of scaled excesses over high thresholds (see Embrechts et al, 1997);

• •

  the $\alpha$-stable distribution (see Nolan, 1997);

• •

  the $g$-and-$h$ distribution (see Hoaglin, 1985).

A knowledge of the dynamic components embedded in the input should promote the construction of a robust model. Analysis of the dynamics embedded within the data through a time series will permit better responsiveness to and conformance with reality. Both short-memory models, such as autoregressive (AR) processes and generalized autoregressive conditional heteroscedasticity (GARCH) models (Engle 1982; Bollerslev 1986), and long-term models, such as the Gegenbauer process (Guégan 2005), can be used.

The capture of dependencies can be achieved through copulas. Recall that a copula is a multivariate distribution function linking a large set of data through its standard uniform marginal distributions (Sklar 1959). In the literature, it has often been mentioned that using copulas is difficult when we have more than two risks: that is, apart from using elliptical copulas such as the Gaussian or the Student $t$ type (Gourier et al 2009). However, elliptic copulas fail to capture asymmetric and extreme shocks. The use of a Student $t$ copula with three degrees of freedom to capture the dependence between the largest losses mechanically implies a higher correlation between the smallest losses. One alternative is to use the Archimedean or extreme value copulas (Joe 1997), which are capable of capturing dependencies embedded in different parts of marginal distributions. However, the direct calibration of these copulas is not straightforward. To overcome the restrictions faced while employing these copulas, recent developments using either nested copulas (Joe 2005) or vine copulas (Brechmann et al 2012; Guégan and Maugis 2010; Guégan and Hassani 2013a) can be turned to.

In terms of risk measurement, although risks in banks were previously evaluated using standard deviation-based measures, the financial industry has moved toward quantile-based downside risk measures including VaR and ES. VaR measures the losses that may be expected for a given probability and corresponds to the quantile of a selected distribution. In addition, to measure the importance of an exposure beyond the VaR percentile, and to capture some diversification benefits, ES has been introduced. This has been widely dealt with in the literature: see, for example, Artzner et al (1999), Rockafellar and Uryasev (2000, 2002) and Delbaen (2000). The relationship between VaR and ES for some distributions can be found in Guégan and Hassani (2014).

Recently, several extensions have been analyzed, proposing new routes for risk measures. These are summarized below.

• •

  A spectral risk measure (Acerbi and Tasche 2002), which can provide a convex combination of different ESs with appropriate weights.

• •

  A distortion risk measure, which can be described as follows.

  • –

    When the distribution is shifted with a function close to the Gaussian one, as in Wang (2000) and Sereda et al (2010), the shift distribution remains unimodal. Thus, Guégan and Hassani (2014) proposed distorting the initial distribution with polynomials of odd degree in order to create several humps in the distribution. This helps to catch all of the information in the extremes of the distribution and introduces a new coherent risk measure computed under the distorted distribution.

  • –

    Guégan et al (2015) proposed a new measure combining the construction of a confidence interval around VaR with a spectral approach to provide a conservative envelope around the cdf, therefore focusing on the various indicators that can be used for risk management purposes.

It is important to make a distinction between the model, that is, the theoretical framework, and its application to a particular problem involving the estimation of parameters, calibration, etc. A model may be appropriate, but model risk can still be generated by its parameterization.

Finally, solutions to a problem may engender larger issues. Therefore, it is particularly important to analyze the trade-off between precision and stability before implementing a solution.

The challenger–champion relationship appears to be a sensible way of addressing model risk in general, as it avoids relying on a single approach to capture a pattern and provides some information on a model’s reliability (if the results tend to converge, then either both are right or – in the worst-case scenario – both are misleading). A model is considered the “champion” when it is used as the main solution (because it has been approved by regulators, for example). Such models have usually been industrialized, whereas challenger models are those that have been developed in parallel to champion ones, usually for validation purposes. At the very least, by having an alternative – by way of this process – we can avoid falling into dogmas and “best practice” traps. We acknowledge the fact that it is sometimes wrong to use a particular model, and this approach is a humble and pragmatic way of dealing with such issues. The challenger model should be built from zero and should rely, as much as possible, on alternative assumptions and/or methodology.

In addition, the challenger–champion relationship potentially enlarges the domain of a model’s validity and ensures the continuity of quantification in case one model fails. This is actually something that has been suggested in the latest pieces of regulation on stress testing by the Prudential Regulation Authority (2013) and the European Banking Authority (2014b).

Corporate architecture requires senior management to have a basic understanding of mathematical figures and models. Indeed, the Federal Reserve (2011) states that: “As part of their overall responsibilities, a bank’s board and senior management should establish a strong model risk management framework that fits into the broader risk management of the organization. That framework should be grounded in an understanding of model risk – not just for individual models but also in the aggregate. The framework should include standards for model development, implementation, use and validation.

While the board is ultimately responsible, it generally delegates to senior management the responsibility for executing and maintaining an effective model risk management framework. Duties of senior management include establishing adequate policies and procedures and ensuring compliance, assigning competent staff, overseeing model development and implementation, evaluating model results, ensuring effective challenge, reviewing validation and internal audit findings, and taking prompt remedial action when necessary.” Therefore, they should not be afraid of speaking out when they do not understand certain aspects of the business. Meanwhile, developers should be trained to communicate in the most effective way and should be able to translate technicalities into words a layperson will understand. For example, they should be able to explain their model in one page, one paragraph or one sentence. Here, another communication trade-off between simplification and accuracy must be handled carefully, as an oversimplified explanation could result in a loss of accuracy and therefore mislead the management team.

Mathematics, if considered as a branch of philosophy, is useful for representing concepts and simplifying them. It certainly does not complicate them. However, the symbolism in mathematics is quite complicated: it is like a foreign language and requires some practice before it becomes comprehensible to anyone. Therefore, model developers, model owners and model users (ie, the people who ultimate use them to make decisions) should be trained to develop the skills that they lack. This would help to prevent or at least attenuate the risk of a model being misused or misinterpreted.

Usually, we observe a disconnect between model developers and users. The developer is the supplier and the user is the customer. As for any other product, the model should be developed to satisfy customer demands, both in terms of what it does and in terms of visualization. Obviously, the developer should uphold the integrity of the outcomes, ie, models should be developed in such a manner that they capture and represent a situation as precisely as possible to ensure the actionability⁷⁷ 7 That is, the decisions taken that rely on those models. of their outcomes. As characterized by the user modeling field of research, the main goal is to customize and adapt systems to meet users’ specific needs.

Practically, a model should be adapted to its user, and while an ergonomic environment is probably the most important thing for users, it does not prevent them from misunderstanding a model. The users of a model should be able to understand and explain the main features of that model as well as its main assumptions and limitations. They should not necessarily be expected to explain it quantitatively; however, from a practical point of view, they should be able to anticipate when a model is going to fail, relying, for example, on the results of a crash-test program.

In theory, model governance, as summarized below, is a fairly simple process that just needs to be followed rigorously. However, doing an inventory of models is a huge task, as the number of models used in a financial institution can be very large. Indeed, models are not limited to capital quantification or pricing; they are, in fact, used everywhere, from marketing to strategy and beyond. Most people do not even know they are using a model to make a decision because so many models have been integrated into their day-to-day work.

An appropriate governance may be summarized as follows.

1. (1)

   The models should be crash tested and the developers themselves should develop the challenges.

2. (2)

   The developers should ensure that, even if some issues cannot be dealt with, the model is at least conservative.

3. (3)

   All of the pieces developed should be properly documented. The key assumptions, sensitivities and limitations should be adequately detailed. The code should be fully explained, as each developer has their own way of programming.

4. (4)

   The models should be independently validated.

5. (5)

   The models should follow a proper approval process, performance monitoring should be conducted and their materiality should be appropriately reported.

6. (6)

   The models should be overseen and challenged by senior management.

7. (7)

   An appropriate change management process should be adopted when a model is deployed.

We believe that, in the very near future, once model risk has been properly apprehended (ie, once we have progressed from understanding it to implementing functional corporate governance), it will be possible to hedge residual risk and even insure it. However, we believe that it is too early for this, as the industry is (more or less) still in a period of inherent risk assessment.