A tale of two (or three, or four) models

Working out how to compare the performance of derivatives pricing models seems like a simple problem. Yet a satisfactory solution is hard to come by. The financial literature is limited to a handful of studies, and the standard approach – so-called P&L explain or P&L attribution – is considered too simplistic.

The search for a better way has taken on added urgency with the introduction of new market risk capital rules. Under the Fundamental Review of the Trading Book, banks must pass a series of tests before they are allowed to use their own internal risk models.

The P&L attribution test, a key plank of the regime, uses a mean and variance ratio to compare two P&L figures, one generated by a bank’s front-office pricing models and one generated by its risk models.

Banks employ similar techniques to compare pricing models and select the best ones from a risk management perspective. But some feel the practice falls short of industry needs – particularly for hedging – and question its reliability.

“It’s useful to have but it is quite a simplistic test,” says Alexandre Antonov, chief analyst at Danske Bank.

Antonov, together with Rajiv Sesodia, head of traded risk models at Standard Chartered, and Jan Baldeaux, a quantitative analyst at ANZ Bank, proposes an alternative approach.

Their technique consists of a backward-looking test that compares the models’ replicating portfolios, which are combinations of assets or derivatives designed to mimic the payoff of the original position. In the Black-Scholes world, replicating portfolios are the building blocks of arbitrage-free pricing and hedging.

In practice, the authors extract the hedging components within the payoff, recalculate the hedging portfolio using an alternative model, and observe its movements. If these movements can be described by a deterministic function of the market factors on the current position, then the model perfectly fits the market and the instrument. A deterministic function will give the same results given the same inputs.

Any deviation between hedge and original position will give the hedging error, the difference between perfect hedging and imperfect hedging.

We cannot say [the test] will always give the same answer to the comparison between two models… but we estimate it is about 95% accurate

Alexandre Antonov, Danske Bank

The approach aims to provide front-office desks with an objective way of gauging whether a new model outperforms the existing one. The method could also find a home in the model validation team in a bank.

The research shows that a more sophisticated model like the Heston model performs better than Black-Scholes, even with the adjustments that are normally plugged into it. “This is a quite strong conclusion, because there are publications that reached the opposite one,” explains Antonov.

In particular, one study in 2018 by Frederic Vrins, finance professor and chairman of the Leuven finance research centre, and his then student, Nathan Lassance, found that the Black-Scholes option pricing model was preferable to the Heston model.

The opposing conclusions of the two studies could stem from one crucial difference between techniques. Antonov et al’s method observes the hedging error throughout the whole duration of the contract, whereas Vrins and Lassance’s technique observes the hedging error at maturity.

Vrins explains that, at the time of his study, there was very little research into optimal models over the full lifetime of the hedge.

“When we started our research, we were surprised not to find a study that analysed the hedging error up to the maturity of the derivative. Most of them deal with a very short hedging horizon or focused on fitting the implied volatility surface,” Vrins says.

Antonov admits there are weaknesses in the technique he describes, and that its output cannot be blindly taken on board at all times. “Of course we cannot say it will always give the same answer to the comparison between two models for any set of data and contract, because it’s based on statistics, but we estimate it is about 95% accurate,” he explains.

However, he is confident in the benefits it can bring to derivatives desks and would like to see it implemented soon: “I’m planning to present the method internally to the model validation team at Danske Bank,” he reveals.

Others in the industry say the new test is an improvement on the existing method for comparing models.

“The P&L explain defines performance over a fairly short term and in a narrow way,” says Colin Turfus, senior model validation quant at a major European bank. “I like this new approach insofar as it is more of a global perspective of the performance of the model from the beginning to the end of the trade.”

Editing by Kris Devasabai and Alex Krohn