Aggregated scores

MYRIAD-Q aggregated figure of merit (2024)

In [1], F. Barbaresco et al. present a methodology to aggregate benchmarking scores to a single final score according to decision-maker preferences.

Motivation

The motivations are numerous. The authors claim that existing figures of merit for measuring the performance of quantum computers are too technical and require other methodologies to derive operational indicators of performance associated with quantum computers. To address this gap, the proposed method aims to develop an assessment framework that can serve as a long-term tool for evaluating the performance of quantum computing systems.

Protocol

The protocol is based on quantum applications across four domains in which quantum computers are believed to offer computational advantages:

• Quantum simulation
• Optimization
• Linear system solving
• Prime factorization

Several figures of merit (also called criteria) are identified for each domain. These figures of merit are then aggregated into a single score using Multi-Criteria Decision Aiding Analysis (MCDA) techniques in accordance with the MYRIAD-Q methodology.
The first step involves normalizing each figure of merit using a marginal utility function. This function maps the values of each criterion onto a common scale within the range \(\mathopen{[} 0, \infty \mathclose{[}\). For each figure of merit, \(k\) different values are selected. The decision maker is asked to rank these values in a preferred order \({x_1, x_2, ..., x_k}\), and to identify the rate of satisfaction for which \(x_i\) is preferred to \(x_{i-1}\). A six-level scale quantifies the rate of satisfaction from very weak gain to extreme gain. The marginal utility function \(u_i\) associated with each criterion is then built, reflecting the subjective preference of the decision maker.
The second step aggregates the normalized utility values using the two-additive Choquet integral. This approach permits to model potential overlaps between criteria. For instance, a high utility \(u_i\) in one criterion may strongly correlate with a high utility \(u_j\) in another. To determine the interaction coefficients required for the Choquet integral, the decision-maker is presented with hypothetical scenarios and asked to express preferences between them, along with corresponding satisfaction levels. An example scenario might ask whether a configuration with a high score in \(u_i\)​ and a low score in \(u_j\)​ is preferable to the inverse. From these comparative assessments, the Choquet integral’s parameters are derived algorithmically.

Limitations

As mentionned by the authors, the MYRIAD-Q score strongly depends on the decision maker subjective point of view, this figure of merit cannot be compared unless the exact same detailed protocol is replicated.

Quantum Application Score (QuAS) (2024)

The Quantum Application Score (QuAS) [2] was introduced by K. J. Mesman et al. in 2024 and aims to define a framework to compute a single score from Key Performance Indicators (KIPs) chosen by the user.

Motivation

The motivation for the definition of this framework is to define a protocol with several degrees of freedom concerning the problem instances being benchmarked and the figures of merit being measured. The authors wish to integrate an equitable analysis between the different KPIs being benchmarked using this method.

Protocol

The initial stage of the protocol is dedicated to identifying the set of KPIs relevant to the benchmark study. The authors illustrate this with an example involving three primary KPIs: the problem size, the solution quality, and the runtime of the quantum device. In their example, the quality of the solution obtained with the quantum computer is expressed concerning a reference solution found classically.
For a fixed-size problem, data points are collected using different problem instances and varying runtimes. Each axis is normalized to a \([0, 1]\) range, enabling data visualization in a two-dimensional plane. The Pareto front between the quality of the solution and the runtime can be approximated using lamé curves (see Figure). The area under each curve is computed, with the integral of the blue curve exceeding that of the red curve in the given example. These integrals are summed across all problem sizes, giving the final QuAS score. The quantum computer yielding the highest QuAS score is identified as the most effective method for solving the target problem.

Each KPI can be weighted according to the user’s choice, and offsets can be added to each quantity to normalize the score.

Limitations

As mentioned by the authors, the QuAS score defines a methodology and not a precise setting of KPIs and weights associated with each KPI. Hence, comparing two QuAS scores from different studies cannot be done.
The QuAS framework is only adapted to assess the performance of quantum computers on problems in which solutions can be expressed with a single number, such as single-objective optimization problems. Still, it is not suited for some other problems, such as multi-objective optimization problems.

References

1. [1]F. Barbaresco et al., “BACQ–Application-oriented Benchmarks for Quantum Computing,” arXiv preprint arXiv:2403.12205, 2024.
2. [2]K. J. Mesman, W. van der Schoot, M. Möller, and N. M. P. Neumann, “QuAS: Quantum Application Score for benchmarking the utility of quantum computers,” in 2024 IEEE International Conference on Quantum Computing and Engineering (QCE), 2024, vol. 1, pp. 921–929.