Circuit Layer Operations Per Second (CLOPS) benchmark
The CLOPS protocol [1] assesses the execution speed and reliability of a quantum computer in running a series of parameterized square quantum circuits. The original formulation [1] introduces the \(CLOPS_h\) figure of merit, while a subsequent update to the protocol [2] defines an alternative figure of merit \(CLOPS_v\). Both figures of merit are reported in the results table for completeness.
List of acronyms
CE: Constructor Evaluation (checked if the evaluation is done by the chip manufacturer)
SP: Scientific paper (checked if a scientific paper explain the results)
M: number of templates
K: number of different circuits for each template
QV layers: number of Quantum volume layers (equivalent to \(\log_2 (QV)\))
| Ref | Company | Year | CE | SP | M | K | Shots | QV layers | CPU / QPU | Technology | #qubits | \(CLOPS_v\) | \(CLOPS_h\) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (missing reference) | IBM | 2021/10 | x | x | 100 | 10 | 100 | 5 | ibmq_bogota | Superconducting | 5 | 1 419 | |
| (missing reference) | IBM | 2021/10 | x | x | 100 | 10 | 100 | 5 | ibmq_toronto | Superconducting | 27 | 951 | |
| (missing reference) | IBM | 2021/10 | x | x | 100 | 10 | 100 | 5 | ibmq_brooklyn | Superconducting | 65 | 753 | |
| [3] | QuTech | 2023/08 | x | 100 | 10 | 100 | 2 | Starmon-5 | Superconducting | 5 | 372 | ||
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 9 | ibm_marrakesh (Heron r2) | Superconducting | 156 | 2 400 | 195 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 9 | ibm_fez (Heron r2) | Superconducting | 156 | 2 400 | 195 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 9 | ibm_torino (Heron r1) | Superconducting | 156 | 2 400 | 210 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 7 | ibm_sherbrooke (Eagle r3) | Superconducting | 127 | 2 400 | 30 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 7 | ibm_kyiv (Eagle r3) | Superconducting | 127 | 2 400 | 30 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 7 | ibm_brisbane (Eagle r3) | Superconducting | 127 | 2 400 | 180 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 7 | ibm_quebec (Eagle r3) | Superconducting | 127 | 2 400 | 32 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 7 | ibm_kawasaki (Eagle r3) | Superconducting | 127 | 2 400 | 29 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 7 | ibm_brussels (Eagle r3) | Superconducting | 127 | 2 400 | 220 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 7 | ibm_rensselaer (Eagle r3) | Superconducting | 127 | 2 400 | 32 000 | |
| (missing reference) | IBM | 2025/02 | x | 100 | 10 | 100 | 7 | ibm_strasbourg (Eagle r3) | Superconducting | 127 | 2 400 | 220 000 | |
| [4] | IQM | 2024/08 | x | x | 100 | 10 | 100 | 5 | Garnet | Superconducting | 20 | 2 600 |
Motivation
The Circuit Layer Operations Per Second (CLOPS) [5] was introduced by IBM in 2021. CLOPS is designed to simultaneously capture three critical aspects of quantum computation: quality, speed, and circuit scale. Specifically, it quantifies the number of Quantum Volume (QV) circuits \cite{cross2019validating} that a quantum processor can execute reliably per unit of time.
Protocol details
The initial protocol detailed in [5] is based on the quantum volume protocol and defines \(M\) template circuits. Each template circuit is composed of \(K\) different real quantum circuits with a set of parameter \(\vec{\theta}_i\) used to initialize each \(SU(4)\) gate. The permutation layers \(\pi_i\) are the same for all the circuits generated from a single template (see Fig. 1).
The intial protocol introduced in [5] defines the figure of merit \(CLOPS_h\) as:
\[CLOPS_h = \frac{M \times K \times S \times D}{\mathrm{time\_taken}}\]where \(M\) is the number of templates, \(K\) is the number of simulations done for each template with different parameters for the layers \(SU(4)\), \(S\) is the number of shots and \(D=\log_2 QV\) layers. The initial protocol sets \(M=100\) and \(K=10\), which leads to the execution of \(1000\) different quantum circuits. Each circuit is run with 100 shots. The \(time\_taken\) accounts for the total execution time including data transfer to the QC, transpilation of high-level quantum gates to low-level control pulses, qubit measurement and reset time as well as inter-circuit delays.
Assumptions
- The compilation of each template circuit (high-level compilation and circuit optimization) is not accounted in the measured \(time\_taken\).
- For each template circuit, ten parameterized circuits are generatacross all the pairs of qubits. This initied. The parameters \(\vec{\theta}_i\) for the \(i\)-th circuit are only instantiated once the execution of the \((i-1)\)-th circuit has been completed.
- Multiple template circuits may be run in parallel on the quantum chip.
- It is essential to specify the quantum volume used in the benchmark to interpret the results. Indeed, a lower quantum volume might lead to a higher CLOPS score.
Limitations
- The CLOPS score comparison is only meaningful if the same quantum volume is used for all the experiments (this is due to the use of the logarithm of the quantum volume in the computation of the CLOPS score).
- The limits of the quantum volume (e.g., scalability) also apply to the metric \(CLOPS_v\).
Update of the CLOPS protocol
An update concerning the measurement of CLOPS values has been proposed in [2] with a new protocol measuring \(\mathrm{CLOPS_h}\) (\(\mathrm{h}\) for hardware). This new protocol changes how layers are considered. In the initial protocol, a layer was defined by a random permutation of all the qubits followed by two-qubit gates (\(SU(4)\)) across all the pairs of qubits. This initial protocol was practical for fully connected topologies that could easily implement the random permutation. The new protocol \(\mathrm{CLOPS_h}\) splits in sublayers all sequences of gates that cannot be run in parallel (artificially inflating the overall number of layers). It has the effect of inflating the CLOPS score for quantum computers that are sparsely connected.
References
- [1]A. W. Cross, L. S. Bishop, S. Sheldon, P. D. Nation, and J. M. Gambetta, “Validating quantum computers using randomized model circuits,” Physical Review A, vol. 100, no. 3, p. 032328, 2019.
- [2]A. Wack and D. McKay, “Updating how we measure quantum quality and speed.” 2023 [Online]. Available at: https://www.ibm.com/quantum/blog/quantum-metric-layer-fidelity. [Accessed: 26-Feb-2025]
- [3]W. van der Schoot, R. Wezeman, P. T. Eendebak, N. M. P. Neumann, and F. Phillipson, “Evaluating three levels of quantum metrics on quantum-inspire hardware,” arXiv preprint arXiv:2310.01120, 2023.
- [4]L. Abdurakhimov et al., “Technology and Performance Benchmarks of IQM’s 20-Qubit Quantum Computer,” arXiv preprint arXiv:2408.12433, 2024.
- [5]A. Wack et al., “Quality, Speed, and Scale: three key attributes to measure the performance of near-term quantum computers.” 2021 [Online]. Available at: https://arxiv.org/abs/2110.14108