Algorithmic qubit benchmark

Benchmarking results

List of acronyms:
AC: Advanced Compilation method
CE: Constructor Evaluation (checked if the evaluation is done by the chip manufacturer)
COI: Conflict of Interest
EM: Error mitigation
SP: Scientific paper (checked if a scientific paper explain the results)

Motivation

The Algorithmic Qubit (AQ) benchmark, introduced by IonQ in 2020 [5] is derived from the Volumetric benchmarking protocol [6]. The main motivation presented by the authors is to define a single number figure of merit to evaluate the performance of quantum computers for solving a representative set of quantum algorithms.

Protocol

A detailed description of the AQ protocol is available in an associated GitHub repository [7]. The protocol consists of sampling the output distribution of a quantum circuit and comparing this output to the ideal output distribution by computing the classical fidelity.

For validating a $n$ AQ, several algorithms are used to generate the following quantum circuits using $n$ qubits:

• 3 circuits based on the Quantum Fourier Transform
• 3 circuits based on the Quantum Phase Estimation
• 3 circuits based on the Amplitude Estimation
• 1 circuit based on the Monte Carlo Sampling algorithm
• 3 circuits based on a VQE simulation
• 1 circuit based on a Hamiltonian simulation

Each circuit is then compiled to a fixed gate set consisting of $CX$, $R_x$, $R_y$, and $R_z$ using Qiskit compiler 0.34.2, with compilation seed set to $0$. The depth and width of the compiled circuit are recorded as $w_d$ and $w_c$ (these quantities are used to build the volumetric plot).

It is important to emphasize that this first compilation step is performed solely to determine the quantities $w_d$ and $w_c$. Subsequently, the circuit may be further compiled and optimized before execution on the quantum device. The only restriction is that the final circuit run on the quantum computer must implement the same unitary transformation as the initial circuit. Error mitigation techniques are permitted, provided that their application is clearly reported. Each circuit is sampled $n_\mathrm{shot}$ times. For each circuit, the ideal sampling probability is denoted by $\widetilde{p}_c$ and the experimentally obtained distribution by $p_c$. The classical fidelity $F(p_c, \widetilde{p}_c)$ is computed and the statistical error $\epsilon_c$ is extracted:

The number of algorithmic qubits associated with a quantum computer is defined as the largest size $n$ such that all generated circuits satisfying $w_c \le n$ and depth $\sqrt{w_d} \le n$ achieve a fidelity beyond the validity threshold $F(p_c, \widetilde{p}_c) - \epsilon_c > 0.37$.

Controversy and limitations

This protocol has been criticized for several reasons discussed in [3]. The authors highlight the problematic role of error mitigation techniques, which can artificially enhance the apparent performance of quantum computers and reduce the observed performance gap between different devices. They demonstrate that certain error mitigation methods, particularly those not expected to scale favorably with system size, can lead to misleading results. Additionally, they point out concerns regarding the use of Qiskit compiler version 0.34.2 for calculating the number of CX gates and hence, depth of the circuit. As this compilation method is used to compute the values used in the heatmaps $w_d$ and $w_c$, a suboptimal compilation could artificially inflate a device’s reported performance. In particular, the authors show that alternative compilers can significantly reduce the overall number of gates (especially for the largest circuits).

Another limitation of the protocol is the restricted number of different circuits used during the benchmark which could affect the robustness of the results.

References

1. [1]I. Q. staff, “Algorithmic qubits: a better single-number metric.” 2023 [Online]. Available at: https://ionq.com/resources/algorithmic-qubits-a-better-single-number-metric. [Accessed: 26-Feb-2025]
2. [2]J.-S. Chen et al., “Benchmarking a trapped-ion quantum computer with 30 qubits,” Quantum, vol. 8, p. 1516, 2024.
3. [3]Quantinuum, “Debunking algorithmic qubits.” 2024 [Online]. Available at: https://www.quantinuum.com/blog/debunking-algorithmic-qubits. [Accessed: 26-Feb-2025]
4. [4]IonQ, “IonQ Unveils Its First Quantum Computer in Europe, Online Now at a Record #AQ36.” 2024 [Online]. Available at: https://ionq.com/news/ionq-unveils-its-first-quantum-computer-in-europe-online-now-at-a-record?utm_source=linkedin&utm_medium=social&utm_campaign=QuantumBasel&utm_content=press-release&utm_term=45627. [Accessed: 26-Feb-2025]
5. [5]IonQ, “Quantum Benchmarking. Understanding Algorithmic Qubits.” 2020 [Online]. Available at: https://ionq.com/algorithmic-qubits. [Accessed: 26-Feb-2025]
6. [6]T. Lubinski et al., “Application-oriented performance benchmarks for quantum computing,” IEEE Transactions on Quantum Engineering, vol. 4, pp. 1–32, 2023.
7. [7]IonQ, “Rules for AQ v1.0.” 2022 [Online]. Available at: https://github.com/ionq/QC-App-Oriented-Benchmarks/blob/master/_doc/AQ.md. [Accessed: 26-Feb-2025]