Benchmarking the Operation of Quantum Heuristics and Ising Machines: Scoring Parameter Setting Strategies on Real World Optimization Applications*

We discuss guidelines for the performance evaluation of parameterized stochastic solvers of optimization problems, with particular attention to systems that employ quantum or unconventional processors, such as QAOA, Quantum Annealing, or Coherent Ising Machines. We illustrate through an example a benchmarking procedure grounded in the statistical analysis of the expectation of a given performance metric outside a test environment, taking into account all resources related to real-world deployment.

By properly discussing these aspects, it is possible to identify evaluation principles of parameter tuning strategies and more complex use of the heterogeneous components of the solver. In particular, we examine in detail and provide the software to compute the scoring of an illustrative baseline family of parameter setting strategies that feature an exploration-exploitation tradeoff.