summaries: # Overview  The paper proposes a framework incorporating deep reinforcement learning for resolving Ansatz optimization problems, which is with wide impacts on quantum chemistry. The authors have a good introduction and related work section, which covers most standing works in quantum reinforcement learning and variational circuit learning.   Compared with Rotosolve and COBYLA, the proposed DDQN-based method attains state-of-the-art results on estimating the ground-state energy of lithium hydride.  Overall, I like the idea but the baseline comparison and some time complexity could better incorporate to add the depth of this paper. The current version is more application-oriented and less novel as a general ML framework considering existing works in the neural architecture search (NAS) community.   - Justification of DDQN and comparison   When discussing "why selecting DDQN for the RL", there is very little discussion.   Since this work is not the first works on applying the heuristic method for quantum circuit architectures search (QCAS), the contribution is more on framing the Ansatz optimization and the benchmark results. It would be more valuable to address the selection and justification of the algorithm. (e.g., DRL-based vs Neuroevolution-based or at least the variants inside DQNs)  For the general audience in the NeurIPS community, instead of only showing the DDQN-based method for QCAS, providing more in-depth discussion between variants of the DRL algorithm (e.g., DQN, DQN with dueling, DDQN with dueling) and its complexity could gain better values of this paper.    The variational quantum eigensolver (VGE) is a method that utilizes a Noisy Intermediate Scale Quantum (NISQ) computer to find the ground state of quantum systems. At its core, an Ansatz for the ground-state wave-function is parameterized in terms of a quantum circuit, i.e. a series of single-qubit and two-qubit gates acting on a reference state, and the optimal parameters of the circuit are determined by a classical algorithm that uses the NISQ to evaluate the performance (energy) of the Ansatz.In this manuscript, the authors propose a reinforcement learning (RL) based approach to identify the small circuits (i.e. with the low number of gates, and thus less prone to errors) that still allow to reach chemical accuracy. The RL method is based on double deep-Q learning and curriculum learning. The authors employ their method to find the ground state energy of LiH approximating the electronic Hamiltonian as a 4 or 6 qubits Hamiltonian. For 3 different bond distances, the method proposed by the authors identifies circuits that achieve chemical accuracy, but whose size is lower as compared to other Ansatze This paper studies the ansatz for Variational Quantum Eigensolver (VQE) quantum circuits. The authors propose a deep reinforcement learning framework to generate the ansatz for VQE circuit, aiming to achieve low estimated energy and shallow circuit depth. The authors leverage curriculum learning to gradually reduce the energy threshold (increase difficulty) to avoid learning failure. The method is evaluated on LiH molecule in different settings and shows shallower circuit than baseline HE and UCCSD circuits.  This work demonstrates that reinforcement learning with intrinsic motivation can be effectively used to search over the space of ansatzes for the variational quantum eigensolver (VQE) algorithm. This algorithm is used in quantum chemistry to approximate the ground state energy of molecules (e.g. LiH). The author(s) describe how DDQN trained with Adam can learn to optimize a VQE-relevant reward function, in an environment which runs an optimization subroutine (COBYLA or rotosolve) at each step to optimize the rotation angles of the ansatz's gates. Their LiH experiments used a simulated noiseless quantum environment, and they benchmarked the performance against two well-known ansatzes. The results show that in simulation, RL has promising potential for circuit depth improvement.
abstractText: The study of Variational Quantum Eigensolvers (VQEs) has been in the spotlight in recent times as they may lead to real-world applications of near-term quantum devices. However, their performance depends on the structure of the used variational ansatz, which requires balancing the depth and expressivity of the corresponding circuit. At the same time, near-term restrictions limit the depth of the circuit we can expect to run. Thus, the optimization of the VQE ansatz requires maximizing the expressivity of the circuit while maintaining low depth. In recent years, various methods for VQE structure optimization have been introduced but the capacities of machine learning to aid with this problem have not yet been extensively investigated. In this work, we propose a reinforcement learning algorithm that autonomously explores the space of possible ansatzes, identifying economic circuits which still yield accurate ground energy estimates. The algorithm uses a feedback-driven curriculum learning method that autonomously adapts the complexity of the learning problem to the current performance of the learning algorithm and it incrementally improves the accuracy of the result while minimizing the circuit depth. We showcase the performance of our algorithm on the problem of estimating the ground-state energy of lithium hydride (LiH) in various configurations. In this well-known benchmark problem, we achieve chemical accuracy and state-of-the-art results in terms of circuit depth.
authors.name: Mateusz Ostaszewski
authors.name: Lea M. Trenkwalder
authors.name: Eleanor Scerri
authors.name: Vedran Dunjko
references.authors: John Preskill
references.title: Quantum computing in the NISQ era
references.venue: and beyond. Quantum,
references.authors: Alberto Peruzzo Jarrod McClean Peter Shadbolt Man-Hong Yung Xiao-Qi Zhou Peter J Love Alán Aspuru-Guzik Jeremy L
references.title: O’brien. A variational eigenvalue solver on a photonic quantum processor
references.venue: Nature Communications,
references.authors: Abhinav Kandala Antonio Mezzacapo Kristan Temme Maika Takita Markus Brink Jerry M Chow Jay M Gambetta
references.title: Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets
references.authors: Harper R. Grimsley Sophia E. Economou Edwin Barnes Nicholas J. Mayhall
references.title: An adaptive variational algorithm for exact molecular simulations on a quantum computer
references.venue: Nature Communications,
references.authors: Ho Lun Tang V.O. Shkolnikov George S. Barron Harper R. Grimsley Nicholas J. Mayhall Edwin Barnes Sophia E. Economou
references.title: qubit-ADAPT-VQE: An adaptive algorithm for constructing hardware-efficient ansatze on a quantum processor
references.authors: Pierre-Luc Dallaire-Demers Nathan Killoran
references.title: Quantum generative adversarial networks
references.venue: Phys. Rev. A,
references.authors: Marcello Benedetti Erika Lloyd Stefan Sack Mattia Fiorentini
references.title: Parameterized quantum circuits as machine learning models
references.venue: Quantum Science and Technology,
references.authors: Leo Zhou Sheng-Tao Wang Soonwon Choi Hannes Pichler Mikhail D. Lukin
references.title: Quantum approximate optimization algorithm: Performance, mechanism, and implementation on nearterm devices
references.venue: Phys. Rev. X,
references.authors: Jonathan Romero Alán Aspuru-Guzik
references.title: Variational quantum generators: Generative adversarial quantum machine learning for continuous distributions
references.venue: Advanced Quantum Technologies,
references.authors: M.-H. Yung J. Casanova A. Mezzacapo J. McClean L. Lamata A. Aspuru-Guzik E. Solano
references.title: From transistor to trapped-ion computers for quantum chemistry
references.venue: Scientific Reports,
references.authors: Abhinav Kandala Antonio Mezzacapo Kristan Temme Maika Takita Markus Brink Jerry M. Chow Jay M. Gambetta
references.title: Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets
references.authors: Marcello Benedetti Mattia Fiorentini Michael Lubasch
references.title: Hardware-efficient variational quantum algorithms for time evolution, 2020
references.authors: Jeffrey L. Elman
references.title: Learning and development in neural networks: the importance of starting small
references.authors: Yoshua Bengio Jérôme Louradour Ronan Collobert Jason Weston
references.title: Curriculum learning
references.venue: In ACM International Conference Proceeding Series,
references.authors: IG Csizmadia BT Sutcliffe MP Barnett
references.title: A group orbital study of lithium hydride
references.venue: Canadian Journal of Chemistry,
references.authors: Steven Herbert Akash Sengupta
references.title: Using reinforcement learning to find efficient qubit routing policies for deployment in near-term quantum computers
references.venue: arXiv preprint arXiv:1812.11619,
references.authors: M Sohaib Alam
references.title: Quantum logic gate synthesis as a markov decision process
references.venue: arXiv preprint arXiv:1912.12002,
references.authors: Mohammad Pirhooshyaran Tamas Terlaky
references.title: Quantum circuit design search
references.venue: arXiv preprint arXiv:2012.04046,
references.authors: Thomas Fösel Murphy Yuezhen Niu Florian Marquardt Li Li
references.title: Quantum circuit optimization with deep reinforcement learning, 2021
references.authors: Shi-Xin Zhang Chang-Yu Hsieh Shengyu Zhang Hong Yao
references.title: Neural predictor based quantum architecture search
references.venue: arXiv preprint arXiv:2103.06524,
references.authors: En-Jui Kuo Yao-Lung L Fang Samuel Yen-Chi Chen
references.title: Quantum architecture search via deep reinforcement learning
references.venue: arXiv preprint arXiv:2104.07715,
references.authors: Yuxuan Du Tao Huang Shan You Min-Hsiu Hsieh Dacheng Tao
references.title: Quantum circuit architecture search: error mitigation and trainability enhancement for variational quantum solvers
references.authors: Arthur G. Rattew Shaohan Hu Marco Pistoia Richard Chen Steve Wood
references.title: A domainagnostic, noise-resistant, hardware-efficient evolutionary variational quantum eigensolver
references.venue: arXiv: Quantum Physics,
references.authors: D Chivilikhin A Samarin V Ulyantsev I Iorsh AR Oganov
references.title: Kyriienko. Mog-vqe: Multiobjective genetic variational quantum eigensolver
references.venue: arXiv preprint arXiv:2007.04424,
references.authors: Nikolaus Hansen Andreas Ostermeier
references.title: Completely derandomized self-adaptation in evolution strategies
references.venue: evol. Comput.,
references.authors: M Bilkis M Cerezo Guillaume Verdon Patrick J Coles Lukasz Cincio
references.title: A semi-agnostic ansatz with variable structure for quantum machine learning
references.venue: arXiv preprint arXiv:2103.06712,
references.authors: Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra Martin Riedmiller
references.title: Playing atari with deep reinforcement learning
references.venue: In NIPS Deep Learning Workshop
references.authors: Michael JD Powell
references.title: A direct search optimization method that models the objective and constraint functions by linear interpolation
references.venue: In Advances in optimization and numerical analysis,
references.authors: Mateusz Ostaszewski Edward Grant Marcello Benedetti
references.title: Structure optimization for parameterized quantum circuits
references.authors: Rémy Portelas Cédric Colas Lilian Weng Katja Hofmann Pierre-Yves Oudeyer
references.title: Automatic Curriculum Learning For Deep RL: A Short Survey
references.venue: IJCAI International Joint Conference on Artificial Intelligence,
references.authors: Jacob T. Seeley Martin J. Richard Peter J. Love
references.title: The bravyi-kitaev transformation for quantum computation of electronic structure
references.venue: The Journal of Chemical Physics,
references.authors: P. Jordan E. Wigner
references.title: Über das paulische äquivalenzverbot
references.venue: Zeitschrift für Physik,
references.authors: R.S. Sutton A.G. Barto
references.title: Reinforcement Learning: An Introduction
references.authors: Yasunari Suzuki Yoshiaki Kawase Yuya Masumura Yuria Hiraga Masahiro Nakadai Jiabao Chen Ken M Nakanishi Kosuke Mitarai Ryosuke Imai Shiro Tamiya
references.title: Qulacs: a fast and versatile quantum circuit simulator for research
references.authors: Mark R. Hoffmann Jack Simons
references.title: A unitary multiconfigurational coupled-cluster method: Theory and applications
references.venue: The Journal of Chemical Physics,
references.authors: Marco Cerezo Andrew Arrasmith Ryan Babbush Simon C Benjamin Suguru Endo Keisuke Fujii Jarrod R McClean Kosuke Mitarai Xiao Yuan Lukasz Cincio
references.title: Variational quantum algorithms
sections.heading: 1 Introduction
sections.text: As we are entering the so-called Noisy Intermediate Scale Quantum (NISQ) [1] technology era, the search for more suitable algorithms under NISQ restrictions is becoming ever more important. A truly compatible NISQ application must first be amenable to architecture constraints and size limits. Furthermore, to minimize the adverse effects of gate errors and decoherence, it is important that the circuits are as gate-frugal and as shallow as possible.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  Perhaps the most promising classes of such algorithms are based on variational circuit methods, with which we have high expectations when applied to problems in quantum chemistry. A key problem in this field is the computing of ground state energies and low energy properties of chemical systems (the chemical structure problem). This problem is believed to be intractable in general, yet the quantum Variational Quantum Eigensolver (VQE) [2] algorithm can provide solutions in regimes which lie beyond the reach of classical algorithms, while maintaining NISQ-friendly properties [3, 4].
  VQE is a hybrid quantum-classical algorithm, where a parametrized quantum state is prepared on a quantum computer, the parameters of which are selected using classical optimization methods.
  The objective is to prepare the state |ψ(~θ)〉 which can be used to approximate the ground state of a given Hamiltonian H by the variational principle
  Emin ≤ min ~θ (〈ψ(~θ)|H|ψ(~θ)〉) , (1)
  where Emin is the true ground state energy of H . The parametrized state is prepared by applying U(~θ), which is a parametrized quantum circuit (typically with a fixed architecture), where the angles ~θ = (θ1...θn) specify the rotation angles of the local unitary rotations present in the circuit. This circuit, known as the ansatz is applied to an initial state |ψ0〉, usually chosen to be the fiducial “all zero” state |00...0〉, to prepare the state |ψ(~θ)〉 = U(~θ)|ψ0〉. It is well established that the structure of the ansatz can dramatically influence the VQE’s performance [4, 5], as the closeness of the estimated ground state to the true one depends on the state manifold accessible by the ansatz. Thus finding new architecture construction methods could lead to breakthroughs in quantum variational methods for chemistry (e.g. for strongly-correlated systems, for which standard ansatzes might fail), but also in other domains which utilize variational circuits such as machine learning and optimization [6, 7, 8, 9].
  Currently, the foremost ansatzes fall primarily in two classes: chemistry-inspired (e.g. the unitary coupled-cluster ansatz [2, 10]) and hardware-inspired (e.g. the hardware efficient ansatz [11]). Architectures from both of these classes entail using a fixed architecture [2, 10, 11, 12] determining the unitary U(~θ), and hence the corresponding ansatz. The ansatz circuit is then usually decomposed into two-qubit CNOT and one-qubit rotation gates parametrized by (θ1, θ2, . . . , θn) ∈ [0, 2π]n to be optimized by a classical subroutine. However, the architecture itself can also be optimized. This results in a hard structure optimization problem, as it is a combinatorial optimization problem that must balance two competing factors. On the one hand, the ansatz needs to be expressive enough to guarantee a good approximation of the ground state energy. On the other hand, the depth and size of the circuit need to be controlled in order for the latter to be compatible with NISQ restrictions.
  Contributions In this work, we propose a general optimization procedure for VQE based on deep reinforcement learning (RL) which is designed to yield quantum circuits that are both gate and depth-efficient. We supplement this RL approach with curriculum learning [13, 14], a powerful machine learning method for solving complex problems by leveraging the solutions of previouslysolved simpler instances. Specifically, we introduce a feedback-driven curriculum learning method that autonomously adapts the complexity of the learning problem to the current performance of the learning algorithm. We apply our architecture to the well-established benchmarking problem of finding the ground state energy of the LiH molecule [15, 3] and observe chemical accuracy while maintaining a low-depth quantum circuit and achieving state-of-the-art results in terms of gate efficiency. Apart from ground state estimation, the proposed method has a wider range of applications, the algorithm can be applied to any variational-circuit-based algorithm.
  The rest of the paper is organized as follows. In Section 2, we discuss the related works. In Section 3, we introduce our method with simplistic assumptions about the exact value of the ground state energy. At the end of this section, we provide a method circumventing this unrealistic requirement. As an initial proof-of-concept, in Section 4, we present the results assuming access to the exact value of energy. In Section 5, we provide the results for the full method that utilizes a rough proxy of energy obtained with classical methods. We conclude the paper with a discussion of possible future directions.
sections.heading: 2 Related Work
sections.text: The automatic construction of quantum circuits with the use of heuristics/machine learning methods is a topic that has recently been gaining more attention [16, 17, 18, 19, 20]. In the recent article [21], the authors explore various methods of deep reinforcement learning for quantum architecture search. In contrast to our approach the authors generate circuits which are designed to prepare multi-qubit GHZ states. Several works concern the construction of quantum circuits in the context of quantum chemistry. In [22] the authors propose the Quantum Architecture Search algorithm, based on the AutoML method – Neural Architecure Search, which is benchmarked on the H2 molecule – a simpler system than LiH. On the other hand, more complex molecules are considered in [23], the authors use evolutionary algorithms to optimize the structure of VQE circuits for quantum chemistry and combinatorial optimization. The gate set in this algorithm is composed of one-qubit universal gates and two-qubit controlled universal gates, which after the optimization procedure, are decomposed into CNOT gates and one-qubit universal rotation gates. Another evolutionary strategy was explored in [24], where a multi-objective genetic algorithm is used to optimize the structure of the VQE ansatz in the context of quantum chemistry problems. The topology of the circuits is optimized by a non-dominated sorting genetic algorithm while the parameters are globally tuned by Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [25]. Yet another different approach, explored in the manuscript [26], involved applying a set of rules to both grow and remove quantum gates during the optimization. The last three manuscripts focus on solving problems in quantum chemistry by constructing the shortest possible quantum circuits.
sections.heading: 3 Methods
sections.text: In this section, we discuss our approach for constructing VQE ansatzes using RL. We discuss the state and action representations, as well as the reward function used in this work. We then introduce a novel adaptation of curriculum learning applied to VQE circuit construction with different parameter optimization strategies.
sections.heading: 3.1 Ansatz optimization as a reinforcement learning problem
sections.text: We describe ansatz optimization as a reinforcement learning environment, where the states encode the current circuit. The actions encode a gate that is added to the current circuit. The environmental transitions are deterministic because the subsequent state is defined by the previous state appended by the taken action. At each time step t, the agent receives the reward R according to the following formula:
  R =  5 if Et < ξ −5 if t ≥ L and Et ≥ ξ max ( Et−1−EtEt−1−Emin ,−1) otherwise , (2)
  The goal of the agent is to reach the minimal energy Emin within a predefined threshold ξ. The energy Et is calculated for the current circuit representing the quantum state |ψ(~θ)〉 according to:
  Et = 〈ψ(~θ)|H|ψ(~θ)〉, (3) where ~θ describes the angles of the rotation gates. The vector ~θ needs to be separately optimized to obtain the reward R. This optimization subroutine is a crucial hyperparameter of this RL environment and it is explained in more detail in the following two sections. The details on the reward shaping are discussed in Supplementary Materials.
  We utilize deep RL methods, thus the states and actions are represented in a neural network-friendly form. Namely, each state is represented as an ordered list of layers that are composed of single quantum gates. Thus the environmental state, which is represented by this list, fully describes the whole circuit (with a specified gate ordering). For constructing the circuits, we use CNOT and one qubit rotation gates – which are realizable on currently available quantum devices. The CNOT gates are encoded by two values indicating position of control and target qubits (the position is enumerated starting from 0). The rotation gates are encoded by two integers (starting from 0). The first number indicates the qubit register and the second specifies the axis of rotation (X – 1, Y – 2, Z – 3). We deliberately omit the continuous parameters describing the angles of the rotation gates from the state representation. Instead, the energy estimated for the state is appended to state representation. The
  user needs to define the maximum number of circuit layers (L). If a given circuit has fewer layers than L, the remaining ones are filled with identity operators (c.f. Fig. 1). The agent constructs the circuit iteratively, starting from an empty list, and adding a single quantum gate at the end of the current circuit. This way the agent starts each episode with exactly the same conditions, i.e. an empty state, avoiding an additional source of randomness tied to the initial state. Thus, the set of all possible actions is equivalent to the set of all possible quantum gates that the agent can select. The size of the set of all possible actions for the agent is then 3|Q|+2 (|Q| 2 ) = |Q|(|Q|+2), where |Q| is
  the number of qubits. The agent can choose from 3|Q| single-qubit gates and from 2 (|Q|
  2
  ) two-qubit
  gates. We use a simple one-hot encoding for the actions.
sections.heading: 3.2 Specification of the agent and the environment
sections.text: We employ a Double Deep-Q network [27] (DDQN) with an -greedy policy and an ADAM optimizer for better stability. More technical details about the Reinforcement Learning procedure are described in the Experiments section (Section 4). As described in the previous section, to obtain a reward R for each circuit, an optimization subroutine needs to be applied to determine the continuous values of the rotation gate angles. We use well-developed methods for continuous optimization such as Constrained Optimization By Linear Approximation (COBYLA) [28] and Rotosolve [29]. Whilst we could use an agent for this purpose as well, we abstain from doing so for two main reasons. Firstly, in this work, we aim at assessing the efficacy of RL for circuit structure optimization, rather than for parameter searching. The latter is investigated extensively in other literature and thus treated as a separate subroutine that can be solved by other, more developed, tools. The second reason is practical, as training the agent to reliably predict values of angles would result in significantly longer training periods.
  For the above reasons, our approach is hybrid: the agent learns how to, and is rewarded for, constructing particular circuits given a set of parameters determined by an independent optimization algorithm. We apply the angle optimization subroutine only after the steps in which the agent appends a rotation gate to the circuit, thus eliminating the necessity of including the parameters in the state representation. The number of angles optimized at a given step is a hyperparameter in our method, the choice of which will be analyzed in Sec. 4.2. In this work, we consider optimizing all angles at once (global strategy), as well as optimizing a few angles at a time (local strategy) . These different experimental settings allow us to check whether the rough energy approximation obtained from only optimizing a subset of the circuit angles is sufficient to create sufficiently good circuits.
sections.heading: 3.3 Feedback-driven curriculum learning
sections.text: As explained, the agent is rewarded for creating circuits that pass a given threshold. In our experiments, we set the threshold to so-called chemical accuracy, which corresponds to an energy that differs from the ground state energy by less than 0.0016 Hartree. Even in cases where the exact value of minimum energy is known, setting the value of threshold ξ directly to chemical precision from the beginning of the training may result in the agent failing due to poor exploration. A possible solution is to introduce the agent to tasks of increasing difficulty known as curriculum learning [13] which was found to be beneficial for neural network training [14]. However, this approach requires the user to design a task sequence which, without a priori knowledge of the relationship between
  the threshold value and the corresponding task difficulty, may lead to the agent reaching undesired local minima.1
  Inspired by the body of works exploring automatic curriculum learning reviewed in RL [30], we propose a feedback-driven curriculum learning method in which the difficulty of the current task is dynamically adjusted based on the current performance of the agent. Hence, the agent does not rely on a human-defined task schedule and builds its knowledge gradually paced by the agent’s performance. The proposed method, illustrated in Fig. 2, starts with an arbitrary threshold ξ1 that is decreased according to two rules. The first rule decreases the threshold greedily to the best energy obtained so far ξ2 after each G episodes. Additionally, the amortization factor δ is added to the threshold at the moment of a greedy shift and each time the agent fails to solve multiple consecutive episodes. To gradually decrease the amount of added amortization δ, after a few successful episodes we decrease the amortization δ by a certain factor until the agent reaches the energy that was greedily obtained at first. Since in this feedback-driven curriculum learning method the threshold is successively decreased, we will also refer to this approach as moving threshold.
  Note that due to the dynamic nature of our reward system we can substitute the true value of Emin, which is in practical cases unknown, with an arbitrary value α that is lower than minimum energy Emin and set a threshold ξ to a value that guarantees that the exact energy falls into the range Emin ∈ (α, α+ ξ). Note that, in principle, the method does not pose any restrictions on the starting threshold and the agent can start learning from any point, so we can set the ξ arbitrarily big to make sure that the Emin is in the range. The α value can also be set in the same vein, however, a simple method to roughly estimate the lower bound of energy can be utilized instead. If the agent is able to create circuits with chemical precision relative to any (reasonably) higher energy, the training with ever-increasing difficulty will ultimately lead to the chemical accuracy without computing the exact energy. Experiments proving the validity of this argument are presented in Section 4.2.
sections.heading: 4 Experiments
sections.text: We start this section with a description of the chemical problems on which we evaluate the proposed approach. Next, we will describe the experimental setup and present our results.
sections.heading: 4.1 Experimental setup
sections.text: In our analysis of RL for VQE circuit synthesis, we focus on the problem of finding the ground state energy of the lithium hydride (LiH) molecule for various intramolecular distances, as well as Hamiltonians on differing qubit numbers, which stem from different approximations of the true chemical problem. All experiments are divided into two parts. In the first part, we explain how the choice of optimizer and number of optimized parameters per step impact the performance of the RL
  1We tested multiple hard-coded schedules for shifting the threshold. However, the algorithm did not pass the desired lowest threshold while using these pre-defined sequences of thresholds.
  approach. This part is evaluated on less challenging problem instances, i.e. with a smaller quantum system, action space, and state space. The second part, on the other hand, will be performed on the same molecule but simulated on a higher dimensional system. We assume that the number of measurements is sufficiently large that sampling noise is negligible. Whilst this is a significant assumption to make, and a current limitation of our method, in this work we are interested in a proof of concept for RL-based circuit synthesis. Thus we do not include the additional overhead incurred by finite measurements, which is left for future work.
  Quantum chemistry problems. As mentioned earlier, in this work we focus on finding the ground state energy of the LiH molecule, although our approach design is not limited to this molecule. The Hamiltonians are computed in STO-3G basis. In the first part of our analysis, we consider a simpler approximation of LiH molecule which, after taking into account the symmetry of the molecule and removing the orbitals with weak interaction, results in a Hamiltonian defined on 4 qubits. Thus, the state space and action space are significantly smaller than the ones for the full LiH Hamiltonian. In this model, we utilize the parity mapping to convert molecular Hamiltonian to qubit Hamiltonian [31]. We examine proposed method on three values of LiH bond distances, 1.2Å, 2.2Å and 3.4Å. In the second experiment, we use the larger Hamiltonian which only takes into account the symmetry of the molecule, and is therefore defined on 6 qubits, for which we use the Jordan-Wigner mapping [32]. We opted to switch from the parity to the Jordan-Wigner mapping when considering the 6 qubit Hamiltonian in order to compare our results with previous literature which tackled this problem using a different approach [23]. We only focus on a single geometry for this case, i.e. for bond distance 2.2Å. All Hamiltonians were generated using the Qiskit library [33].
  Implementation details.2 In all experiments we utilize n-step DDQN algorithm, with the discount factor set to γ = 0.88, and the probability of a random action being selected is set by an ε−greedy policy, with ε decayed in each step by a factor of 0.99995 from its initial value = 1, down to a minimal value ε = 0.05. The memory replay buffer size is set to 20,000. The target network in the DDQN training procedure is updated after every 500 actions. After each training episode, we included a testing phase where the probability of random action is set to = 0 and the experience replay procedure is turned off, i.e. the experiences obtained during the testing phase are not included in the memory replay buffer.
  In our experiments, we use differing step sizes in the n-step trajectory rollout updates [34], which specify exactly how the Q-function approximations are updated. This hyperparameter of the model is set to n = 1 for the first part of the experiments. The reported results correspond to the experiment performed with the value n = 6. In the moving threshold approach, the threshold is changed greedily after 2000 episodes with an amortization radius of 0.0001. After 50 successfully solved episodes, the amortization radius is decreased by 0.00001. The initial threshold value is set to ξ = 0.005. Simulations of quantum circuits were performed using Qulacs library [35] (MIT License). The hyperparameters were selected through coarse grain search. The employed network is a fully connected network with 5 hidden layers with 1000 neurons each for the 4-qubit case and 2000 neurons each for the 6-qubit case. The maximal number of gates is equal to 40. All experiments were performed on three computing clusters - 4 Titan RTX GPUs, 4 Titan V GPUs, and 4 Tesla V100 GPUs.
  Evaluation. To validate the capabilities of the reinforcement learning approach we compare it with well-established architectures, namely the Hardware Efficient (HE) [11] ansatz and UCCSD [36, 37] ansatz. The number of layers of the HE ansatz is tailored to each Hamiltonian considered. We report the smallest number of layers for which we have achieved chemical accuracy using our chosen optimizers. Moreover, the naive approach of UCCSD is used, and it is likely possible to find a shallower implementation of this ansatz, but for the purposes of this manuscript, it is an acceptable benchmark.
  In all experiments, we compare the minimal depth and the number of gates of the obtained circuits (for more details see Supplementary Materials). By depth, we mean the length of the longest path between input and the output along qubit wires, without taking into account quantum gate dependency. Each RL experiment runs on 10 trials, i.e. experiments run on 10 different random seeds. For the first part of experiments, we consider two optimization methods: Rotosolve and COBYLA.
  2The code is available on https://github.com/mostaszewski314/RL_for_optimization_of_VQE_ circuit_architectures
  Both of these optimization methods can be applied with a local strategy, where the optimizer updates the last five rotation gates after each agent step, or a global strategy, where after each time step a full update is performed. For the local optimization strategy, we set the number of available iterations of Rotosolve to 5, whilst for the global strategy, the number of iterations is chosen to be 25. By iterations (for Rotosolve) we mean one complete cycle updating all the parameters under consideration. We chose 100 iterations for COBYLA for both strategies. Due to the fact that the number of iterations is fixed, convergence is not always guaranteed. However, in most cases, these values were sufficient and more iterations did not improve the energy estimate.
  In the LiH 4-qubit experiments, we were able to establish the strategy and the optimizer to use in the LiH experiments with 6-qubits. For the setup with 6-qubit, we chose global optimization in each step and COBYLA with 200 iterations.
sections.heading: 4.2 Results
sections.text: LiH - 4-qubit - different bond distances. In this section, we discuss the results of the LiH experiments with 4 qubits. In Fig. 3, the depth (left) and the total number of gates (right) of the circuits which achieve chemical accuracy are presented, with the average taken over the minimal values from the different trials. For bond distances 1.2Å and 2.2Å, the agent proposed quantum circuits that were shallower and contained fewer gates than the standard approaches, in every trial. For bond distance 3.4Å, the agent using local COBYLA optimization found quantum circuits satisfying chemical accuracy in 2 out of 10 trials, whereas the agent using local Rotosolve did so in 3 out of 10 trials. On the other hand, when using global optimization, the agent found circuits satisfying chemical accuracy in every trial, regardless of the optimizer used. In almost all cases, the average minimal depth and gate count were less than those obtained using standard approaches, i.e. HE and UCCSD, the exception being when the agent used local Rotosolve, which resulted in circuits deeper than the HE ansatz. Given the above results, COBYLA seems to outperform Rotosolve for this task. Moreover, taking into account the number of successful trials, we conclude that our approach is significantly improved when all angles are optimized.
  LiH - 6-qubits - moving threshold. In this section, we discuss the results of the LiH experiments with 6 qubits. We evaluated the fixed threshold approach on the 6-qubit LiH Hamiltonian case,
  which, however, did not give any positive results. The experiments where global optimization after each step was used did not yield any circuits achieving chemical accuracy. This seems to stem from the reward function with the chemical accuracy threshold being too sparse. Whilst gradually decreasing the threshold is the obvious next step, we opted for the curriculum learning approach based on the moving thresholds, in order to decrease the threshold automatically rather than manually. As one can see in Table 1, curriculum agents provide better results than standard approaches. To the best of our knowledge, the only previous work tackling this problem for the 6 qubit LiH Hamiltonian is [23]. Whilst we cannot quantitatively compare our results with their work due to differing gate compilations, we note out that our approach seems to generate circuits roughly 5 times shallower than the ones obtained in [23].
  By looking closely at how the moving threshold guides the agent in the direction of chemical accuracy, we can analyze the mean difference between the exact energy and the energy estimate using the agent’s circuit from the end of each episode. In Fig. 4 we plot this error for a particular case at different scales to understand better what happens at different training stages. The blue dots represent the error in the agent energy estimates, whilst the orange curve represents the moving threshold. The left of Fig. 4 shows this error for the entire training period but does not provide much information due to the large deviation in the error. On the other hand, in the right plot of Fig. 4, we focus more on episodes which resulted in errors closer to chemical accuracy. The agent is guided by a decreasing moving threshold, with the threshold occasionally increasing abruptly due to amortization, which helps the agent adapt to a new lower threshold.
sections.heading: 5 Learning procedure with lower-bound approximation to the ground-state energy
sections.text: So far we have relied on the assumption that the agent uses the energy error with respect to the exact solution as a learning metric. This assumption in practice is of course not reasonable, as estimating the energy to this precision is the main objective of VQE algorithms to start with. One way to relax the assumption of knowing the exact energy is to provide reasonable information about the lower and upper bounds, and classical and cheaper methods, respectively, and which incurs only a logarithmic overhead in the precision by using a binary search for the best achievable energy. Another solution stems from our moving threshold approach.
  The moving threshold approach discussed in the previous sections not only allows us to solve more demanding problems, like the 6-qubit Hamiltonian but also removes the requirement of knowing the exact energy. In order to demonstrate this method, we performed an additional experiment on the 4- qubit LiH Hamiltonian at bond distance 2.2Å. Instead of using the exact energy, i.e. ≈ −7.8448Ha, we used the negative sum of the absolute values of the Hamiltonian’s Pauli coefficients, i.e. for a HamiltonianH = ∑M j=1 cjPj , we take− ∑M j=1 |cj | to guide the agent. This is an example of a very rough easy-to-compute lower bound on the energy but requires the Hamiltonian to be local. For this LiH geometry, this weight sum evaluates to −10.0604Ha. We set the amortization radius to 0.005, the number of episodes after greedily shifting the threshold to 500, and the amortization radius is decreased after every 25 successfully solved episodes. The initial threshold value we set to ξ = 4. We used global COBYLA with 100 iterations to optimize the angles of the quantum circuit.
  As one can see in Fig. 5, the agent successfully manages to reach the exact energy within chemical accuracy, despite having a lower bound below the exact value. Moreover, it can be seen that the initial threshold is well above the error that the agent reaches, which also removes the requirement of human input to set the threshold schedule. Similar experiments were performed on bond distances 1.2Å and 3.4Å with the same results.
sections.heading: 6 Discussion
sections.text: The molecular electronic structure problem is a promising near-term application for quantum computers. The performance of a chosen variational ansatz depends on the structure and depth of its corresponding circuit. Near-term quantum devices pose restrictions on the depth of the VQE circuit. Thus, VQE structure optimization for near-term quantum computers is a multi-objective optimization problem that requires maximizing the expressivity and reducing the depth of the corresponding
  circuits. Reinforcement learning by design aims at maximizing the discounted sum of rewards, and as a consequence, this will naturally lead to finding shorter better-performing solutions. This property of RL is one of the main reasons we selected this method.
  In this work, we present a deep reinforcement learning architecture with the feedback-driven curriculum learning to optimize the structure of VQE circuits. The goal of this approach is to design circuits that estimate the ground state energy of molecules within chemical accuracy while keeping the circuit depth as low as possible. In our approach, instead of confronting the RL agent with the full-scale problem from the beginning, the agent first starts training on a simpler instance and autonomously adjusts the task complexity until chemical accuracy is achieved. In particular, we demonstrate that our approach yields the ground state energy of LiH within chemical accuracy for several bond distances. While other VQE approaches also reach chemical accuracy for this benchmark, our approach consistently outperforms the others in terms of circuit depth. Hence, the unique combination of deep RL and feedback-driven curriculum learning for structure learning yields an interesting new approach for solving VQE problems.
  We emphasize that whilst we focus specifically on the molecular electronic structure problem for LiH, the reinforcement learning approach presented in this paper can be adapted for optimizing other VQE architectures [38, 7]. The method is also directly compatible with other reinforcement learning algorithms, and the angle optimization algorithms are not restricted to the methods investigated in this paper.
  In summary, the methods proposed in this work provide state-of-the-art results in electronic structure problem experiments and lay the foundations for the application of deep reinforcement learning methods to VQE optimization problems. Moreover, as is often the case when new methods are employed, we envision numerous follow-up avenues which will reveal the true capacities of such automated ML methods for VQE-type problems.
  Broader Impact
  Quantum computers may offer significant improvements in chemistry, with applications in drug and material design that could have a widespread positive impact on society (e.g. in the discovery of novel effective pharmaceuticals).
  Our work presents novel approaches for enhancing VQE-based methods targeting quantum chemistry problems and thus contributes to this objective. In particular, this research focuses on the use of reinforcement learning to automatically program existing quantum devices. Whilst our work mainly focuses on finding the LiH molecule ground state energy, the benefits of such a technique extend to research questions that can be reformulated as a VQE optimization problem.
  We foresee no negative impact stemming from our research, no significant consequences from system failures, nor do we believe our methods leverage any bias in any data.
  From the energy consumption perspective, we estimate that the single experiment takes from 12 to 48 hours of computation time using a single CPU. We did not perform any experiments on a QPU machine.
  Acknowledgments and Disclosure of Funding
  This manuscript builds on the results presented in the NeurIPS workshop paper [39]. MO acknowledge the support of the Foundation for Polish Science (FNP) under grant number POIR.04.04.0000-17C1/18-00. LMT acknowledges the support from the Austrian Science Fund (FWF) through the projects DK-ALM:W1259-N27 and SFB BeyondC F7102. This work was also supported by the Dutch Research Council(NWO/OCW), as part of the Quantum Software Consortium programme (project number 024.003.037). VD and ES acknowledges the support of SURFsara through the QC4QC project. This research was partially funded by the Grant of Priority Research Domain at Warsaw University of Technology - Artificial Intelligence and Robotics. The authors also wanted to thank Arthur G. Rattew for helpful discussions.
title: Reinforcement learning for optimization of variational quantum circuit architectures