Two-tone optomechanical instability and its fundamental implications for backaction-evading measurements*

We report a new type of optomechanical instability that arises in two-tone backaction-evading (BAE) measurements of mechanical motion, a protocol designed to overcome the standard quantum limit. We demonstrate the effect both in the optical and microwave domains using different optomechanical systems, and find excellent agreement with theory. In contrast to the well-known parametric instability that occurs in single-tone, blue-detuned pumping, and results from a two-mode squeezing interaction between the optical and mechanical modes, the two-tone instability results from single-mode squeezing of the mechanical mode due to small detuning errors in the two pump frequencies. The instability can occur even with balanced intracavity fields and for both signs of detuning errors. The required tuning accuracy increases with pump power, putting an intrinsic limit on the sensitivity of BAE measurements and on other two-tone schemes.