Measuring Quantum Optomechanical Self-induced Oscillations: Photon Correlation and Homodyne Tomography*

Motivated by recent experimental advances in fabricating systems with large optomechanical couplings, we study the self-induced mechanical oscillations in the strong quantum regime for a single cell optomechanical system. We show that, under strong optomechanical coupling $g_M\ge\kappa$, the \emph{persistent }state of the mechanical oscillator can have non-classical, \emph{strongly negative} Wigner density, which can be measured by non-destructive homodyne tomography. We further propose to detect the onset of the quantum self-induced oscillation using the easier-to-measure photon two-point correlation functions $g^{(2)}(t)$. We show that there are two distinct signatures in the long-term time-average and the line-shape of $g^{(2)}(t)$ at the onset of self-induced oscillations. We show that $g^{(2)}(t)$ exhibits long-term coherence extending much beyond the optical decay time $1/\kappa$, the decay of which in the red- and blue-detune regime we explain using models of optomechanical cooling and phase noise.