*

Excitonic insulator is a quantum coherent phase resulting from formation of a macroscopic population of electron-hole pairs. For a semimetal with narrow overlap of the conduction and valence bands, a finite exciton binding energy could lead to excitonic instability. Candidate material Ta₂NiSe₅ shows a second-order structural phase transition at T_c=328K, with two mirror symmetries broken.

Using polarization-resolved Raman spectroscopy, we demonstrate that Ta₂NiSe₅ is an excitonic insulator below T_c [npj Quantum Mater. 6, 52 (2021)]. The order parameter of the structural transition is of quadrupolar symmetry. In this symmetry channel, we observe strongly coupled excitonic and optical phonon modes at low energy. The resulting Fano lineshape can be decomposed to reveal an overdamped exciton mode that exhibits critical softening on cooling towards T_c. In contrast, the energy of the bare optical phonons increases on cooling, which indicates that the phase transition does not result from lattice instability. We conclude that the transition is driven by the critical excitonic mode, and the transition temperature is enhanced by the coupling of this mode to the lattice modes of the same symmetry.

We further show that for Ta₂Ni(Se_1−xS_x)₅ the excitonic instability, the transition temperature T_c, and the magnitude of the structural change across T_c are suppressed with increasing sulfur content x [PRB 104, 045102 (2021); arXiv 2104.07032 (2021)]. The Fano lineshape at low energy persists up to x=0.67, up to which point Ta₂Ni(Se_1−xS_x)₅ has a semimetallic high-temperature phase. However, Ta₂NiS₅ is a semiconductor with more than 0.3 eV gap; the asymmetric lineshape is absent because the exciton mode appears at high energy. As both the exciton and optical phonons of Ta₂NiS₅ do not show critical softening, we conclude that its phase transition is driven by ferroelastic instability.