Ordering of magnetic impurities and tunable electronic properties of topological insulators*

We study collective behavior of magnetic adatoms randomly distributed on the surface of a topological insulator. As a consequence of the spin-momentum locking on the surface, the RKKY- type interactions of two adatom spins depend on the direction of the vector connecting them, thus interactions of an ensemble of adatoms are frustrated. We show that at low temperatures the frustrated RKKY interactions give rise to two phases: an ordered ferromagnetic phase with spins pointing perpendicular to the surface, and a disordered spin-glass-like phase. The two phases are separated by a quantum phase transition driven by the magnetic exchange anisotropy. Ferromagnetic ordering occurs via a finite-temperature phase transition. The ordered phase breaks time-reversal symmetry spontaneously, driving the surface states into a gapped state, which exhibits an anomalous quantum Hall effect and provides a realization of the parity anomaly. We find that the magnetic ordering is suppressed by potential scattering. Our work indicates that controlled deposition of magnetic impurities provides a way to modify the electronic properties of topological insulators.