In a growing number of materials, phonons have been found to generate a thermal Hall effect, but the underlying mechanism remains unclear. Inspired by previous studies that revealed the importance of Tb³⁺ ions in generating the thermal Hall effect of Tb₂Ti₂O₇, we investigated the role of Tm³⁺ ions in TmVO₄, a paramagnetic insulator with a different crystal structure. We observe a negative thermal Hall conductivity in TmVO₄ with a magnitude such that the Hall angle, |K_xy / K_xx|, is approximately 1 x 10^-3 at H = 15 T and T = 20 K, typical for a phonon-generated thermal Hall effect. In contrast to the negligible K_xy found in Y₂Ti₂O₇, we observe a negative K_xy in YVO₄ with a Hall angle of magnitude comparable to that of TmVO₄. This shows that the Tm³⁺ ions are not essential for the thermal Hall effect in this family of materials. Interestingly, at an intermediate Y concentration of 30 % in Tm_1-xY_xVO₄, 

K_xy was found to have a positive sign, pointing to the possible importance of impurities in the thermal Hall effect of phonons.