CrI₃ has received renewed interest in the past few years and shows intimate lattice-magnetism coupling. It is reported in few-layer CrI₃ that monoclinic and rhombohedral layer stackings favor interlayer antiferromagnetic (AFM) and ferromagnetic (FM) couplings respectively. However, the question remains open how the lattice structure responds to the magnetic phase transitions in few-layer CrI₃. Here, using polarization-resolved magneto-Raman spectroscopy, we have investigated layer-number, temperature, and magnetic field dependence of Raman spectra. We find that while the fully symmetry A_g and the antisymmetric magnetism-coupled phonon modes show little indication of monoclinicity in few-layer CrI₃, the doubly degenerated E_g mode of monolayer CrI₃ splits in N-layer (N > 1) flakes and suggests the monoclinic crystal structure. We further show that the aforementioned split in the frequency decreases at lower temperatures, suggestive of the tendency towards the monoclinc-to-rhombohedral structural phase transition that is present in the bulk. Finally, we present the magnetic-field dependence where a further monoclinic distortion is revealed across the magnetic field-induced layered AFM to FM phase transition.