Invariance in Deep Network Learning: Mathematical Representation, Probabilistic Symmetry, Variable Exchangeability, and Sufficient Statistics*

Algebraic Lie groups provide the foundation for describing many physical symmetries, which play a key role in studying the geometry of smooth manifolds and analyzing complex high-dimensional observations. In particular, Noether’s symmetries articulate the correspondence between physical symmetries and conservation quantities, which often correspond to physical laws that can be prescribed as differential equations. This interplay between mechanical dynamics, symmetries, information, and geometry characterizes the importance of Lie group actions on modeling classical physical systems, obtaining rigorous statistical inference, and training of deep artificial intelligence (AI) networks. 

Lie group characterize data symmetries in the underlying learning problems. Classical machine learning approaches for examining symmetries include data augmentation (data-driven) and architectural modifications to construct invariant models through weight constraint designs (model-based). A model-based G-invariant design often composed several equivariant functions followed by a final invariant function. A concurrent work suggests that building group invariance and partial invariance into string theory Kreuzer Skarke dataset improve model performance. Non group invariant models benefit from group invariant preprocessing.

From a statistical point of view, classical symmetry is related to probabilistic distributional symmetry, where exchangeability and stationarity are the primary examples. For instance, a sufficient sample statistic contains all the information needed for an inferential procedure and is directly related to probabilistic symmetry. That is, sufficiency describes the information that is relevant to the statistical inference. Probabilistic symmetry and invariance identify information that is irrelevant to the statistical inference. 

In this talk, we will review and investigate the relations between mathematical invariance, DNN invariants, (probabilistic) symmetries, physical modeling, PDE solutions, geometries and information.