Thinking in machines, not statistics*

When asked to summarize a long string of data, we can either model the trajectory distribution directly or infer machines that could have likely produced the observed trajectory. I will argue that thinking in terms of machines, rather than in terms of trajectory distributions, can lead to improved inference algorithms and more accurate plug-in estimators of various information-theoretic quantities. I will focus on the predictive information bottleneck as an illustrative example.