Reading List From MIT Probabilistic Computing Project

Reading List

These papers, books, and essays serve as a shared basis of knowledge for the MIT Probabilistic Computing Project.

Non-Technical Background

• Artificial Intelligence — The Revolution Hasn’t Happened Yet by Michael I. Jordan
• The Structure of Scientific Revolutions by Thomas Kuhn
• The Innovator’s Dilemma: The Revolutionary Book That Will Change the Way You Do Business by Clayton Christensen
• Epigrams on Programming by Alan Perlis (Turing award page, Foreword to SICP)
• The Effective Executive: The Definitive Guide to Getting the Right Things Done by Peter F. Drucker
• Technology and Courage by Ivan Sutherland
• Mathematical Writing by Don Knuth
• You and Your Research by Richard Hamming
• Whirlwind I Computer
• Worse Is Better by Richard Gabriel
• Who Inspired the Leamer-Rosenthal Prizes? Part II by Ed Leamer
• The Laws of Medicine: Field Notes From an Uncertain Science by Siddhartha Mukherjee (Interview)
• The War of Art: Winning the Inner Creative Battle by Steven Pressfield
• Study of Telemedicine Finds Misdiagnoses of Skin Problems by Melinda Beck
• Many Analysts, One Dataset: Making Transparent How Variations in Analytical Choices Affect Resultsby Silberzahn et al.
• What Is It Like To Be A Bat? by Thomas Nagel
• When U.S. air force discovered the flaw of averages by Todd Rose
• The origins of SageMath – creating a viable open source alternative to Magma, Maple, Mathematica, and Matlab by William Stein
• Politics and the English Language by George Orwell
• The Sciences of the Artificial by Herbert Simon
• Building Robust Systems: An Essay by Gerald Jay Sussman
• Statistics: What is the question? by Leek and Peng

Technical Background

• Mathematics: Its Content, Methods and Meaning: Chapter 11. The Theory of Probability by A. N. Kolmogorov
• Structure and Interpretation of Computer Programs by Harold Abelson, Gerald Jay Sussman, and Julie Sussman
• Classifier Technology and the Illusion of Progress by David J. Hand
• Machine Learning: The High-Interest Credit Card of Technical Debt by Sculley et al.
• On Computable Numbers and the Entscheidungsproblem by Alan Turing (Skim)
• Noncomputable Conditional Distributions by Ackerman, Freer, and Roy (Skim)
• Natively Probabilistic Computation; PhD Dissertation by Vikash Mansinghka
• Design of LISP-Based Processors by Steele and Sussman
• Lisp: A Language for Stratified Design by Hal Abelson and Gerlad Sussman
• Steps Toward Artificial Intelligence by Marvin Minsky
• On Chomsky and the Two Cultures of Statistical Learning by Peter Norvig
• Statistical Modeling: The Two Cultures by Leo Breiman
• [Critique of Paper by “Deep Learning Conspiracy” (Nature 521 p 436)]39] by Jürgen Schmidhuber
• Artificial Intelligence A Modern Approach (Introduction p. 3-31) by Russell and Norvig
• Confessions of a Used Programming Language Salesman by Erik Meijer

Probabilistic Programming

• VentureScript Tutorial
• BayesDB Documentation
• Picture: an imperative probabilistic programming language for scene perception by Kulkarni, Kohli, Tenenbaum, and Mansinghka
• Bayesian Statistics Without Tears: A Sampling-Resampling Perspective by Smith and Gelfand
• When are probabilistic programs probably computationally tractable? Freer, Mansinghka, and Roy
• Bayesian Optimization as a Probabilistic Meta-Program; MEng thesis by Ben Zinberg

Probabilistic Programming (Optional, Outdated)

• Venture: a general purpose probabilistic programming platform with efficient stochastic inference by Mansinghka, Selsam, and Perov
• A stochastic programming perspective on nonparametric Bayes by Roy, Mansinghka, Goodman and Tenenbaum

Foundations of Probabilistic Modeling (Required)

• Bayesianism and Causality, or, Why I am Only a Half-Bayesian by Judea Pearl
• Why I Am Not a Bayesian by Clark Glymour
• Probability Theory: The Logic of Science (Chapters 1-3) by E. T. Jaynes
• Information Theory, Inference, and Learning Algorithms (Chapter 28, Model Comparison and Occam’s Razor) by David MacKay

Standard Library of Modeling Techniques

• Information Theory, Inference, and Learning Algorithms (Chapter 45, Gaussian Processes) by David MacKay
• Unifying Rational Models of Categorization via the Hierarchical Dirichlet Process by Griffiths, Canini, Sanborn, and Navarro
• Discovering latent classes in relational data by Kemp, Griffiths, and Tenenbaum
• Topics in Semantic Representation by Griffiths and Steyvers
• Finding Scientific Topics by Griffiths and Steyvers
• Continuous Time Dynamic Topic Models by Wang, Blei, and Heckerman
• Practical Bayesian Optimization of Machine Learning Algorithms by Snoek, Larochelle, and Adams
• A Unifying Review of Linear Gaussian Models by Roweis and Ghahramani

Standard Library of Inference Techniques

• Composable Probabilistic Inference with Blaise by Keith A. Bonawitz
• An Introduction to MCMC for Machine Learning by Andrieu, De Freitas, Doucet, and Jordan
• The Markov Chain Monte Carlo Revolution by Persi Diaconis
• Convergence of Sequential Monte Carlo-based Sampling Methods by Huggins and Roy

Useful Examples (Optional)

• MrBayes: Bayesian Inference of Phylogenetic Trees by Huelsenbeck and Ronquist [pdf] [webpage]
• FastSLAM: FastSLAM: A Factored Solution to the Simultaneous Localization and Mapping Problem
• First few chapters of Bayesian Spectrum Analysis and Parameter Estimation by Bretthorst
• Recovering Intrinsic Images from a Single Image by Tappen, Freeman, and Adelson
• Bayesian Inference for PCFGs via Markov chain Monte Carlo by Johnson, Griffiths and Goldwater
• Global Seismic Monitoring as Probabilistic Inference by Arora, Russell, Kidwell, and Sudderth

Automatic Model Discovery (Optional)

• CrossCat: A Fully Bayesian Nonparametric Method for Analyzing Heterogeneous, High Dimensional Data by Mansinghka et. al.
• Structure Discovery in Nonparametric Regression through Compositional Kernel Search by Duvenaud, Lloyd, Grosse, Tenenbaum, Ghahramani
• Exploiting compositionality to explore a large space of model structures by Grosse et al
• Model Selection in Compositional Spaces; PhD Thesis by Roger Grosse
• The discovery of structural form by Kemp and Tenenbaum
• Structured priors for structure learning by Mansinghka, Kemp, Tenenbaum, and Griffiths
• Information Theory, Inference, and Learning Algorithms (Chapter 28, Model Comparison and Occam’s Razor) by David MacKay

Reference Material on Approximate Inference (Optional)

• Probabilistic Inference using Markov chain Monte Carlo Methods by Neal
• A Tutorial on Particle Filtering and Smoothing: Fifteen years later by Doucet and Johansen
• Hamiltonian Monte Carlo tutorial by Neal
• Exact and approximate inference via systematic stochastic search by Mansinghka et al.
• Factor graphs and the sum-product algorithm by Kschischang, Frey, and Loeliger
• An Architecture for Parallel Topic Models by Smola and Narayanamurthy
• Efficient BackProp by LeCun, Bottou, Orr and Muller
• Black Box Variational Inference by Ranganath, Gerrish and Blei

Selecting Actions (Optional)

• Towards common-sense reasoning via conditional simulation: legacies of Turing in Artificial Intelligenceby Freer, Roy, Tenenbaum
• Rational Metareasoning and Compilation for Optimizing Decisions Under Bounded Resources by Eric Horvitz