EP30.FSF1115 — Bayesian Statistics
Course description
This course is part of the specialized knowledge block in the Applied Mathematics undergraduate program. The course provides foundational and modern knowledge of Bayesian Statistics, encompassing: subjective probability reasoning, the process of building generative models, computational approximation methods (Grid Approximation, MCMC), and Causal Inference using Directed Acyclic Graphs (DAGs). The course follows a “code-first” approach where theory is seamlessly integrated with computational practice (Stan/PyMC) to solve practical problems in Finance and Data Science.
Course requirements
Attendance: Participate at least 80% of total class.
Requirements: Read textbook, do Homework. Do not disturb others.
Technology: Laptop/iPad allowed for practicing/calculating only.
Detailed content
The course content includes:
- Introduction, History & Probability Review
- Beta-Binomial Model and Conjugate Families
- Exponential Families and Posterior Reliability
- Monte Carlo Methods and Posterior Predictive Distributions
- Metropolis-Hastings and MCMC Diagnostics
- Gibbs Sampling and the Multivariate Normal
- The Multivariate Normal Model.
- Bayesian Linear Regression I
- Bayesian Linear Regression II and Hierarchical Models
- Hierarchical Models.
- Hamiltonian Monte Carlo and Priors Missing Data and Course Synthesis
Lecturers
- Bui Quoc Hoan
- Vuong Van Yen
Teaching plan
Week 1
Introduction, History & Probability Review
References
Ch 2
Activities
Course introduction; the Bayesian vs. frequentist debate; historical development from Bayes and Laplace to modern statistics. Notation: sample spaces, events, probability axioms.
Probability review - axioms, conditional probability, independence, law of total probability, Bayes' theorem. Exchangeability and de Finetti's theorem: the theoretical foundation for using priors.
Welcome Lab - introduction to R and RStudio; exploring probability distributions; MLE review exercises.
Assessment
Attendance
Week 2
Beta-Binomial Model and Conjugate Families
References
Ch 3
Activities
The Beta-Binomial conjugate model — deriving the posterior analytically; interpreting hyperparameters as pseudo-data; sequential updating. Prior sensitivity analysis.
The Gamma-Poisson conjugate model. Exponential family likelihoods and their natural conjugate priors. How to identify the sufficient statistic and write the conjugate prior kernel.
Lab: Maximum Likelihood Estimation in R; comparing MLE and posterior mean; simulation study on prior sensitivity.
Assessment
Homework
Week 3
Exponential Families and Posterior Reliability
References
Ch 3
Activities
Exponential families -canonical form, natural parameters, sufficient statistics. Deriving conjugate priors from the exponential family structure. Examples: Binomial, Poisson, Normal, Exponential.
Posterior reliability -definition of posterior coverage (credible intervals). Highest Posterior Density (HPD) regions vs. equal-tail credible intervals. The Laplace approximation to the posterior.
Lab: Exponential families and transformations; computing credible intervals in R; comparing HPD and equal-tail intervals.
Assessment
Homework
Week 4
Monte Carlo Methods and Posterior Predictive Distributions
References
Ch 4
Activities
Introduction to Monte Carlo integration - why sampling is powerful; the posterior predictive distribution; prior predictive distribution. Using samples to answer arbitrary questions about parameters.
Lecture 8 (Wed Feb 05): Monte Carlo error and convergence - the CLT for Monte Carlo estimators; the O(1/√N) convergence rate; practical sample size choice. Posterior predictive checks as model validation.
Lab: Mixture densities in R; sampling from mixture posteriors; visualizing posterior predictives.
Assessment
Homework
Week 5
The Normal Model I
References
Ch 5
Activities
The Normal model with unknown mean and variance - the Normal-Gamma conjugate family; deriving the joint posterior p(θ, σ²|y). Sampling from the joint posterior via the two-step decomposition.
The Normal model II - the semi-conjugate approach; the normal prior on θ and inverse-gamma prior on σ²; Gibbs sampling preview. Posterior inference: credible intervals for mean and variance.
Lab: Normal data analysis in R; posterior sampling; comparing posterior summaries with frequentist counterparts.
Assessment
Group Work
Week 6
Estimators and Decision Theory
References
Ch 5
Activities
Review of estimation theory - bias, variance, mean squared error. Bias-variance decomposition. The James-Stein estimator and shrinkage. Why Bayesian estimators often have lower MSE than MLE.
Bayesian decision theory (brief) - loss functions (squared error, absolute error, 0-1 loss); Bayes estimators as posterior means, medians, and modes. Admissibility of Bayesian estimators.
Lab: Estimators in R; simulation study comparing bias and MSE of different estimators; shrinkage visualization.
Assessment
Homework
Week 7
EXAM I + Metropolis-Hastings
References
Ch 6, 10
Activities
EXAM I: In-class examination covering Weeks 1–6. Bring pencil, pen, and provided formula sheet. Closed-book.
Motivation for MCMC - why grid approximation and conjugacy fail for complex models; the curse of dimensionality. The Metropolis algorithm - proposal distributions, acceptance ratio, detailed balance, ergodicity.
No lab this week.
Assessment
Homework
Week 8
Metropolis-Hastings and MCMC Diagnostics
References
Ch 6, 10
Activities
Finishing Metropolis algorithm - the Metropolis-Hastings generalization; asymmetric proposals; tuning the proposal variance. Mixing and convergence intuition.
MCMC diagnostics - trace plots, autocorrelation plots (ACF), effective sample size (ESS), the Gelman-Rubin R-hat statistic; thinning and burn-in. Practical guidelines for running MCMC in applied settings.
Lab: Implementing Metropolis-Hastings from scratch in R; visualizing chains; computing diagnostics; confidence bands.
Assessment
Homework
Week 9
Gibbs Sampling and the Multivariate Normal
References
Ch. 6, 7
Activities
Gibbs sampling - the full conditional distributions; the two-component and multi-component Gibbs sampler; convergence properties; relationship to Metropolis-Hastings.
Lab: MCMC diagnostics practice; computing ESS and R-hat; autocorrelation plots in R.
Week 10
The Multivariate Normal Model.
References
Ch 7
Activities
The multivariate Normal distribution - density, parameters, key properties; marginals and conditionals of MVN. The semi-conjugate prior for μ: the MVN prior and its posterior update.
Semi-conjugate prior for Σ - the Inverse-Wishart distribution; the Gibbs sampler for the joint posterior of (θ, Σ). The Wishart distribution as a sum-of-squares matrix.
Lab: Introduction to rstanarm; fitting Normal models in Stan; interpreting Stan output and diagnostics.
Assessment
Group Work
Week 11
Bayesian Linear Regression I
References
Ch 9
Activities
Bayesian simple linear regression - the g-prior and other conjugate priors on β; posterior distributions; point estimates vs. posterior distributions. Comparing Bayesian and OLS estimates.
Bayesian multiple linear regression - the Zellner g-prior; posterior for regression coefficients; posterior predictive distribution for new observations; credible intervals for predictions.
Lab: Probit regression in R; posterior sampling for a probit model; interpreting latent variable representation.
Assessment
Homework
Week 12
EXAM II + Bayesian Linear Regression II and Hierarchical Models
References
Ch. 9 §3
Activities
EXAM II: In-class examination covering all material from Weeks 1–11. Bring pencil, pen, and formula sheet.
Bayesian model averaging (BMA) - the problem of model uncertainty; posterior model probabilities; Bayesian model averaging for prediction; comparison to AIC/BIC model selection.
No lab this week
Assessment
Homework
Week 13
Hierarchical Models.
References
Ch 8
Activities
Introduction to hierarchical modeling - the problem with fixed and random effects; the hierarchical Normal model; partial pooling vs. no-pooling vs. complete pooling. The shrinkage estimator.
Bayesian inverse problems - a modern application case study; inferring physical quantities from indirect noisy measurements; connection to regularization (Tikhonov).
Lab: Bayesian inverse problems in R; simulation study on partial pooling; visualizing shrinkage.
Assessment
Homework
Week 14
Hamiltonian Monte Carlo and Priors
References
Ch. 10 (supp.)
Activities
Hamiltonian Monte Carlo (HMC) - the leapfrog integrator; the dual averaging step-size adaptation; the No-U-Turn Sampler (NUTS); why HMC explores posterior geometry more efficiently than random-walk MH.
Prior distributions in practice - weakly informative priors; the role of scale in hierarchical models; Jeffreys' invariance priors; regularizing priors for regression; empirical Bayes.
Lab: Review session / office hours; student Q&A for final exam preparation.
Assessment
Computer Test
Week 15
Missing Data and Course Synthesis
References
Supplementary notes
Activities
Missing data in the Bayesian framework - Rubin's typology (MCAR, MAR, MNAR); Bayesian imputation as posterior inference over missing values; the multivariate Normal model for imputation; comparison to multiple imputation.
Course synthesis and review - the complete Bayesian workflow; connecting Part I and Part II; Bayesian vs. frequentist: a final perspective; open problems and future directions in Bayesian statistics.
Lab: Review materials and practice problems posted online.
Assessment
Attendance
Final Examination
Activities
Format: Capstone Project (Report & Presentation).
Assessment
Final Exam
Learning resources
- A First Course in Bayesian Statistical Methods — Peter D. Hoff, Springer, 2009
- Bayesian Data Analysis (3rd Edition) — Gelman, A., et al., CRC Press, 2013
- Statistical Rethinking: A Bayesian Course with Examples in R and Stan (2nd Edition) — McElreath, R., CRC Press, 2020
- The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation. 2nd ed. Springer. (Advanced; for graduate students seeking theoretical depth.) — Robert, C. P., 2007
- R or Python (Standard libraries: Stan/PyMC)