Stephen Bates Profile
Stephen Bates

@stats_stephen

2,873
Followers
304
Following
6
Media
203
Statuses

Assistant Professor, MIT EECS. Developing rigorous stats & ML methods for data-driven science and reliable AI systems.

Joined April 2018
Don't wanna be here? Send us removal request.
Pinned Tweet
@stats_stephen
Stephen Bates
3 years
📰 Excited to share our new work on risk control in prediction! Multiple testing leads to practical calibration algorithms with PAC guarantees for any statistical error rate. Works with any model + data distribution! #Statistics #MachineLearning
@ml_angelopoulos
Anastasios Nikolas Angelopoulos
3 years
Thrilled to share Learn then Test, a tool to calibrate any model to control risk (eg. IOU, recall in object detection). No assns on model/data. See arXiv + Colab ✍️w/ @stats_stephen , E.J. Candes, M.I. Jordan, @lihua_lei_stat ! 🧵1/n
4
24
89
4
14
53
@stats_stephen
Stephen Bates
9 months
Excited to share that I've joined MIT as an assistant professor in EECS! I'm thrilled to join many thoughtful, inspiring colleagues. Looking ahead, I'm working to develop statistical principles for AI models so that we can use them for science and reliable automated systems.
Tweet media one
52
21
911
@stats_stephen
Stephen Bates
2 years
Want to learn about concentration inequalities and high-dimensional statistics? Roman Vershynin just released 41 lecture videos! They go along with his beautiful book. This is an amazing new resource!
2
102
540
@stats_stephen
Stephen Bates
2 years
Excited to share a new, simple regression adjustment to get causal estimates with longitudinal data! Causal inference with longitudinal data is hard!! Why? A 🧵👇 w/ @edwardhkennedy , @robtibshirani , V Ventura, and L Wasserman 1/n
7
89
398
@stats_stephen
Stephen Bates
6 months
If you're a prospective PhD student interested in statistics & uncertainty for ML 🤖⁉️, AI for science 🧪, or data with strategic behavior ♟️, I have openings in my group! Consider applying to the MIT EECS PhD program and mention me in your application.
6
59
218
@stats_stephen
Stephen Bates
3 years
Need to give your ML model 🤖 reliable uncertainty quantification? Check out our new Gentle Intro to Conformal Prediction tutorial + video. You get valid confidence sets with any model for any (unknown) distribution, no retraining. with @ml_angelopoulos
2
48
193
@stats_stephen
Stephen Bates
7 months
Excited to share that our work on prediction-powered inference just came out in Science!
@ScienceMagazine
Science Magazine
7 months
A new statistical framework dubbed “prediction-powered inference” could enable researchers to draw valid and more data-efficient scientific conclusions using datasets enriched with machine-learning predictions. Learn more in Science:
Tweet media one
3
25
124
1
11
105
@stats_stephen
Stephen Bates
2 years
Deep learning predictions should *always* come with error bars. Conformal prediction is a practical, easy-to-use statistical technique for this. Check out our tutorial 👇 for a simple introduction. Lots of real examples with Jupyter notebooks!! 🦾
@ml_angelopoulos
Anastasios Nikolas Angelopoulos
2 years
📢Huge update to Gentle Introduction to Conformal Prediction📢 Notebooks for EVERY example, easy-2-run WITHOUT model/data download. Open+run in Colab!✅ New repo here: New sections on time-series and risk control!✅ More in 🧵
Tweet media one
9
103
512
2
25
98
@stats_stephen
Stephen Bates
3 years
What is cross-validation estimating? It turns out the answer is *not* "the accuracy of the model from my data" but is instead "the average accuracy over many unseen training sets" (at least for regression). New work with Trevor Hastie and Rob Tibshirani.
@robtibshirani
rob tibshirani
3 years
With postdoc Stephen Bates and Trevor Hastie, I have just completed a new paper "Cross-validation: what does it estimate and how well does it do it?"
18
313
1K
4
11
79
@stats_stephen
Stephen Bates
3 years
🚨The Distribution-free Uncertainty Quantification ICML workshop kicks off tomorrow!🚨 Leading off the morning session will be Rina Barber, Michael Jordan, Vladimir Vovk, Larry Wasserman, and Leying Guan.
Tweet media one
3
19
78
@stats_stephen
Stephen Bates
1 year
If you want to use ML outputs in your regression model but need valid confidence intervals, we have a solution for you. Check out Prediction-Powered Inference! New work now online: Valid with any ML model and any data set -- no assumptions. 🤖🙌
@ml_angelopoulos
Anastasios Nikolas Angelopoulos
1 year
📯Prediction-Powered Inference📯 With the rise of AlphaFold etc., people are using ML predictions to replace costly experimental data. But predictions aren't perfect; can we still use them for rigorous downstream inferences? The answer: yes. A 🧵
Tweet media one
15
92
389
1
6
61
@stats_stephen
Stephen Bates
2 years
Thrilled that parts of our "Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification" made it into the legendary Murphy's book! 🤩 Check out the gentle intro + quick video w/ @ml_angelopoulos
@sirbayes
Kevin Patrick Murphy
2 years
I am delighted to announce that a draft of my latest book, “Probabilistic Machine Learning: Advanced Topics”, is now available online at . It covers #DeepGenerativeModels , #BayesianInference , #Causality , #ReinforcementLearning , #DistributionShift , etc.
Tweet media one
38
1K
5K
1
11
48
@stats_stephen
Stephen Bates
3 years
Conformal inference gives rigorous outlier/out-of-distribution detection. We show how to control FDR with conformal p-values -- even though they are dependent, they satisfy the PRDS property! With E. Candès, @lihua_lei_stat , Y. Romano, and M. Sesia
Tweet media one
0
10
39
@stats_stephen
Stephen Bates
7 months
If you're interested in a postdoc at the intersection of stats/ML and the life sciences, come join us!
@Schmidt_Center
Eric and Wendy Schmidt Center
7 months
📢 Our 2024 postdoctoral fellowship app is now live! Are you a machine learning, stats, or applied math researcher interested in tackling biomedical problems? Come join us @broadinstitute ! 🧬 💻
Tweet media one
2
22
43
1
3
31
@stats_stephen
Stephen Bates
3 years
Our work was mentioned in Andrew Gelman's blog! (And it wasn't totally panned.) I feel like I've made it as a statistician. 😎
1
0
28
@stats_stephen
Stephen Bates
9 months
In this moment, I'd also like to say thank you to the many mentors and teachers I've had along the way. I'm particularly grateful to my PhD advisor, Emmanuel Candès, as well as Mike Jordan, @robtibshirani , and Joe Blitzstein @stat110 . I hope I can pay it forward.
2
0
25
@stats_stephen
Stephen Bates
2 years
Excited to announce our new work showing the robustness of conformal prediction to label noise! If you calibrate on noisy labels, you typically get valid, *conservative* coverage on clean labels at test-time. So if you use conformal you're already safe against noisy labels 🛡️
@BatEinbinder
Bat-Sheva Einbinder
2 years
Check out our new paper about conformal prediction with label noise! Through theoretical examples and practical experiments we study the robustness of conformal prediction to label noise.
0
4
10
0
6
24
@stats_stephen
Stephen Bates
3 years
This is a tutorial for the upcoming @ICML2021 Workshop on Distribution-Free Uncertainty Quantification. Join us on there on 7/24 for a deep dive with leaders in this field.
0
4
23
@stats_stephen
Stephen Bates
3 years
Excited to announce the ICML Workshop on Distribution-Free UQ! Great chance to get up to speed on conformal prediction and related ideas from 🔥leaders in the field. Come for the black-box DL models, stay for the finite-sample statistical guarantees.🤓
@ml_angelopoulos
Anastasios Nikolas Angelopoulos
3 years
Announcing the first-ever Workshop on Distribution-Free Uncertainty Quantification (DF UQ) at @ICML2021 . About UQ without any assumptions on the model or #data distribution. All r welcome to submit talks/papers! It's gonna be AWESOME!🧵1/6 #AI #ML
3
44
110
0
1
21
@stats_stephen
Stephen Bates
2 years
Want to put valid confidence intervals on your ML model using conformal prediction? Check out part 2 of our tutorial where we talk about a big idea: conditional coverage.💡
@ml_angelopoulos
Anastasios Nikolas Angelopoulos
2 years
Just released Part Two of our YouTube tutorial on conformal prediction and distribution-free uncertainty quantification! It focuses on conditional coverage and diagnostics to more carefully evaluate the utility of conformal procedures. w/ @stats_stephen
0
17
46
2
4
20
@stats_stephen
Stephen Bates
2 years
Here, we're standing on the shoulders of giants. Jamie Robins pioneered causal inference in this setting, and our work rests on his celebrated g-formula. Unlike many parametric approaches, however, we avoid the null paradox. (See the 📜). n/n
0
0
17
@stats_stephen
Stephen Bates
2 years
Thanks so much 😊 New chapters and more videos coming soon!
@predict_addict
Valeriy M., PhD, MBA, CQF
2 years
The best tutorial on #conformalprediction by @ml_angelopoulos and @stats_stephen has over 10K views on @YouTube . Amazing - this is the recommended tutorial for anyone to start with #conformalprediction #machinelearning #opensource
1
5
46
1
2
16
@stats_stephen
Stephen Bates
6 months
It was so great working with @tifding on this! If you’re at NeurIPS, stop by today to check it out :)
@tifding
Tiffany Ding
6 months
Interested in uncertainty quantification and how to make conformal prediction sets more practically useful? Come to our poster at #NeurIPS23 ! 📍Poster #1623 🕙 Thursday 10:45-12:45 w/ @ml_angelopoulos , @stats_stephen , Michael I. Jordan & Ryan Tibshirani
Tweet media one
2
14
105
0
1
15
@stats_stephen
Stephen Bates
6 months
@InstMathStat Here are the papers! :)
0
0
14
@stats_stephen
Stephen Bates
2 years
In this work, we show that we can simply orthogonalize covariates to the past history and then perform a regression fit as usual. Simple and valid! 😎😎 6/n
1
0
13
@stats_stephen
Stephen Bates
3 years
Check out our new work on uncertainty quantification that preserves privacy. The method gives finite-sample statistical + differential privacy guarantees with no distributional assumptions.
@ml_angelopoulos
Anastasios Nikolas Angelopoulos
3 years
Today we released “Private Prediction Sets”, a differentially #private way to output rigorous, finite-sample uncertainty quantification for any model and dataset. The method builds on conformal prediction. 🧵1/n
1
10
37
0
1
12
@stats_stephen
Stephen Bates
3 years
Do you feel like you're 💤ing too much since the NeurIPS deadline? Great news: we're accepting submissions for the first-ever #ICML2021 workshop on distribution-free UQ! No reformatting, due June 14. #statstwitter , #ml , #machinelearning
0
3
12
@stats_stephen
Stephen Bates
2 years
Why do we need to watch out for post-treatment variables? Post-treatment variables can be mediators and/or cause collider bias due to phantom variables. This dependence confuses regression estimates and makes them invalid. See the 📜! 4/n
1
0
10
@stats_stephen
Stephen Bates
2 years
Protein design is a hard learning task that intrinsically has distribution shift. It turns out that conformal predication can be modified to handle this type of shift! Working on this data with @seafann was very inspiring 🙌
@clara_fannjiang
Clara Fannjiang
2 years
'conformal prediction under feedback covariate shift for biomolecular design'🧬 is out now @PNASNews : check out the code at !
0
4
36
0
0
9
@stats_stephen
Stephen Bates
2 years
We now have a catch-22 with longitudinal data: including the covariates is invalid because they are post-treatment variables, but omitting them is invalid because they are pre-treatment variables that remove confounding. What to do?? 🤔 5/n
1
0
9
@stats_stephen
Stephen Bates
3 years
Thanks @stephenrra ! Glad you found it helpful. :)
@stephenrra
Stephen Ra
3 years
A well-written and accessible intro to conformal prediction and distribution-free uncertainty quantification by @ml_angelopoulos and @stats_stephen with examples in PyTorch:
Tweet media one
0
25
122
0
0
7
@stats_stephen
Stephen Bates
3 years
New productivity toy. ⌨️⚡️🦾
Tweet media one
1
0
7
@stats_stephen
Stephen Bates
2 years
With longitudinal data, the covariates are *both* pre-treatment variables and post-treatment variables. In regression adjustment, we must include all pre-treatment variables to adjust for confounding but *not* include any post-treatment variables. 3/n
2
0
7
@stats_stephen
Stephen Bates
3 years
100% agree. Dropping in pre-trained models to remove some variance is a good way to leverage external data sets.
@thos_jones
Tommy Jones
3 years
A thought for #StatsTwitter , pre-trained models that are fine tuned for specific data/goals are taking the ML world by storm. They are going to change how we do statistics. (And get ML dropped into all sorts of software products.)
1
2
4
0
0
6
@stats_stephen
Stephen Bates
2 years
@unsorsodicorda @ml_angelopoulos @predict_addict @yaniv_romano @swamiviv1 @phillip_isola Exactly! When upsampling or fixing corruptions, we want error bars on the latent factors
1
0
6
@stats_stephen
Stephen Bates
3 years
After that, we'll have spotlight talks from @adamjfisch , Christopher Jung, Richard Berk, Eyke Hüllermeier, and Peter Hoff 🤓
1
1
5
@stats_stephen
Stephen Bates
2 years
@graduatedescent Concentration Inequalities, by Boucheron, Lugosi, and Massart
0
0
5
@stats_stephen
Stephen Bates
2 years
@KevinKaichuang @ml_angelopoulos @jlistgarten 😀 Clara Fannjiang is also here on Twitter: @seafann !
0
0
5
@stats_stephen
Stephen Bates
2 years
Before getting into the details, the takeaway is that with this approach we can use regression (linear, log-linear, Cox, ...) to get valid causal estimates with only minor modifications. 🤓 Now, what makes this setting hard? 2/n
1
0
5
@stats_stephen
Stephen Bates
3 years
@atraplet @robtibshirani Thanks! They'll be up soon. :)
0
0
4
@stats_stephen
Stephen Bates
2 years
@unsorsodicorda @_onionesque @Aaroth The new Gibbs + Candes paper looks *awesome* 🤩 Our Conformalized Online Learning paper extends previous ideas of Gibbs + Candes to work for online learning models without a holdout calibration set. I'm excited to look at their new stuff -- it may lead to improved algs here too
2
0
4
@stats_stephen
Stephen Bates
3 years
@NikhGarg @sangeetha_a_j Whenever the internet goes out for a few seconds, I always wonder “what if it never comes back?” 🤯
0
0
4
@stats_stephen
Stephen Bates
1 year
@predict_addict @ChristophMolnar We reference ACP in the Gentle Intro to Conformal Prediction! Huge resource❤️
1
1
4
@stats_stephen
Stephen Bates
2 years
@angelamczhou I already use Gill Sans! The causal effect is… unclear... :)
1
0
4
@stats_stephen
Stephen Bates
3 years
@danieltba Yep! It doesn't change the usual CV point estimate of prediction accuracy -- it gives you better standard errors. If you are choosing the hyperparameter with the best estimated prediction accuracy, this won't change the value you select. We're thinking about extensions for this.
0
0
3
@stats_stephen
Stephen Bates
6 months
There are many other great folks at MIT working on statistics and related ideas: . We hope you consider joining us!
1
0
3
@stats_stephen
Stephen Bates
2 years
@autreche @ml_angelopoulos Thanks Manuel!! So wonderful to hear 😁
0
0
3
@stats_stephen
Stephen Bates
3 years
😱😉
@thegautamkamath
Gautam Kamath
3 years
Tweet media one
4
6
42
0
0
3
@stats_stephen
Stephen Bates
3 years
@unsorsodicorda @adamjfisch It's fast! It only takes as much time as making the usual point prediction
1
0
3
@stats_stephen
Stephen Bates
3 years
Spot on for just about every ML subfield
@KLdivergence
Kristian Lum
3 years
Crap, just realized I did not include the word "desiderata" in my latest fair ML paper. Major oversight on my part. Huge violation of community norms.
6
4
106
0
0
3
@stats_stephen
Stephen Bates
2 years
0
1
3
@stats_stephen
Stephen Bates
3 years
@ChelseaParlett @silvascientist If there really were a Bayes, Hastie and Tibshirani paper, I’d be *very* excited :P
1
0
3
@stats_stephen
Stephen Bates
3 years
0
0
3
@stats_stephen
Stephen Bates
3 years
@unsorsodicorda @adamjfisch Oh, I see what you mean. There are more complex versions of CP that do take more time, but for DNN classifiers there are good, fast ways of doing it.
0
0
2
@stats_stephen
Stephen Bates
1 year
@MikeMomentary I’d say calculus, some analysis, linear algebra, and a first course in probability. It’s not easy, but he does a fantastic job presenting the material
0
0
2
@stats_stephen
Stephen Bates
9 months
@Aaroth Thanks Aaron! I'm excited Penn is much closer now. :)
1
0
2
@stats_stephen
Stephen Bates
2 years
@kchonyc @ml_angelopoulos So great to hear that you're enjoying it! 😊
0
0
2
@stats_stephen
Stephen Bates
3 years
Thanks for the great summary + discussion @FeinbergVlad !
@FeinbergVlad
Vlad Feinberg
3 years
Cross validation: does it really estimate generalization error? Most recently, the Sisu reading group went through some recent work by @stats_stephen on just that! Check out my notes and some fun conjecture & discussion in our writeup.
0
3
10
0
0
2
@stats_stephen
Stephen Bates
2 years
@unsorsodicorda Yes! 📈📈
1
0
2
@stats_stephen
Stephen Bates
3 years
@ChristophMolnar @robtibshirani Nadeau and Bengio also introduce a split-the-data-in-half CV scheme.
1
0
2
@stats_stephen
Stephen Bates
3 years
@rkoenker @ml_angelopoulos Thanks! And good point — quantile loss is more descriptive. We’ve changed the term and the updated version will be up soon.
0
0
1
@stats_stephen
Stephen Bates
9 months
@BrownBaya2 Thanks Brown!
1
0
1
@stats_stephen
Stephen Bates
9 months
@DrRituRaman @sherwang Thanks Ritu! Glad to be joining you here :)
0
0
1
@stats_stephen
Stephen Bates
3 years
@TedWestling Definitely more than a year =\
0
0
1
@stats_stephen
Stephen Bates
3 years
@proneat @sam_power_825 @PierreAlquier @_onionesque I’d check out Jelani Nelson’s recent work:
1
0
1
@stats_stephen
Stephen Bates
9 months
@ml_angelopoulos Thanks Anastasios! And likewise :)
0
0
1
@stats_stephen
Stephen Bates
2 years
1
0
1
@stats_stephen
Stephen Bates
9 months
0
0
1
@stats_stephen
Stephen Bates
9 months
@octonion Thanks Christopher!
0
0
1
@stats_stephen
Stephen Bates
7 months
@Aaroth Congrats Aaron!
1
0
1
@stats_stephen
Stephen Bates
9 months
@stephenrra Thanks Stephen! (feels weird to say it that way?)
0
0
1
@stats_stephen
Stephen Bates
9 months
@rizbicki Thanks Rafael!
0
0
1
@stats_stephen
Stephen Bates
9 months
@predict_addict Thanks Valeriy!
0
0
1
@stats_stephen
Stephen Bates
9 months
@eugene_ndiaye Thanks Eugene!
0
0
1
@stats_stephen
Stephen Bates
2 years
@jweisber @detectiveenters Calibrating Bayesian uncertainty models is a foundational topic; see eg Ch 1 of BDA 3. Conformal is a specific way of doing this that is simple with nice frequentist guarantees. Not sure exactly if this would fit in your course, but it does have quite widespread utility imo.
0
0
1
@stats_stephen
Stephen Bates
5 months
@lihua_lei_stat @StanfordGSB Congratulations Lihua!
1
0
1
@stats_stephen
Stephen Bates
9 months
@martinjzhang Thanks Martin!
0
0
1
@stats_stephen
Stephen Bates
8 months
1
0
1
@stats_stephen
Stephen Bates
9 months
@yubai01 Thanks Yu!
0
0
1
@stats_stephen
Stephen Bates
9 months
@melodyyhuang Thanks Melody! Glad that we get to be neighbors for a bit longer.
0
0
1
@stats_stephen
Stephen Bates
9 months
@heyitsmehugo @predict_addict Thanks so much Hugo! That's really nice to hear.
0
0
1