📰 Excited to share our new work on risk control in prediction! Multiple testing leads to practical calibration algorithms with PAC guarantees for any statistical error rate. Works with any model + data distribution!
#Statistics
#MachineLearning
Thrilled to share Learn then Test, a tool to calibrate any model to control risk (eg. IOU, recall in object detection). No assns on model/data.
See arXiv
+ Colab
✍️w/
@stats_stephen
, E.J. Candes, M.I. Jordan,
@lihua_lei_stat
! 🧵1/n
Excited to share that I've joined MIT as an assistant professor in EECS! I'm thrilled to join many thoughtful, inspiring colleagues.
Looking ahead, I'm working to develop statistical principles for AI models so that we can use them for science and reliable automated systems.
Want to learn about concentration inequalities and high-dimensional statistics?
Roman Vershynin just released 41 lecture videos! They go along with his beautiful book. This is an amazing new resource!
Excited to share a new, simple regression adjustment to get causal estimates with longitudinal data!
Causal inference with longitudinal data is hard!! Why? A 🧵👇
w/
@edwardhkennedy
,
@robtibshirani
, V Ventura, and L Wasserman
1/n
If you're a prospective PhD student interested in statistics & uncertainty for ML 🤖⁉️, AI for science 🧪, or data with strategic behavior ♟️, I have openings in my group!
Consider applying to the MIT EECS PhD program and mention me in your application.
Need to give your ML model 🤖 reliable uncertainty quantification? Check out our new Gentle Intro to Conformal Prediction tutorial + video. You get valid confidence sets with any model for any (unknown) distribution, no retraining.
with
@ml_angelopoulos
A new statistical framework dubbed “prediction-powered inference” could enable researchers to draw valid and more data-efficient scientific conclusions using datasets enriched with machine-learning predictions.
Learn more in Science:
Deep learning predictions should *always* come with error bars.
Conformal prediction is a practical, easy-to-use statistical technique for this. Check out our tutorial 👇 for a simple introduction. Lots of real examples with Jupyter notebooks!! 🦾
📢Huge update to Gentle Introduction to Conformal Prediction📢
Notebooks for EVERY example, easy-2-run WITHOUT model/data download. Open+run in Colab!✅
New repo here:
New sections on time-series and risk control!✅
More in 🧵
What is cross-validation estimating? It turns out the answer is *not* "the accuracy of the model from my data" but is instead "the average accuracy over many unseen training sets" (at least for regression). New work with Trevor Hastie and Rob Tibshirani.
🚨The Distribution-free Uncertainty Quantification ICML workshop kicks off tomorrow!🚨 Leading off the morning session will be Rina Barber, Michael Jordan, Vladimir Vovk, Larry Wasserman, and Leying Guan.
If you want to use ML outputs in your regression model but need valid confidence intervals, we have a solution for you.
Check out Prediction-Powered Inference! New work now online:
Valid with any ML model and any data set -- no assumptions. 🤖🙌
📯Prediction-Powered Inference📯
With the rise of AlphaFold etc., people are using ML predictions to replace costly experimental data.
But predictions aren't perfect; can we still use them for rigorous downstream inferences?
The answer: yes. A 🧵
Thrilled that parts of our "Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification" made it into the legendary Murphy's book! 🤩
Check out the gentle intro
+ quick video
w/
@ml_angelopoulos
Conformal inference gives rigorous outlier/out-of-distribution detection. We show how to control FDR with conformal p-values -- even though they are dependent, they satisfy the PRDS property!
With E. Candès,
@lihua_lei_stat
, Y. Romano, and M. Sesia
📢 Our 2024 postdoctoral fellowship app is now live! Are you a machine learning, stats, or applied math researcher interested in tackling biomedical problems? Come join us
@broadinstitute
! 🧬 💻
In this moment, I'd also like to say thank you to the many mentors and teachers I've had along the way. I'm particularly grateful to my PhD advisor, Emmanuel Candès, as well as Mike Jordan,
@robtibshirani
, and Joe Blitzstein
@stat110
. I hope I can pay it forward.
Excited to announce our new work showing the robustness of conformal prediction to label noise!
If you calibrate on noisy labels, you typically get valid, *conservative* coverage on clean labels at test-time. So if you use conformal you're already safe against noisy labels 🛡️
Check out our new paper about conformal prediction with label noise!
Through theoretical examples and practical experiments we study the robustness of conformal prediction to label noise.
This is a tutorial for the upcoming
@ICML2021
Workshop on Distribution-Free Uncertainty Quantification. Join us on there on 7/24 for a deep dive with leaders in this field.
Excited to announce the ICML Workshop on Distribution-Free UQ! Great chance to get up to speed on conformal prediction and related ideas from 🔥leaders in the field. Come for the black-box DL models, stay for the finite-sample statistical guarantees.🤓
Announcing the first-ever Workshop on Distribution-Free Uncertainty Quantification (DF UQ) at
@ICML2021
.
About UQ without any assumptions on the model or
#data
distribution. All r welcome to submit talks/papers!
It's gonna be AWESOME!🧵1/6
#AI
#ML
Want to put valid confidence intervals on your ML model using conformal prediction?
Check out part 2 of our tutorial where we talk about a big idea: conditional coverage.💡
Just released Part Two of our YouTube tutorial on conformal prediction and distribution-free uncertainty quantification!
It focuses on conditional coverage and diagnostics to more carefully evaluate the utility of conformal procedures.
w/
@stats_stephen
Here, we're standing on the shoulders of giants. Jamie Robins pioneered causal inference in this setting, and our work rests on his celebrated g-formula. Unlike many parametric approaches, however, we avoid the null paradox. (See the 📜).
n/n
Interested in uncertainty quantification and how to make conformal prediction sets more practically useful? Come to our poster at
#NeurIPS23
!
📍Poster
#1623
🕙 Thursday 10:45-12:45
w/
@ml_angelopoulos
,
@stats_stephen
, Michael I. Jordan & Ryan Tibshirani
In this work, we show that we can simply orthogonalize covariates to the past history and then perform a regression fit as usual.
Simple and valid! 😎😎
6/n
Check out our new work on uncertainty quantification that preserves privacy. The method gives finite-sample statistical + differential privacy guarantees with no distributional assumptions.
Today we released “Private Prediction Sets”, a differentially
#private
way to output rigorous, finite-sample uncertainty quantification for any model and dataset.
The method builds on conformal prediction. 🧵1/n
Do you feel like you're 💤ing too much since the NeurIPS deadline? Great news: we're accepting submissions for the first-ever
#ICML2021
workshop on distribution-free UQ!
No reformatting, due June 14.
#statstwitter
,
#ml
,
#machinelearning
Why do we need to watch out for post-treatment variables? Post-treatment variables can be mediators and/or cause collider bias due to phantom variables. This dependence confuses regression estimates and makes them invalid. See the 📜!
4/n
Protein design is a hard learning task that intrinsically has distribution shift. It turns out that conformal predication can be modified to handle this type of shift!
Working on this data with
@seafann
was very inspiring 🙌
We now have a catch-22 with longitudinal data: including the covariates is invalid because they are post-treatment variables, but omitting them is invalid because they are pre-treatment variables that remove confounding.
What to do?? 🤔
5/n
A well-written and accessible intro to conformal prediction and distribution-free uncertainty quantification by
@ml_angelopoulos
and
@stats_stephen
with examples in PyTorch:
With longitudinal data, the covariates are *both* pre-treatment variables and post-treatment variables. In regression adjustment, we must include all pre-treatment variables to adjust for confounding but *not* include any post-treatment variables.
3/n
A thought for
#StatsTwitter
, pre-trained models that are fine tuned for specific data/goals are taking the ML world by storm. They are going to change how we do statistics. (And get ML dropped into all sorts of software products.)
Before getting into the details, the takeaway is that with this approach we can use regression (linear, log-linear, Cox, ...) to get valid causal estimates with only minor modifications. 🤓
Now, what makes this setting hard?
2/n
@unsorsodicorda
@_onionesque
@Aaroth
The new Gibbs + Candes paper looks *awesome* 🤩
Our Conformalized Online Learning paper extends previous ideas of Gibbs + Candes to work for online learning models without a holdout calibration set. I'm excited to look at their new stuff -- it may lead to improved algs here too
@danieltba
Yep! It doesn't change the usual CV point estimate of prediction accuracy -- it gives you better standard errors. If you are choosing the hyperparameter with the best estimated prediction accuracy, this won't change the value you select. We're thinking about extensions for this.
@unsorsodicorda
@adamjfisch
Oh, I see what you mean. There are more complex versions of CP that do take more time, but for DNN classifiers there are good, fast ways of doing it.
@MikeMomentary
I’d say calculus, some analysis, linear algebra, and a first course in probability. It’s not easy, but he does a fantastic job presenting the material
Cross validation: does it really estimate generalization error? Most recently, the Sisu reading group went through some recent work by
@stats_stephen
on just that! Check out my notes and some fun conjecture & discussion in our writeup.
@jweisber
@detectiveenters
Calibrating Bayesian uncertainty models is a foundational topic; see eg Ch 1 of BDA 3. Conformal is a specific way of doing this that is simple with nice frequentist guarantees.
Not sure exactly if this would fit in your course, but it does have quite widespread utility imo.