Fereshte Khani Profile
Fereshte Khani

@fereshte_khani

4,814
Followers
863
Following
40
Media
322
Statuses

@OpenAI , ex- @Microsoft , CS Ph.D. @stanfordAILab , working on reliability, and alignment in machine learning.

San Francisco, CA
Joined May 2018
Don't wanna be here? Send us removal request.
Pinned Tweet
@fereshte_khani
Fereshte Khani
1 year
Our new preprint proposes a framework for collaborative NLP development (CoDev) that enables multiple users to align a model with their beliefs. An alternative to a single user or a central institution dictating how models should behave!
Tweet media one
4
12
93
@fereshte_khani
Fereshte Khani
6 months
Not the best time to announce this but I joined @OpenAI . Yesterday was my third day 😅
201
92
6K
@fereshte_khani
Fereshte Khani
2 years
In our #Neurips2022 paper, we introduce MaskTune. A method that forces a model to explore new features by masking previously discovered features and finetuning the model over the masked data.
Tweet media one
10
71
477
@fereshte_khani
Fereshte Khani
2 years
🚨I’m hiring interns for summer 2023! 🚨 If you are interested in reliable, robust, and fair ML, apply and contact me at fkhani @microsoft .com. Please retweet & share!
9
83
364
@fereshte_khani
Fereshte Khani
2 years
🚨I’m hiring interns for summer 2022! 🚨 If you are interested in fair, robust, and reliable ML, apply and contact me at fkhani @microsoft .com. Please retweet & share!
7
116
288
@fereshte_khani
Fereshte Khani
1 year
🚨 Internship Alert 🚨 There are some interesting internship opportunities on LLMs sponsored by our group, please apply!
4
41
199
@fereshte_khani
Fereshte Khani
2 years
I’ve been asked to give a short presentation about biases in the ML cycle and I thought I would share it here too. 1/n
Tweet media one
4
27
148
@fereshte_khani
Fereshte Khani
2 years
It’s been a month since I joined the office of applied research at @Microsoft , and I cannot be happier with my decision. Super grateful to work with these amazing people! @bhecht , @jteevan , @ylongqi , @mengtingwan , @shiladsen , @halfak and Bahar Sarrafzadeh
5
1
115
@fereshte_khani
Fereshte Khani
3 years
1-ML models often rely on spurious features such as backgrounds in images or identity terms in the comments, which undermines fairness and robustness goals. Why do ML models rely on such features even when nonspurious features can determine the target perfectly?
Tweet media one
2
18
99
@fereshte_khani
Fereshte Khani
3 years
Ever wondered why removing spurious features (e.g., background) can lead to drop in accuracy? Check our recent paper for the answer.  Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately #FAccT2021 Joint work with @percyliang
0
10
64
@fereshte_khani
Fereshte Khani
3 years
Machine learning models always use features that are imperfect and noisy. But the same amount of noise on each feature should affect everyone equally, right? Check my recent blog post to see how the same amount of feature noise leads to discrimination[1/6]
3
15
48
@fereshte_khani
Fereshte Khani
3 years
Thank you  @trustworthy_ml for highlighting my research. Looking forward to more discussion. If you are interested in these areas, please get in touch!
@trustworthy_ml
Trustworthy ML Initiative (TrustML)
3 years
1/ In this week’s TrustML-Highlight, we are glad to feature Fereshte Khani @fereshte_khani 🎉🎊🎊🎊. Fereshte is a 5th year PhD student at @Stanford , supervised by @percyliang . She is also on the job market this year!!
Tweet media one
1
10
43
0
2
41
@fereshte_khani
Fereshte Khani
5 months
I'm presenting this work on Thursday 14 Dec 10:45am! Come and say Hi :-) I also recently become interested in Safe/Constrained RL (especially exploration) and reasoning, so happy to chat about these topics!
Tweet media one
1
1
39
@fereshte_khani
Fereshte Khani
10 months
How can you align machine learning models with the perspectives of multiple users? check my new blog post for some answers 😊
1
6
34
@fereshte_khani
Fereshte Khani
3 years
3- In my new blog post, I explain why ML models rely on spurious features even when core features perfectly determine the target; and why accuracy drops after removing spurious features in the absence of the mentioned two reasons.
1
8
33
@fereshte_khani
Fereshte Khani
1 year
1-The second part of my blog posts on lessons for ML researchers from the legal precedents of anti-discrimination is now live!
1
7
31
@fereshte_khani
Fereshte Khani
8 months
Collaborative development of NLP models got accepted in #NeurIPS2023 🎉🎉 See you in New Orleans!
@fereshte_khani
Fereshte Khani
1 year
Our new preprint proposes a framework for collaborative NLP development (CoDev) that enables multiple users to align a model with their beliefs. An alternative to a single user or a central institution dictating how models should behave!
Tweet media one
4
12
93
3
2
28
@fereshte_khani
Fereshte Khani
1 year
1- There are ~90k discrimination charges each year with an average payout of ~300 million. How does the legal process work, and what can machine learning researchers learn from it? check my recent blog post:
Tweet media one
1
6
26
@fereshte_khani
Fereshte Khani
8 months
I misread the #NeurIPS2023 rebuttal discussion deadline and couldn’t submit my response to a reviewer who gave us a weak accept because they thought our paper would have a moderate impact. Now that our paper got accepted I put my response here 🙃 In evaluating the impact of our
0
1
25
@fereshte_khani
Fereshte Khani
6 months
❤️😅
@OpenAI
OpenAI
6 months
We have reached an agreement in principle for Sam Altman to return to OpenAI as CEO with a new initial board of Bret Taylor (Chair), Larry Summers, and Adam D'Angelo. We are collaborating to figure out the details. Thank you so much for your patience through this.
6K
13K
67K
1
1
20
@fereshte_khani
Fereshte Khani
7 months
“Declaration of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.” -thinking fast and slow The parallel between humans and ML models are very interesting!
Tweet media one
2
1
18
@fereshte_khani
Fereshte Khani
1 year
Happy to know that chatGPT knows about taarof 😄
Tweet media one
1
2
15
@fereshte_khani
Fereshte Khani
1 year
Stop by @TMLunLimited in #ICLR2023 at 1:40 for my talk on the collaborative development of NLP models Data disparities result in an inaccurate model, similar to understanding an elephant by solely feeling its body.
Tweet media one
1
2
14
@fereshte_khani
Fereshte Khani
3 years
4-Note that understanding the reasons behind the accuracy drop is crucial for mitigating such drops. Focusing on the wrong mitigation method fails to address the accuracy drop.
Tweet media one
0
0
13
@fereshte_khani
Fereshte Khani
1 year
I will be in Kigali #ICLR2023 🤩 DM me if you want to meet!
0
0
12
@fereshte_khani
Fereshte Khani
5 months
@_jasonwei People've been working on selective classification for a long time (). I remember after publishing my first paper on this topic (), I got many emails that why I haven't cited selective classification 😅😅
0
0
10
@fereshte_khani
Fereshte Khani
3 years
2-One remedy is to remove spurious features to achieve more fair/robust and hopefully more accurate models. But in practice, removing such features reduces the accuracy?!? Maybe the nonspurious features are inadequate, or removing spurious features corrupts the nonspurious ones.
Tweet media one
1
0
12
@fereshte_khani
Fereshte Khani
2 years
In the future, GANs might generate breathtaking images, LMs might generate creative stories, ML might even be able to generate stunning music, but we should never forget that ML is NOTHING without data, so the first step should be figuring out how to give credit to data creators.
@sama
Sam Altman
2 years
I think it should also push towards more equality. In the future, I imagine we will need to figure out how to fairly distribute 1) wealth, 2) access to AGI systems, 3) governance decisions about how AGI systems are used.
33
19
208
0
0
11
@fereshte_khani
Fereshte Khani
1 year
1) What are my goals? 2) Does the current system accurately measure progress toward my goals? 3) Is my strategy effective according to the system's metrics? When I was young I only need to optimize 3 now I need to optimize 1,2,3 together and it is hard!
0
1
10
@fereshte_khani
Fereshte Khani
1 year
respect!
@TIME
TIME
1 year
The Women of Iran are TIME's 2022 Heroes of the Year #TIMEPOY
Tweet media one
2K
28K
97K
0
0
10
@fereshte_khani
Fereshte Khani
2 years
Potential subjects include learning with heterogeneous data (Microsoft has a lot of tenants with different data distributions), making LLMs reliable, and using LLMs to make other models reliable!
1
1
9
@fereshte_khani
Fereshte Khani
2 years
For example: How can we ensure that the culture of big firms doesn't overshadow the culture of small ones? In contrast to common metrics such as engagement, how can we maximize productivity and organizational success? And many more interesting research questions.
0
0
9
@fereshte_khani
Fereshte Khani
2 years
My presentation on biases in the ML cycle for the Microsoft Hackathon!
@fereshte_khani
Fereshte Khani
2 years
I’ve been asked to give a short presentation about biases in the ML cycle and I thought I would share it here too. 1/n
Tweet media one
4
27
148
0
0
8
@fereshte_khani
Fereshte Khani
1 year
Data war? The free usage of individuals' data has now expanded to the free usage of companies' data. With @elonmusk blocking @OpenAI accessing Twitter data, the future of data dividends should be interesting. Great blog post and kudos to ChatGPT for its great answer!
@nickmvincent
Nick Vincent
1 year
After (much) more poking around, I remain very blown away by ChatGPT. But, the idea that people should swap out search engines for language models could seriously erode the sustainability of these systems. Post here: It seems ChatGPT agrees!
Tweet media one
2
5
20
1
0
8
@fereshte_khani
Fereshte Khani
8 months
My taste in writing is much higher than my ability to write 🤦‍♀️ #WritingStruggles
0
0
7
@fereshte_khani
Fereshte Khani
1 year
"Some people say give the customers what they want, but that's not my approach. Our job is to figure out what they're going to want before they do." -Jobs
0
0
7
@fereshte_khani
Fereshte Khani
3 years
“Simple justice requires that public funds, to which all taxpayers of all races [colors, and national origins] contribute, not be spent in any fashion which encourages, entrenches, subsidizes or results in racial [color or national origin] discrimination.” John F. Kennedy 1963
0
0
7
@fereshte_khani
Fereshte Khani
2 years
When Marco first shared his template with me, I thought I was wasting my time filling it out, but now I use it for every single project that I do. Great blog posts, highly recommended!
@marcotcr
Marco Tulio Ribeiro
2 years
I never tweet, but here is a blog post I wrote for an intern, may be useful for others too... Part 1: Part 2:
9
128
575
0
0
7
@fereshte_khani
Fereshte Khani
2 years
The majority of Microsoft customers are other firms so for any ML model we care about each and every user. There are plenty of opportunities for fair, robust and reliable machine learning.
1
0
7
@fereshte_khani
Fereshte Khani
1 year
These internships are part of an amazing program which provides co-mentorship with both a product team and a research team, offering valuable experience and skill-building opportunities in both areas.
0
0
6
@fereshte_khani
Fereshte Khani
2 years
I'm at #NAACL2022 this week and #ICML2022 next week 🤩 Ping me if you want to talk!
0
0
6
@fereshte_khani
Fereshte Khani
1 year
We are seeking candidates with expertise in language model augmentation (e.g. retrieval), prompt engineering, hallucination mitigation, techniques for improving inference efficiency, LLM interaction paradigms, and related applied research subjects.
1
0
6
@fereshte_khani
Fereshte Khani
1 year
Our second insight (inspired by Taylor expansion and interpretability works such as LIME) is that we can train a local model specific to the user concept to guide our exploration.
Tweet media one
1
0
4
@fereshte_khani
Fereshte Khani
1 year
Do we have an embedding-to-sentence model?
1
0
6
@fereshte_khani
Fereshte Khani
1 year
If you are still searching for an internship and have experience working with LLMs, especially on alignment or interacting with users to elicit their preferences, this is an amazing opportunity for you!
@glenweyl
⿻(((E. Glen Weyl/衛谷倫))) 🇺🇸/🇩🇪/🇹🇼 🖖
1 year
If you are 1) in you're in/around/interested in the Plurality community, 2) have a technical background in GFMs/LLMs especially related to alignment, 3) are available for a summer internship and 4) are in a degree program (ideally PhD), please reach out to glenweyl @microsoft .com
1
14
28
0
0
5
@fereshte_khani
Fereshte Khani
2 years
Advancement in ML results in models and algorithms that can find “simple functions” that fit training data and generalize very well (likely due to the simplicity of such functions).
1
0
4
@fereshte_khani
Fereshte Khani
3 years
In addition to overcoming the associated effects of being in a group with lower average skills, blue people also need to pass a higher bar to get hired. The more feature noise, the greater the discrimination! [5/6]
Tweet media one
1
0
3
@fereshte_khani
Fereshte Khani
2 years
I'm working on biases in data and how they interact with biases in the training. I'm also interested in testing biases and the feedback loops. Please reach out if you want to chat about these topics. n/n
0
0
5
@fereshte_khani
Fereshte Khani
1 year
A great tutorial on alignment!
@JacobSteinhardt
Jacob Steinhardt
1 year
My tutorial slides on Aligning ML Systems are now online, in HTML format, with clickable references! [NB some minor formatting errors were introduced when converting to HTML]
1
7
47
0
0
4
@fereshte_khani
Fereshte Khani
1 year
Luckily we have good proxies for all concepts (local models) so whenever we make a change we query proxies to see if we create new interference!
Tweet media one
1
1
4
@fereshte_khani
Fereshte Khani
2 years
In our work, in order to avoid shortcut learning we force a model to explore more by blocking the previously discovered patterns. MaskTune results in very good worst-group accuracy without having information about minority groups in training or validation time
Tweet media one
1
1
4
@fereshte_khani
Fereshte Khani
1 year
Joint work with @marcotcr ! Much appreciation to @scottlundberg for all his help and discussion throughout this work. Also thanks to @bhecht and @ZexueHe for their invaluable early feedback. Please reach out if you have any questions/feedback!
0
0
4
@fereshte_khani
Fereshte Khani
1 year
12/12 - Joint work with @percyliang ! and thank you @AlexTamkin @jmschreiber91 @NeelGuha @PeterHndrsn @megha_byte and @mzhangio for their great feedback!
0
0
4
@fereshte_khani
Fereshte Khani
2 years
Tweet media one
@alexshams_
Alex Shams
2 years
In recent days, protests have broken out across Iran condemning the death of a young woman, Mahsa Amini, at the hands of morality police. United in anger at police brutality and restrictive moral codes, Iranians are demanding freedom and an end to government repression:
Tweet media one
Tweet media two
33
1K
4K
0
0
4
@fereshte_khani
Fereshte Khani
2 years
💡Fiction idea💡: A world where people of similar age exchange their bodies with each other every 4 years. Do people help each other to take care of their bodies, or do people kill the ones who do not care about their bodies?
0
0
4
@fereshte_khani
Fereshte Khani
2 years
This study opens interesting future directions: 1) How many “disjoint” explanations can explain data? 2) Besides blocking pixels how else we can define and achieve “disjoint” explanations?
0
0
4
@fereshte_khani
Fereshte Khani
3 years
In the crop type prediction from satellite images, we got access to new auxiliary data (climate information). How should we best leverage this auxiliary data for better in- and out-of-distribution performance?
@sangmichaelxie
Sang Michael Xie
3 years
🍔🍟"In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness" Real-world tasks (crop yield prediction from satellites) are often label-scarce. Only some countries have labels - how do we generalize globally?
Tweet media one
1
37
165
0
0
3
@fereshte_khani
Fereshte Khani
1 year
A great tutorial on Parameter-efficient Fine-tuning!
@seb_ruder
Sebastian Ruder
1 year
Join our tutorial on Modular and Parameter-efficient Fine-tuning this morning at #EMNLP 2022 w/ @PfeiffJo @licwu . 👨‍🏫 Slides: 📍Location: Capital Suite 7
Tweet media one
Tweet media two
1
15
107
0
0
3
@fereshte_khani
Fereshte Khani
2 years
Slides for this presentation: I mostly focus on data collection, training, and feedback loop.
@fereshte_khani
Fereshte Khani
2 years
I’ve been asked to give a short presentation about biases in the ML cycle and I thought I would share it here too. 1/n
Tweet media one
4
27
148
0
1
3
@fereshte_khani
Fereshte Khani
1 year
Given these two insights, we prompt LLM such that it generates sentences that proxy and model disagree. Now whenever a user resolves a disagreement we either fix the local model (thus improving user proxy concept) or fix some bugs in the global model.
Tweet media one
1
1
3
@fereshte_khani
Fereshte Khani
2 years
Distribution bias: Historical discrimination creates a gap between the distribution of different groups (e.g., There is a gap between the number of men and women in the Senate). Sampling bias: data can only be sampled from some groups, or data can misrepresent some groups. 6/n
1
0
3
@fereshte_khani
Fereshte Khani
2 years
However, this “simplicity” can be harmful to some groups or force the model to learn shortcuts when there is not enough data to be a paradox for these shortcuts. E.g., see and
2
0
2
@fereshte_khani
Fereshte Khani
2 years
Label bias: for example, historically well-qualified women did not get hired. Feature bias: for example, the number of previous arrests is biased against minorities as it is more probable for them to get arrested with the same amount of illegal drugs.5/n
1
0
3
@fereshte_khani
Fereshte Khani
2 years
Algorithm/Training: Majority bias: ML models usually work better for the majority (e.g., generalization bounds). Simplicity bias: ML algorithms tend to find the simplest model which can cause discrimination for a population with a more complicated function. 7/n
1
0
3
@fereshte_khani
Fereshte Khani
1 year
lol
@fhuszar
Ferenc Huszár
1 year
Tweet media one
Tweet media two
1
8
59
0
1
2
@fereshte_khani
Fereshte Khani
2 years
Business need: Developers' biases can lead to ML models that only address a particular group's need and ignore other groups, or similarly only cause harm to a specific protected group. 2/n
1
0
3
@fereshte_khani
Fereshte Khani
2 years
Data: ML models learn patterns from previously collected data. The data usually reflect the longstanding discrimination against protected groups. This discrimination can be manifested as: 4/n
1
0
3
@fereshte_khani
Fereshte Khani
1 year
2- In the last year of my Ph.D., tired of n different definitions of fairness and m >> n papers on how to optimize them, I decided to read the legal procedure. This blog post is what I wish I had when I began thinking about this issue.
1
1
3
@fereshte_khani
Fereshte Khani
1 year
CoDev aids users in clarifying their concepts (an area humans often struggle with) and assists ML models to handle conflicts between concepts (an area ML often struggle with due to its inability to accommodate local updates).
Tweet media one
1
0
2
@fereshte_khani
Fereshte Khani
1 year
I show how to enable many experts/users to engage with the model and align it with their beliefs while not disrupting one another.
Tweet media one
0
0
2
@fereshte_khani
Fereshte Khani
1 year
But the concept is so big! It might take ages for our random walk to discover areas that aren't aligned with the user (i.e., finding bugs)!
Tweet media one
1
0
2
@fereshte_khani
Fereshte Khani
1 year
There are times when I watch a movie or read a book and wish I had done so when I was a child, like when Dorothy tapped her heels in The Wizard of Oz or when I read "All right then, I'll go to hell" in Huck Finn. A Beautiful Day in the Neighborhood was another of those moments.
Tweet media one
1
0
2
@fereshte_khani
Fereshte Khani
1 year
6- I expand each step and explain the ML efforts so far and what is missing.
Tweet media one
1
0
2
@fereshte_khani
Fereshte Khani
2 years
“You think the writing is communicating your ideas to your readers, it is not! Professional writing is changing their ideas. Nobody cares what ideas you have.”
1
0
2
@fereshte_khani
Fereshte Khani
2 years
Inductive/implicit bias: It is not clear how the unknowns about ML models affect different groups. Misspecification: Not having the true function in the family can affect groups differently. Bias amplification: ML can amplify the biases that exist in the data. 8/n
1
0
2
@fereshte_khani
Fereshte Khani
1 year
4- Despite our desire to have an easy definition for discrimination, proving discrimination is a lengthy process that involves both parties presenting evidence and a judge's decision. This process can provide guidance on how to handle fairness in machine learning.
1
0
2
@fereshte_khani
Fereshte Khani
1 year
It's tricky for users to operationalize their concepts! For instance in the "religion should be neutral" concept, the user might check some sentences but miss others, leading to the assumption that the model is aligned with the concept when in fact it isn't.
Tweet media one
1
0
2
@fereshte_khani
Fereshte Khani
3 years
The optimal decision for the government (and what the ML models would do) is to choose a higher threshold for hiring blue people compared to red people. In other words, A blue person has a lower chance of getting hired than a red person with the same skill level. [4/6]
Tweet media one
1
0
1
@fereshte_khani
Fereshte Khani
2 years
Then he says: And now deer reader, you can do what I have done and it will set you free. But this is not a self-help book!” Stolen focus (), a great book by @johannhari101
0
0
2
@fereshte_khani
Fereshte Khani
2 years
Problem formulation: The difference between the true need (e.g., content relevancy) and its proxy (e.g., clicked content) might adversely affect some groups. Furthermore, the problem formulation might make it easy for the model to be used unintendedly to harm some groups. 3/n
1
0
2
@fereshte_khani
Fereshte Khani
8 months
@savvyRL I would love to! I’ll ping u after I finish the camera-ready 😊
0
0
0
@fereshte_khani
Fereshte Khani
1 year
Our first insight is that we can use LLM to explore a concept by repeatedly using a set of 3-5 sentences from that concept as a prompt and asking for similar examples to be generated. This technique enables us to perform a random walk within the user's concept.
Tweet media one
1
0
2
@fereshte_khani
Fereshte Khani
6 months
"With no effort, he had learned English, French, and Latin. I suspect, however, that he was not very capable of thought. To think is to forget differences, generalize, make abstractions. In the teeming world of Funes, there were only details, almost immediate in their presence."
0
0
2
@fereshte_khani
Fereshte Khani
2 years
A union for training data 😁
0
0
2
@fereshte_khani
Fereshte Khani
1 year
Interference is another big hurdle in ML! It's impossible to alter one part without affecting the rest!
Tweet media one
1
0
2
@fereshte_khani
Fereshte Khani
2 years
“The point systems often don’t measure what we want to measure. They artificially simplify or distort our values. Our goals are messy, complex and multifaceted, but then they are collapsed down to scoring systems.”
1
0
2
@fereshte_khani
Fereshte Khani
2 years
Implicit change: Model predictions change the incentives for individuals (e.g., if the probability of arrest is high without committing the crime for one group, members of that group become incentivized to commit the crime regardless). 14/n
1
0
2
@fereshte_khani
Fereshte Khani
2 years
Explicit changes: Models can also alter human behavior more explicitly (e.g., a college-admitted individual goes through training which increases her skill level, or a recommendation system can alter a person’s food choices by recommending many types of junk food). 15/n
1
0
2
@fereshte_khani
Fereshte Khani
1 year
7- One of the interesting pieces of evidence for proving disparate treatment is direct evidence of motive which is any statement/act by the employer that indicates a bias against members of a protected group.
1
0
1
@fereshte_khani
Fereshte Khani
2 years
“A writing teacher should not ask a student: why do you think that? He should ask: why should I think that?"
1
0
1
@fereshte_khani
Fereshte Khani
1 year
"I think that it's much more dramatic that two men could be working out their feelings of anger -- much more dramatic than showing something of gunfire"
1
0
1
@fereshte_khani
Fereshte Khani
2 years
“If this was a self-help book, I would be able to serve up a delightfully simple conclusion to this story. Those books have a very satisfying structure. The author identifies a problem—usually one he’s had himself– and he talks you through how he personally solved it.
1
0
1
@fereshte_khani
Fereshte Khani
3 years
The government wants to hire anyone with a skill level above 0. To do so, the government takes an exam from everyone, which is a noisy indicator of people's skill level (the noise is the same for everyone). [3/6]
Tweet media one
1
0
0
@fereshte_khani
Fereshte Khani
2 years
Models change over time as well to fit the population (e.g., A/B testing for model selection). Although any model that incorporates user feedback changes over time, ML makes the process faster by training the models rapidly using the newly collected data. 16/n
1
0
1
@fereshte_khani
Fereshte Khani
3 years
@2plus2make5 Interesting that you all believed her narrative. I thought she is experiencing some sort of AI-Schizophrenia, and none of these have happened in the real world.
1
0
1