Shibani Santurkar @ShibaniSan Twitter profile

Last Seen Profiles

@LukeSoCal

@CaliDiSanti

@HollySeiboldVA

@phinataa

@assemblea

@GamerEmpireNet

@charlorugby

@ZenoDLCGame

@Butterc93283694

@plati_ro

@MagzLawrence

@JMnenonik

@UCDavisLibrary

@Cryptosensus1

@V_anana

@Sky_SoHIGH

@thethinker012

@SabrielJOU

@federalesdechi

@sconnect

@ccochrane64

@tauruks

@KShiffPR

@fanyayun

@GRMDAILY

@css_essex

@FossFootball

@MayaBerger

@pnavdeep26

@cappucxino

@AnDa

@UrtubeyJM

@ChrissyCostanza

@susygofficial

@badi

@ECOCtes

Shibani Santurkar

@ShibaniSan

6 months

OpenAI is nothing without its people

23

46

849

Shibani Santurkar

@ShibaniSan

2 years

Does language supervision (as in CLIP) help vision models transfer better? You might expect a clear-cut answer: 'captions always help' or 'not at all'. But w/ @yanndubs @rtaori13 @percyliang @tatsu_hashimoto , we find that the picture is nuanced.🧵

Is a Caption Worth a Thousand Images? A Controlled Study for...

The development of CLIP [Radford et al., 2021] has sparked a debate on whether language supervision can result in vision models with more transferable representations than traditional image-only...

arxiv.org

2

42

202

Shibani Santurkar

@ShibaniSan

6 months

❤️

Sam Altman

@sama

6 months

i love the openai team so much

5K

4K

73K

1

73

Shibani Santurkar

@ShibaniSan

6 months

💙💙💙💙💙💙💙

OpenAI

@OpenAI

6 months

We have reached an agreement in principle for Sam Altman to return to OpenAI as CEO with a new initial board of Bret Taylor (Chair), Larry Summers, and Adam D'Angelo. We are collaborating to figure out the details. Thank you so much for your patience through this.

6K

13K

67K

1

2

56

Shibani Santurkar

@ShibaniSan

1 year

Auto data selection is comparable to expert curated data for pretraining LMs! The leverage: n-gram overlap between pretrain and downstream predicts downstream acc well (r=0.89). But it's not the whole story - lots to uncover on the effect of pretrain data on downstream tasks.

Sang Michael Xie

@sangmichaelxie

1 year

Data selection typically involves filtering a large source of raw data towards some desired target distribution, whether it's high-quality/formal text (e.g., Wikipedia + books) for general-domain LMs like GPT-3 or domain-specific data for specialized LMs like Codex.

1

11

0

8

38

Shibani Santurkar

@ShibaniSan

6 months

💛

Ilya Sutskever

@ilyasut

6 months

I deeply regret my participation in the board's actions. I never intended to harm OpenAI. I love everything we've built together and I will do everything I can to reunite the company.

7K

4K

33K

1

0

21

Shibani Santurkar

@ShibaniSan

2 years

Come talk to us at our NeurIPS poster from 8:30-10am PT today (now) at spot A2!

Aleksander Madry

@aleks_madry

2 years

Can we perform surgery on the prediction rules of an already trained classifier? It turns out yes (and with only a single example too!) with @ShibaniSan , @tsiprasd , Mahi Elango, David Bau, and Antonio Torralba Paper: Blog post:

3

29

140

0

3

18

Shibani Santurkar

@ShibaniSan

2 years

So proud!

Aleksander Madry

@aleks_madry

2 years

Congratulations, @tsiprasd ! Extremely well deserved—it was an honor to be a (small) part of your (now, honorable ;) PhD journey.

1

4

49

0

1

14

Shibani Santurkar

@ShibaniSan

3 years

@aleks_madry @zacharylipton It's been a blast! Thank you for being an incredible advisor @aleks_madry

0

14

Shibani Santurkar

@ShibaniSan

6 months

🚢 🚢 🚢

OpenAI

@OpenAI

6 months

ChatGPT with voice is now available to all free users. Download the app on your phone and tap the headphones icon to start a conversation. Sound on 🔊

2K

3K

18K

0

1

13

Shibani Santurkar

@ShibaniSan

2 years

Based on our findings, we design simple interventions to improve CLIP’s ability to leverage web-scraped captions: by filtering them and using GPT-J to perform text data augmentations via paraphrasing.

0

1

11

Shibani Santurkar

@ShibaniSan

2 years

(ii) *What is in the caption matters* Given a data budget, CLIP’s performance depends on whether captions directly discuss parts of the image (left) or are complementary to it (right). In fact, one descriptive COCO caption is worth 5x YFCC ones!

2

0

10

Shibani Santurkar

@ShibaniSan

2 years

We find that: (i) *Scale is crucial* When the dataset used to train CLIP/SimCLR is fairly large, CLIP >> SimCLR. If not, SimCLR >> CLIP. Also, the transition point between these regimes is dataset dependent (vertical lines).

1

0

8

Shibani Santurkar

@ShibaniSan

2 years

(iii) *Caption variability hurts CLIP* Captions often vary in how they describe an object (e.g., “bike”/”cycle”/”bicycle”/…), and the parts of the image they focus on. This makes it harder for CLIP to learn but luckily can be mitigated by sampling multiple captions per image!

2

1

7

Shibani Santurkar

@ShibaniSan

2 years

We perform an apples-to-apples comparison of CLIP with a matched image-only approach (a variant of SimCLR). We train both with the same loss function, architecture, training data, data augmentations, etc., to isolate the effect of language (caption) supervision.

1

0

4

Shibani Santurkar

@ShibaniSan

4 years

@RWerpachowski @EliSennesh @aleks_madry @Epsilon_Lee @tsiprasd An important direction for future work...

0

4

Shibani Santurkar

@ShibaniSan

6 months

@ilyasut 🤍💜

0

4

Shibani Santurkar

@ShibaniSan

4 years

@JaydeepBorkar @chipro @_srishtiyadav @akanksha_atrey Thanks you @JaydeepBorkar ! Would also love to see @charapod @whybansal included :)

1

0

4

Shibani Santurkar

@ShibaniSan

6 years

@optiML @aleks_madry @tsiprasd @andrew_ilyas Thanks! Actually, our results go beyond the DLNs. We are able to analyze the effect of adding BatchNorm to a single fully connected layer assuming that the loss (as a function of the layer's output) has non-zero first and second derivatives.

2

0

2

Shibani Santurkar

@ShibaniSan

4 years

@sh_reya @aleks_madry @tsiprasd Thank you! - Yes, the train and test subpopulations need not be disjoint. We chose to focus on this extreme since it is the most challenging (and perhaps cleanest) setting. Still we agree that there are many interesting variants to study (our codebase can be used for this too).

1

0

2

Shibani Santurkar

@ShibaniSan

4 years

@sh_reya @aleks_madry @tsiprasd - The source accuracy does drop when we fine-tune. But, if we fine-tune on both domains, source accuracy remains essentially unchanged while still reaching almost the same target accuracy.

1

0

2

Shibani Santurkar

@ShibaniSan

4 years

@BenErichson @HanieSedghi @aleks_madry @tsiprasd Interesting! Would be curious to see if this is also the case for our subpopulation shift benchmarks.

1

0

2

Shibani Santurkar

@ShibaniSan

3 years

@aspenkhopkins Thanks Aspen! ♥️

0

2

Shibani Santurkar

@ShibaniSan

3 years