Simran Arora @simran_s_arora Twitter profile

Pinned Tweet

Simran Arora

2 months

Excited to release Based, an architecture that combines two✌️ simple, familiar, attention-like primitives – short (size-64) sliding window attention and softmax-approximating linear attention – to enable high quality and efficient inference! 💨 🚀 joint w/ @EyubogluSabri ,…

13

89

423

Last Seen Profiles

@AndrewBoff

@HospitalCivil

@aiftv1

@disneymusicjp

@queen6cilla

@telecomparis

@Sammy67_the_2st

@TabiNekoShiro

@Emma_Land_Live

@celestia_devs

@discordfora

@agusbo_

@himemoriluna

@lily_sheep01

@asupanmuu18

@NurseriesBig

@AlysonHollywood

@GodziPat

@miss_edibles

@meganrabuda

@comvokill

@BroodSports

@FCFormica

@4mb_traffic

@OhMySenpaiShop

@miaresu_VR

@teletubisolo

@lettice_Mitsuki

@SFCsupervisor

@Mochalattefox

@saint1jcee

@michi426ANS

@sudharam

@haziqaiman97_

@i_dreampictures

@RMU_ID

Simran Arora

@simran_s_arora

1 year

LMs can be expensive for document processing. E.g., inference over the 55M Wiki pages costs >$100K (>$0.002/1k toks)💰 We propose a strategy that reduces inference cost by 110x and can even improve quality vs. running inference over each doc directly! 💻

GitHub - HazyResearch/evaporate: This repo contains data and code for the paper "Language Models...

This repo contains data and code for the paper "Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes" - HazyResearch/evaporate

github.com

8

135

784

Simran Arora

@simran_s_arora

2 years

Ran out of OpenAI credits?💰 We present a prompting strategy that enables open-source and off-the-shelf GPT-J-6B to outperform *few shot* GPT3-175B on 15 popular language benchmarks! 🚀 Paper and code: 📜 💻

GitHub - HazyResearch/ama_prompting: Ask Me Anything language model prompting

Ask Me Anything language model prompting. Contribute to HazyResearch/ama_prompting development by creating an account on GitHub.

github.com

8

112

662

Simran Arora

@simran_s_arora

5 months

KV-cache got you down? Sharing Based✌️, a simple architecture built from PyTorch 101 building blocks (convolutions, linear attention). It gives exciting quality vs the modern architectures & its hidden state size is fixed, enabling 4.5x >throughput vs attn

8

59

250

Simran Arora

@simran_s_arora

4 months

Book chapters, legal cases, code repositories, etc. can contain tens of thousands of tokens! 📚 We’re excited to release long-context retrieval models at 2k, 8k, and 32k sequence lengths to help explore some of these settings! W/ @JonSaadFalcon @realDanFu

togethercomputer/m2-bert-80M-32k-retrieval · Hugging Face

huggingface.co

4

31

139

Simran Arora

@simran_s_arora

2 years

Reasoning over both public and *private* data is necessary for personalized ML systems. But how can we use personal context without exposing it to the world? 🔑🔒 We explore this question in new work on personalized and private retrieval systems!!

AI at Meta

@AIatMeta

2 years

Building systems that can securely reason over private data to answer questions or reason about the world is challenging & important. Meta is introducing a new data set, ConcurrentQA, and a new privacy methodology to encourage research in these areas.

17

37

167

1

36

104

Simran Arora

@simran_s_arora

2 years

A key promise of machine learning is the ability to assist users with personal tasks over privacy-sensitive data🕵️‍♀️ Can foundation models enable strong privacy guarantees in this setting? Excited to share recent work w @HazyResearch exploring the question!

3

22

68

Simran Arora

@simran_s_arora

2 years

We've @hazyresearch been thinking a lot about foundation models and if we're entering the "data centric" era of FMs! Here are some of our thoughts around what we can learn from the past data-centric revolutions and what is really exciting and new with FMs:

Foundation Models are Entering their Data-Centric Era

hazyresearch.stanford.edu

2

14

62

Simran Arora

@simran_s_arora

5 months

✌️

Zoology (Blogpost 0): Overview

hazyresearch.stanford.edu

Hailey Schoelkopf

@haileysch__

5 months

work in ML, they said. it’ll be fun, they said. Now I’m reading about the Based architecture and its HellaSwag score

31

50

1K

2

7

61

Simran Arora

@simran_s_arora

9 months

Excited to share our recent work towards sub-quadratic architectures that improve efficiency and may be more accessible in low-resource environments, while preserving Transformer-level quality!!! Releasing M2-BERT today 💎

Dan Fu

@realDanFu

9 months

You've heard of models that are sub-quadratic in sequence length, but what if they were sub-quadratic in model *dimension* too? Announcing a preview of Monarch Mixer - a fully sub-quadratic & hardware-efficient architecture that matches BERT in quality! w @simran_s_arora 1/

5

41

158

0

9

39

Simran Arora

@simran_s_arora

10 months

What's the right way to search *multiple* sources (e.g., Amazon and Walmart product descriptions) at once? Our new work—led by @soumyachat — provides resources and evals for Multi-Distribution IR! To be presented at SIGIR REML'23, with @lateinteraction !

Resources and Evaluations for Multi-Distribution Dense Information...

We introduce and define the novel problem of multi-distribution information retrieval (IR) where given a query, systems need to retrieve passages from within multiple collections, each drawn from...

arxiv.org

1

8

36

Simran Arora

@simran_s_arora

4 years

Excited to share our work to better understand the value of contextual embeddings! “Contextual Embeddings: When are they worth it?” with @avnermay , @JianZhangCS , and @hazyresearch to appear at #acl2020nlp next week. Paper: Video:

Contextual Embeddings: When are they worth it? (ACL 2020)

Contextual embeddings have revolutionized NLP, but are highly computationally expensive. In this work we focus on the question of when contextual embeddings ...

www.youtube.com

3

9

33

Simran Arora

@simran_s_arora

2 months

This release, extending our preview of Based in December, includes: - Arxiv: - Checkpoints from the paper are on Hugging Face: - Blogposts: , - Code:

GitHub - HazyResearch/based: Code for exploring Based models from "Simple linear attention language...

Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff" - HazyResearch/based

github.com

2

6

33

Simran Arora

@simran_s_arora

2 years

Very excited to share Manifest!!! Makes it so much easier to run inference with these LLMs, led by the amazing @laurel_orr1 🚀

Laurel Orr

@laurel_orr1

2 years

Tired of battling with the wild west of large language model prompting frameworks and APIs?! We’re excited to introduce Manifest, our python framework that makes prompt programming simple, interactive, and reproducible. 💻:

7

57

372

0

2

31

Simran Arora

@simran_s_arora

1 year

We’re super excited about these directions & would love feedback! This is work with amazing collaborators: @bclyang @EyubogluSabri @Avanika15 @AndrewHojel @ImmanuelTrummer @HazyResearch . Thanks @StanfordCRFM @StanfordHAI @StanfordAILab for the support! 📜

Language Models Enable Simple Systems for Generating Structured...

A long standing goal of the data management community is to develop general, automated systems that ingest semi-structured documents and output queryable tables without human effort or domain...

arxiv.org

4

32

Simran Arora

@simran_s_arora

3 years

To what extent can we match proposed architectural modifications for incorporating knowledge, using only simple changes to the data? How can we understand why data modifications will be effective? @Wu_Sen @HazyResearch and I explore these questions in:

1

11

29

Simran Arora

@simran_s_arora

1 year

To address the quality issue, we propose an extended implementation called Evaporate-Code+, which generates many candidate functions to extract each attribute of interest and ensembles them using weak supervision.

1

3

24

Simran Arora

@simran_s_arora

6 months

Excited to share our recent work, Monarch Mixer, towards architectures that are sub-quadratic in both sequence length and model dimension! Blog: Code: Paper:

Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture

Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts and better performance. However, existing architectures such as...

arxiv.org

Dan Fu

@realDanFu

6 months

Excited about models that are sub-quadratic in sequence length and model dimension? Our Monarch Mixer paper is now on arXiv -- and super excited to present it as an oral at #NeurIPS2023 ! Let's dive in to what's new with the paper and the new goodies from this release: Monarch…

4

60

294

1

5

20

Simran Arora

@simran_s_arora

2 years

Prompt-design is a brittle and time-consuming process – finding the “perfect prompt” is painstaking 😤 Our strategy, Ask Me Anything (AMA), instead aggregates the predictions of multiple effectively, yet imperfect, prompts.

1

17

Simran Arora

@simran_s_arora

1 year

In contrast to processing every semi-structured document with the LLM (Evaporate-direct), we explore using the LLM to generate functions that can then be reused to extract from every document (Evaporate-Code)! We identify fundamental tradeoff space between these two strategies.

1

2

18

Simran Arora

@simran_s_arora

1 year

We ask: when running the same prompt over many documents (e.g. “Extract the device classification” attribute from FDA reports for medical devices), what redundancies across documents can we exploit to improve efficiency? Our prototype system, Evaporate, demonstrates one approach.

1

18

Simran Arora

@simran_s_arora

2 years

Popular frameworks for personal ML e.g. federated learning display a tension between privacy and quality. We introduce a simple new framework, FOCUS, based on shipping foundation models to private silos, guaranteeing **perfect secrecy** When and where does this work, if at all?

1

7

17

Simran Arora

@simran_s_arora

1 year

Structuring heterogeneous data lakes is a long-standing problem in the data management community. We draw inspiration both from classical works on the topic from folks like @MikeCafarella , @etzioni , @AlonHalevy and recent works starting to apply LLMs to the problem!

1

15

Simran Arora

@simran_s_arora

5 months

Despite the simplicity (PyTorch 101 modules, no special initialization) we see exciting results vs. Llama-2 style “Transformers++” and modern efficient archs. (Mamba, Hyena, RWKV, Multi-Head Hyena) in overall and AR perplexity! Checkout our report here:

3

15

Simran Arora

@simran_s_arora

1 year

Tradeoffs: The generated functions (think snippets of python regex, beautiful soup, etc.) are FAST, but have variable quality (51% of our generated functions yield extractions with <50 text f1 in quality). Unfortunately, Evaporate-Code performs 24.9% worse than Evaporate-Direct.

1

2

14

Simran Arora

@simran_s_arora

1 year

We evaluate Evaporate on 16 sets of documents across a range of formats, topics, and attribute types. Evaporate-Code+ yields SoTA quality and a 110x average reduction in the number of tokens the LLM needs to run inference over vs. Evaporate-Direct, at 10k documents per setting.

1

13

Simran Arora

@simran_s_arora

3 years

A blog post about our recent work on named entity disambiguation with insights for succeeding on rare entities!

Stanford AI Lab

@StanfordAILab

3 years

"Who wrote The Otherside by Apple?". "What products does Apple sell?". Would a machine know the difference between the wildly different "Apple"s? Check out our blog post about Bootleg, our self-supervised system for better rare entity identification!

0

4

9

0

2

12

Simran Arora

@simran_s_arora

2 months

Attention’s KV cache consumes a lot of memory, making it challenging to efficiently process thousands-to-millions of documents. So, we wanted to try plugging in a recurrent LLM that uses a small, fixed amount of memory (independent of sequence length) during inference.

1

2

11

Simran Arora

@simran_s_arora

1 year

While weak supervision is traditionally applied to human-generated functions, Evaporate operates with machine-generated functions, which we show do not satisfy certain assumptions in the traditional setup. Evaporate-Code+ addresses these challenges to unlock quality improvements.

1

11

Simran Arora

@simran_s_arora

2 years

For instance, to determine if the statement “John went to the park” is valid given a provided context, the LM first generates several questions such as: “Who went to the park?”, “Did John go to the park?”, “Where did John go?”, ...

1

11

Simran Arora

@simran_s_arora

2 years

We evaluate AMA on 14 LMs spanning 4 model families (OPT, BLOOM, EleutherAI, T0) and a range of sizes (125M - 175B parameters) - finding it broadly improves prompting performance! AMA gives an avg. 10.2% absolute (21.4% relative) lift across LMs over their few(3)-shot baselines!

1

11

Simran Arora

@simran_s_arora

5 months

This is the first preview of Based✌️, more to come! You can follow along with our language modeling results in this WandB report (). You can play with Based✌️& other mixers using our synthetic associative recall testbed here:

1

0

10

Simran Arora

@simran_s_arora

5 months

The modules differ in their abilities to perform associative recall (AR): e.g., recalling the next token should be 'Joes' given a prefix 'She went to Trader Joes to buy … later, at Trader ?' AR is critical for in-context learning. @EyubogluSabri 's tweet:

Sabri Eyuboglu

@EyubogluSabri

5 months

Curious whether sub-quadratic LMs like RWKV and Hyena will replace Transformers? We find that Transformers are still much better at associative recall (AR): a simple task known to be essential for in-context learning.

4

38

143

1

0

10

Simran Arora

@simran_s_arora

2 months

For sliding window 🪟 , we use short windows of width 64 since this keeps the GPU tensor core occupancy high on modern GPUs. Beyond size 64, latency grows non-linearly with window size.

1

2

10

Simran Arora

@simran_s_arora

2 months

It’s been so much fun to work with an amazing team on this line of work: @EyubogluSabri , @mzhangio , Aman Timalsina, @SilasAlberti , Dylan Zinsley, @james_y_zou , Atri Rudra, and our amazing advisor @HazyResearch !!! There's been tons of exciting work in this space, and we're…

1

0

10

Simran Arora

@simran_s_arora

4 years

We show that the performance boost from using contextual embeddings tends to decrease in data-rich regimes, and study the types of language for which context is particularly helpful. Drawing inspiration from SGNNs @ravisujith , GLUE diagnostic task @sleepinyourhat , and others!

1

10

Simran Arora

@simran_s_arora

5 months

This is joint work the amazing @mzhangio and @EyubogluSabri , and our amazing advisor @HazyResearch 🚀 Thank you so much to @togethercompute @StanfordAILab @StanfordCRFM @StanfordHAI for their support! And thanks for reading - please share your feedback 🙏

1

0

9

Simran Arora

@simran_s_arora

2 months

But perplexity isn't everything, we find Mamba still underperforms Transformers on real-world recall-intensive tasks (e.g. information extraction) by 47% on average.

1

3

9

Simran Arora

@simran_s_arora

2 years

Next, the LM answers the questions it generated, producing several noisy votes for the input's true label. AMA finally aggregates these votes using weak supervision to produce the final prediction!

1

9

Simran Arora

@simran_s_arora

2 years

AMA first transforms task examples to a series of question-answer pairs. To avoid manual effort, we show how to recursively use the language model (LM) itself to perform these transformations!

1

8

Simran Arora

@simran_s_arora

2 months

The available options at the time – gated convolutions like Hyena, RWKV-v4, BIGS, and H3 – struggled to perform recall as well as attention. There’s been exciting progress on new candidates like Mamba 🐍 , RWKV-v6, GLA, and Hawk/Griffin that are better able to decide what…

1

3

9

Simran Arora

@simran_s_arora

5 months

We motivate Based✌️ with the view that convolutions & attentions are good at modeling different kinds of sequences. Instead of introducing complexities to overcome their individual weaknesses, we combine familiar versions of each: short 1D convolutions! “spiky” linear attentions!

1

0

8

Simran Arora

@simran_s_arora

5 months

And checkout @HazyResearch 's @NeurIPSConf 2023 keynote *tomorrow 12/14 at 8:30 am CST* for more on efficient architectures and other exciting work in Systems for Machine Learning and Machine Learning for Systems 🚀 🦖

0

8

Simran Arora

@simran_s_arora

8 days

@srush_nlp @violet_zct @realDanFu @SonglinYang4 Thanks Sasha, it was really fun to visit you all! And +1, am a big fan of those other works as well

1

0

8

Simran Arora

@simran_s_arora

2 years

HUGE thank you to Together Computer (), Numbers Station (), Snorkel ( @SnorkelAI , ), @StanfordCRFM , @StanfordHAI , @StanfordAILab for the resources to make this possible! 🙏🙏🙏

1

8

Simran Arora

@simran_s_arora

2 years

Thanks for the write up @StanfordHAI ! Excited to explore approaches that put privacy decisions in the hands of users 🔒🔑

Can Foundation Models Help Us Achieve “Perfect Secrecy”?

A new study explores how to apply machine learning to digital assistants in a way that could better protect our data.

hai.stanford.edu

0

5

8

Simran Arora

@simran_s_arora

5 months

And for those of you that insist on architectural purity, we can also unify the modules of Based✌️ – both can be viewed as gated convolution layers, which just differ in how they parameterize the filter weights!

1

0

7

Simran Arora

@simran_s_arora

2 years

We propose a privacy framework inspired by the classical Bell-LaPadula model, developed in the '70s and widely used by the government to manage multi-level access control!🕵️‍♀️ We also release a new dataset, ConcurrentQA, to effectively study the proposed retrieval setting!

1

6

Simran Arora

@simran_s_arora

2 years

We are excited to continue pushing the limits of where strong privacy is achievable and to promote **more challenging** benchmarks for personalized workloads in next steps 📈 We welcome contributions and feedback!

GitHub - simran-arora/focus: This repo contains code for the paper: "Can Foundation Models Help Us...

This repo contains code for the paper: "Can Foundation Models Help Us Achieve Perfect Secrecy?" - simran-arora/focus

github.com

1

7

Simran Arora

@simran_s_arora

5 months

Meanwhile, the Based✌️linear attentions enable Based✌️ to ~ sub-quadratically in sequence length ~ do associative recall, e.g., by recovering the global “look-up” inductive bias of standard attention.

1

0

7

Simran Arora

@simran_s_arora

5 months

Based✌️uses fixed-sized convs. & linear attentions (both computable as recurrences), so we can decode with no KV-cache. When implemented in just PyTorch, Based✌️gives 4.5x > inference throughput vs. competitive Transformers (param-matched Mistral w sliding window attention & FA2)

1

0

7

Simran Arora

@simran_s_arora

10 months

Your retriever must jointly rank documents from two distributions (e.g., Amazon & Walmart), but the catch is that you only have training data from one domain! Are in-domain documents going to dominate the rankings? How do we fix this? We build three benchmarks to explore this!

1

6

Simran Arora

@simran_s_arora

5 months

Standard convolutions are great for modeling local dependencies and settings where we might not expect to need AR (think building up morphemes ~ units of meaning ~ from individual tokens, similar to how in vision we build up higher-level features from neighboring pixels).

1

0

6

Simran Arora

@simran_s_arora

5 months

We further evaluate on our synthetic testbed for AR ability -- called MQAR, which we motivated in our recent Zoology paper: We see heads improve scaling for some more FLOP$! Mamba uses large state dimension to perform more recall per sequence.

2

0

6

Simran Arora

@simran_s_arora

2 years

We are inspired by and share ideas with works such as maieutic prompting ( @jaehunjung_com ), amazing prompting work from @Swarooprm7 , AI-Chains ( @tongshuangwu ), Self-ask ( @OfirPress ), and many others! 💫

1

2

6

Simran Arora

@simran_s_arora

1 year

Checkout this awesome tool from awesome labmates!!

Karan Goel

@krandiash

1 year

We built an interactive data frame powered by foundation models that can wrangle your unstructured data (images, videos, text docs...) Introducing 🔮 Meerkat! 📃 💻 🌐

3

62

206

0

6

Simran Arora

@simran_s_arora

4 years

Excited to share this work! Code: Paper:

GitHub - HazyResearch/bootleg: Self-Supervision for Named Entity Disambiguation at the Tail

Self-Supervision for Named Entity Disambiguation at the Tail - HazyResearch/bootleg

github.com

Laurel Orr

@laurel_orr1

4 years

(1/5) Excited to release Bootleg , a SotA named entity disambiguation system that tackles the long tail of entities that appear infrequently in training. Bootleg achieves SotA on 3 NED benchmarks and outperforms a BERT-based baseline by 40 F1 over the tail.

2

43

119

0

1

6

Simran Arora

@simran_s_arora

2 months

Evaluating the broad set of efficient architecture candidates, we theoretically and empirically show a fundamental tradeoff between recall ability and memory consumption (throughput) during inference. We find, for fixed state-size, Based outperforms baseline linear attentions and…

1

3

6

Simran Arora

@simran_s_arora

2 months

For efficiency, we create IO-aware algorithms for generation. Key is how the algorithm partitions the large linear attention recurrent state to be held across GPU thread register memory, avoiding excessive reads and writes to relatively slow SRAM memory, and further to slow HBM…

1

0

6

Simran Arora

@simran_s_arora

2 years

@jaehunjung_com @Swarooprm7 @tongshuangwu @OfirPress @StationAI !!!

1

0

5

Simran Arora

@simran_s_arora

4 years

thanks for everything

NPR Politics

@nprpolitics

4 years

Justice Ruth Bader Ginsburg, Champion Of Gender Equality, Dies At 87

7K

70K

134K

0

1

5

Simran Arora

@simran_s_arora

2 months

@andersonbcdefg @_akhaliq hi! the blogpost Based was also hybrid (not only linear attn.) -- we had short-convs for local shifts/mixing (also included in Table 4). SWA helped more and is in this latest version. high level point is using some "precise" local mixer alongside the coarse attn. approximation

1

0

4

Simran Arora

@simran_s_arora

2 months

Building up to Based, last year, we built Evaporate and @MeerkatML – prototype systems for handling high throughput data management workloads – e.g. document information extraction, QA, codegen, summarization – with LLMs. These tasks require the LLM to recall info provided in…

1

0

5

Simran Arora

@simran_s_arora

10 months

We explore simple baselines for retrieving from two distributions that lead to an average of 3.8+ and up to 8.0 points improvements in Recall @100 across the datasets. Many avenues for future work: better models, better aggregation, and more complex settings!

0

5

Simran Arora

@simran_s_arora

2 years

Thanks to amazing collaborators @HazyResearch at @StanfordAILab and Jacob Kahn, @PSH_Lewis , Angela Fan at @MetaAI !!! 🙏

0

4

Simran Arora

@simran_s_arora

5 months

@CFGeek Thanks, coming soon!

0

5

Simran Arora

@simran_s_arora

2 years

This is joint work with amazing collaborators: @avanika15 , @MayeeChen , @laurel_orr1 , @NeelGuha , @kushbhatia03 , @chami22 , @fredsala , and the brilliant @HazyResearch !!! We’d love your feedback!

1

2

5

Simran Arora

@simran_s_arora

2 months

Based emerges from a combination of our ICLR 2024 works on the recall problem in sub-quadratic models and on expressive linear attentions ! Sliding window is great for performing precise local shifts and token comparisons needed for…

1

4

Simran Arora

@simran_s_arora

4 months

The M2-Retrieval models build on our recent Monarch Mixer work, which enables models that are sub-quadratic in both sequence length and model dimension. This helps efficiently scale to long sequences!

Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture

Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts and better performance. However, existing architectures such as...

arxiv.org

1

0

4

Simran Arora

@simran_s_arora

3 years

Checkout our accompanying blog post and we'd love to hear your thoughts!

What can we accomplish without changing the architecture? A thought experiment in incorporating...

hazyresearch.stanford.edu

1

0

4

Simran Arora

@simran_s_arora

4 months

Checkout the blogpost and @realDanFu 's post for more!

Long-Context Retrieval Models with Monarch Mixer

hazyresearch.stanford.edu

Dan Fu

@realDanFu

4 months

New year, new model drop! w/ @JonSaadFalcon , @simran_s_arora , excited to release new long-context retrieval models with Monarch Mixer, up to 32K sequence length! First step 2 long-context retrieval, outperforming Mistral, BGE, OpenAI on long-context document retrieval. 1/

4

42

231

0

1

4

Simran Arora

@simran_s_arora

2 months

In downstream evals, 1.3Bn parameter Based improves by 28% accuracy points on average over Mamba on real-world recall-intensive ICL tasks. Based outperforms Mamba on few-shot ICL on SuperGLUE, and matches in overall perplexity, standard LM harness evals, and DNA ppl and…

1

4

Simran Arora

@simran_s_arora

4 months

There’s lots of work on long-context architectures, but architecture isn’t all – the process of building M2-Retrievers emphasized there’s lots to learn in *how we train* models to use the context effectively. For e.g., checkout our discussion on data mixtures and loss functions.

1

0

4

Simran Arora

@simran_s_arora

1 year

Awesome paper on the use of public resources for improved privacy and what it means for resources to be truly public!!! Exciting to figure out how to get even more out of smaller, open-source models over time, re scale! , , and more

Scaling Instruction-Finetuned Language Models

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction...

arxiv.org

Gautam Kamath

@thegautamkamath

1 year

3. Scale makes ML hard to use in a truly private fashion. If you want to do inference on a point without sharing it, you either have to fine-tune&run the model locally (see e.g. this nice paper by @simran_s_arora @HazyResearch ), or use FHE encryption. 10/n

1

0

7

0

1

3

Simran Arora

@simran_s_arora

3 years

We take an information theoretic view of domain knowledge inserted in the data at train and test time in a method called metadata shaping, and show that this approach is competitive with several language model architectures that have been proposed for knowledge-integration.

1

3

Simran Arora

@simran_s_arora

4 months

There’s also more to be learned about where long context retrieval matters. E.g., where can we get away with splitting long documents into small chunks and retrieving over those? We’re building a new benchmark suite, LoCo, as we investigate these questions. We'd love feedback!

1

0

3

Simran Arora

@simran_s_arora

2 months

@MikeK_LA @EyubogluSabri A fixed size state can store limited information, simple communication complexity argument (thm 3.1 in paper), so no not limitless. A better recurrent model will use its state more effectively and/or expand its state in a way that is still efficient to compute on hardware --…

0

3

Simran Arora

@simran_s_arora

4 months

@teortaxesTex yes the results in the paper and repo are v4 so far

0

1

Simran Arora

@simran_s_arora

2 months

@CFGeek @EyubogluSabri hi thanks for the question -- we've seen hedgehog map effective in distillation settings and Taylor map more effective in training from scratch

1

0

2

Simran Arora

@simran_s_arora

1 year

@insitusec Thanks for the question! On the FDA and SWDE benchmarks there are more varied and tricky attribute placements so you can start there. In terms of efficient ways to extract non-structured attributes across many documents, this is an exciting direction for future work!

0

2

Simran Arora

@simran_s_arora

1 year

@StefanB0305 @gordic_aleksa @Avanika15 @MayeeChen @laurel_orr1 @NeelGuha @chamii22 @fredsala @HazyResearch @Stanford Hi! For Table 1 with T0, I include 7 numbers in the attachment. You can use the provided code for other tasks. I also just added the following code to help run aggregation over P3 prompts (Table 2 in paper): Let me know if you have further questions!

1

0

1

Simran Arora

@simran_s_arora

2 years

@ImmanuelTrummer @HazyResearch @StanfordAILab @laurel_orr1 @Avanika15 @BeidiChen @Stanford Thanks for visiting and the amazing talk!!

1

0

2

Simran Arora

@simran_s_arora

7 months

💫

Serina Chang

@serinachang5

7 months

I’m on the academic job market! I’ll have a PhD from @Stanford CS in 2024. My research develops ML + network science methods to tackle complex societal challenges, from pandemics to polarization to supply chains. See my website + research statement for details! Highlights below:

22

130

812

0

2

Simran Arora

@simran_s_arora

2 years

These thoughts are evolving and a shout out to all the inspirational work that helped shape these ideas. Let us know what you think and send more pointers!! @StanfordCRFM @StanfordHAI @SnorkelAI @StanfordAILab

0

1

Simran Arora Retweeted

AK

@_akhaliq

2 years

Ask Me Anything: A simple strategy for prompting language models abs: github:

2

95

499

Simran Arora

@simran_s_arora

2 years

We take the first step in benchmarking a range of foundation models on popular tasks in the privacy literature, within the FOCUS framework -- perfect secrecy is achievable on many tasks! However, there are limitations -- success varies by task difficulty, and model size and bias.

1

0

1

Simran Arora

@simran_s_arora

2 years

Thanks to members of the @StanfordAILab for feedback on this work and to @StanfordHAI for compute resources!

0

1

Simran Arora

@simran_s_arora

4 years

@srchvrs @ravisujith @sleepinyourhat Thanks for sending! Interested to see how the recent exposure and our improving understanding of the trade-offs between computational costs, data-labeling costs, and performance that arise from using different types of embeddings, will inform our embedding choices and designs!

1

0

1