Linus @thesephist Twitter profile

Pinned Tweet

Linus

7 months

At @aiDotEngineer this evening, I shared that the text autoencoder model I've been prototyping with, which I call Contra ✨, is on @huggingface ! Some starter code + demos👇 Colab notebook — Slides — Model —

Contra (Bottleneck T5) - a thesephist Collection

huggingface.co

17

29

260

Last Seen Profiles

@ScottAgness

@emmerrera

@Compass_Log1

@MoiCestSteven

@AHassaballah

@horisankameko

@YeshivaLink

@RealRazzieBinx

@bts7oclock

@onedirection

@StephStaszkoAct

@FortunataFox

@tomflict

@bluedotfestival

@AnanNajjar1

@WestOneMusicGrp

@hawtgirlnov

@thedankoe

@sewistwrites

@_Dirty7

@vapersitos

@RIFWEB

@Wes_engage

@AsMatchApp

@jifferey

@sachicoo66

@annapolinaxxxx

@UKPARTYPEOPLE24

@maggybeck1

@Maz_Ipan

@MintySnep

@NUAlumni

@__morggxo

@JMM_2020

@mpaauk

@JUST__ENT

Linus

@thesephist

1 year

The vibes of this blog post

Sundar Pichai

@sundarpichai

1 year

1/ In 2021, we shared next-gen language + conversation capabilities powered by our Language Model for Dialogue Applications (LaMDA). Coming soon: Bard, a new experimental conversational #GoogleAI service powered by LaMDA.

742

3K

15K

53

1K

11K

Linus

@thesephist

2 years

We're max 2-3 years out from DALL-E 2 for 3D printing. Literally conjuring objects from incantations.

151

445

5K

Linus

@thesephist

8 months

it's important to communicate with your coworkers with kindness and clarity

30

288

4K

Linus

@thesephist

2 years

Made a little CLI that just pipes my programming questions to GPT-3, so I now can ask it stuff when I'm in the command line! LLMs are better than Stack Overflow now — I just ask it, and it gives me a comprehensive answer in one shot, right there in my terminal, in a couple secs.

77

371

4K

Linus

@thesephist

6 months

Somewhere nestled deep within the digits of Pi, there exists the full weights of GPT-4, GPT-5, and all future neural networks.

99

164

3K

Linus

@thesephist

3 years

Thiel Fellowship but for paying engineers to drop out of FANG companies

39

115

3K

Linus

@thesephist

3 years

NEW PROJECT — I made a "personal search engine" that lets me search all my blogs, tweets, journals, notes, contacts, & more at once 🚀 It's called Monocle, and features a full text search system written in Ink 👇 GitHub ⌨️ Demo 🔍

65

172

2K

Linus

@thesephist

11 months

all these fancy nyc cafes with their no laptop policies are getting out of control gonna open a cafe where you can't come in unless you have more than 50 unread emails and a sidebar overflowing w slack notifications and can't leave unless you clear em all out

34

38

2K

Linus

@thesephist

1 year

Life update🎉 I'm very excited to be joining @NotionHQ to continue prototyping and researching ways AI can help us be more creative, thoughtful, and productive! Looking forward to learning from the team and bringing some of my ideas from the past year to a tool loved by many 👇

111

22

2K

Linus

@thesephist

1 year

I built a personal chatbot from my personal corpus[1] a couple weeks ago on fully open-source LMs. On a whim I gave it iMessage. Didn't expect the iMessage bit to matter, but it made a huge difference in how it feels to interact. Much more natural. [1]

26

111

1K

Linus

@thesephist

8 months

building bicycles for the mind... 🚴‍♀️

25

40

981

Linus

@thesephist

2 years

Diffusion models for handwriting generation! Cool work.

25

118

941

Linus

@thesephist

2 years

Half of @amasad tweets these days are like "Replit users can now run their own fusion reactor from their bedroom. We had this idea last weekend and it took our two engineers two lunch breaks to build. By next month we'll have a million teenage coders generating fusion power."

11

52

923

Linus

@thesephist

2 years

Code retention/churn over the last ~15 years for Clojure and Scala's codebases. Source:

22

172

891

Linus

@thesephist

1 year

Small rant about LLMs and how I see them being put, rather thoughtlessly IMO, into productivity tools. 📄 TL;DR — Most knowledge work isn't a text-generation task, and your product shouldn't ship an implementation detail of LLMs as the end-user interface

44

97

810

Linus

@thesephist

1 year

Built a token-wise likelihood visualizer for GPT-2 over the weekend. There are some interesting patterns and behaviors you can easily pick up from a visualization like this, like induction heads and which kinds of words/grammar LMs like to guess.

14

88

784

Linus

@thesephist

1 year

I desperately need someone to take me out for an evening and say not a single word about anything related to AI. I am drowning.

47

27

767

Linus

@thesephist

1 year

Insane ML paper acronyms continue to proliferate in 2023

16

39

682

Linus

@thesephist

11 months

y'all need to stop founding LLM infra companies and start going on dates

14

36

635

Linus

@thesephist

1 year

Here we go, fine-tuning GPT-3 on the vast majority of my public online writing (half million tokens)...

37

27

613

Linus

@thesephist

3 years

A tragically underutilized fact in productivity software today is that most people's entire textual datasets for a lifetime can fit in modern PCs' RAM. Just load it up & search it in memory. We don't need to send everything across the planet. Things can be /so much/ faster.

21

50

608

Linus

@thesephist

6 months

timeline just forked

12

47

591

Linus

@thesephist

8 months

Today's experiment 🪄— Inverting OpenAI's embedding-ada-002 model to reconstruct input texts from just embeddings. A LOT of interesting tidbits here. I'll begin with these (cherry-picked) samples. Left column is input, middle is reconstructed from each paragraph's embedding only

17

62

551

Linus

@thesephist

1 year

👋

8

1

521

Linus

@thesephist

2 years

ML research githubs are all like "This repo lets you reproduce and build on our results. Simply run ./scripts/train_best_model.py --with-ffn=32 --gru-cache=100 --magic-sauce --m_dims=3.1415926535 --unicorns=no_exist * --m-dims should be set to exact value of Pi for best results

10

34

490

Linus

@thesephist

2 years

NEW DEMO! Exploring the "length" dimension in the latent space of a language model ✨ By scrubbing up/down across the text, I'm moving this sentence up and down a direction in the embedding space corresponding to text length — producing summaries w/ precise length control (1/n)

14

54

492

Linus

@thesephist

2 years

some life update🚀 Last week was my last at Ideaflow. Starting 2022, I'm working full-time on building products, prototypes, and experiments investigating how we can build better software tools for creating and thinking. More coming soon, but wanted to get the news out early :)

44

1

489

Linus

@thesephist

2 years

Single-purpose computers! They're great.

11

10

460

Linus

@thesephist

3 years

@mariosangiorgio This is hilariously smart haha Who needs Ctrl-save when you can just increment the pc manually when your editor hangs.

4

101

449

Linus

@thesephist

6 months

Weird idea: chunk size when doing retrieval-augmented generation is an annoying hyperparam & feels naive to tune it to a global constant value. Could we train an e2e chunking model? i.e. system that takes in a long passage, and outputs a sequence of [span, embedding] pairs?

40

19

438

Linus

@thesephist

5 months

Wow, I just got @AnthropicAI 's sparse autoencoder-based feature decomposition technique to work* for text embeddings 🎆 Screenshot below. In order, this output shows: 1. max-activating examples for that feature from the Minipile dataset 2. min-activating examples from the same…

10

33

428

Linus

@thesephist

4 years

is #2 on Hacker News LMAOO I can't 😂

21

18

401

Linus

@thesephist

1 year

This is your periodic reminder that user interfaces are important, and text is a good lowest common denominator, not the endgame. The world and our senses have a lot more to offer.

16

27

382

Linus

@thesephist

3 years

can we not turn twitter into linkedin actually

5

8

372

Linus

@thesephist

1 year

Thinking about notation design again...

13

27

361

Linus

@thesephist

1 year

I sat down with @danshipper to talk about how I work! I go through the tools I use for my work and why, focusing on the ones that leverage LLMs to help me read and think. Also some peek into my past prototypes, and recs for book that inspire my work 📚

Linus Lee Is Living With AI

How a researcher uses generative AI to help him think better and get more done

every.to

12

33

348

Linus

@thesephist

2 months

stanford symbolic systems is the hottest major in tech no room for questions

24

11

350

Linus

@thesephist

5 years

@Mantia @eliz_kilic I don't think a slightly chubbier octothorpe looks bad though if you balance the margins right

17

21

344

Linus

@thesephist

2 years

Brewing currently 🧪 Exploring a language model's latent space on a connected canvas, branching from a single idea through connections to a tree of alternate realities.

13

27

346

Linus

@thesephist

3 years

How to commit to the right opportunities

7

24

343

Linus

@thesephist

2 years

Things that are horrifically harder than they should be: - Text rendering - Rich text editors - Implementing undo/redo that won't make you pull your hair out (when mixed with autocorrect, formatting, page navigation, etc. etc.) Tonight I'm wrestling with the third, apparently!

12

8

340

Linus

@thesephist

5 months

This morning, I've been sketching out ideas for a chat interface to language models that treat branching/multiple timelines as a first-class concept and try to make heavily branch-y threads navigable. Some notes I've been taking...

Better interfaces for dialogue | Notion

These are my working notes for a (possibly new) project. It’s a draft notebook of open questions I have and ideas I think are interesting, including thoughts and perspectives I might not necessarily...

thesephist.notion.site

20

28

341

Linus

@thesephist

2 years

Good tools admit virtuosity — they have low floors and high ceilings, and are open to beginners but support mastery, so that experts can deftly close the gap between their taste and their craft. Prompt engineering does not admit virtuosity. We need something better.

11

43

337

Linus

@thesephist

1 year

Not a single vector DB in sight. I have found peace 🏝️

6

4

340

Linus

@thesephist

8 months

Quick little hack 🦄 — a GPT token probability visualizer Given lots of interest in my little LLM visualization from earlier in the year and a little encouragement from @simonw , I decided to break this out into its own little fully client-side app! 🔗

Perplexity: Interactive language modeling visualization

I built this little tool to help me understand what it's like to be an autoregressive language model. For any given passage of text, it augments the original text with highlights and annotations that...

perplexity.vercel.app

10

51

330

Linus

@thesephist

4 years

It's been a wild week for me. - 2x HN, 2x @ProductHunt - 100k site visits - 1.4k👉2k followers - Good convos w/ founders, VC folks My main takeaway: There is SO MUCH room in the world for projects that don't necessarily aspire to solve the world's problems. Fun is ok, too.

11

8

326

Linus

@thesephist

1 year

You guys are telling me we are going to invent literal superintelligence and we are going to interact with it by sending texts

31

12

314

Linus

@thesephist

3 years

We don't talk enough about the fact that most creative software on the computer works by simulating a fake piece of paper and a fake typewriter or pen just so we don't have to think of or learn fundamentally new interaction modes for this fundamentally new medium.

19

27

318

Linus

@thesephist

3 years

Even the best current "tools for thought" apps require you to remember to manually make all the connections between your ideas. Is anyone working on making the computer participate and help in this process? Suggesting connections? Finding missing links? I want to talk to you 👋

41

15

298

Linus

@thesephist

1 year

Open source: a story in 4 parts

5

16

297

Linus

@thesephist

7 months

a beautiful name for a baby boy

13

8

292

Linus

@thesephist

4 months

Embedding features learned with sparse autoencoders can make semantic edits to text ✨ (+ a reading/highlighting demo) I've built an interface to explore and visualize GPT-4 labelled features learned from a text embedding model's latent space. Here's a little video, more in 👇

11

32

274

Linus

@thesephist

3 years

So I haven't done this before for some reason, but I laid out all my projects listed on side by side, and... ... yeah. I've been busy 😂 A little over 120 projects in all, most of them still functional and online! Gotta celebrate milestones sometimes 💪

17

8

274

Linus

@thesephist

1 year

There Are So Many PromptOps Tools And I'm Sold On None Of Them

20

18

273

Linus

@thesephist

1 year

Most people don't want a Photoshop for Stable Diffusion; they want an Instagram.

10

12

261

Linus

@thesephist

3 months

mentally i am here

11

22

264

Linus

@thesephist

19 days

A while ago I complained here about persistent storage in Google Colab. Have been using @LightningAI Studios for a while now for: - Full VSCode (incl. GH Copilot) - Persisted files shared across notebooks - Multi-GPU/node (!!) It's been great. Feels like a remote ML workstation

7

35

264

Linus

@thesephist

10 months

stand back, I'm a professional -- >>>content.split("...")[2].strip().split('"""')[0].strip().split("\n")

22

8

262

Linus

@thesephist

2 years

Sometimes I feel like there are two visions of the future at the edges of tech right now: To engineer scarcity into everything (crypto) To engineer scarcity out of everything (generative AI) Cyberpunk vs. solarpunk. Singularity vs. singularity.

13

35

251

Linus

@thesephist

11 months

Today @jasonyuandesign was like "I'm gonna show you a quick 5sec demo" and I saw it and I don't think I'll ever look at software the same again.

11

3

254

Linus

@thesephist

4 years

Thinking about launching an OnlyFans but you get to see all my private repositories on GitHub instead 💋

9

18

250

Linus

@thesephist

11 months

Thinking about Makepad's continuous code folding animation again. Feels like we should be able to do this with prose text now — find the key ideas/sentences and zoom out the rest of a document.

6

20

251

Linus

@thesephist

3 years

RT if you're also a grandpa

6

10

245

Linus

@thesephist

4 years

Thinking about building a "personal search engine" A search engine that only indexes my blog, my Tweets, my journal, my calendar/email and contacts, my photos, and browser history. I want to have better memory without having to remember more stuff. What else should it index?

37

5

245

Linus

@thesephist

2 years

NEW LIL PROJECT — Just some burds! 🐦

burds!

... just a bunch of burds, jumpin' around

burds.vercel.app

13

16

244

Linus

@thesephist

3 months

Encouraged by some conversations I've had recently, I put together a list of links/papers/reports you might find interesting if you like my work. Covers interpretability + model visualization, interface thinking, stories/fiction. I'll be adding more.

6

25

241

Linus

@thesephist

10 days

if artificial neural networks are a kind of alien intelligence, can we use it to imagine alien languages? how could a NN teach itself to "write down" information without any human priors of what writing looks like?

21

17

257

Linus

@thesephist

1 year

*sits down for a date* "So, I'm training a chinchilla-optimal open source 7B parameter LLM."

17

4

238

Linus

@thesephist

3 months

Back in 2022 in my ✨experimental✨ era I wrote down a whole bunch of ideas for tools and interfaces I want to make, but didn't get to actually prototype many of them. Here's a thread of the ones I think would still be interesting, starting with this weird mobile browser concept.

3

14

239

Linus

@thesephist

8 months

to date, this is still the best demo I've built/found to explain to folks outside of NLP how an LLM works. Interactively visualizing autoregressive sampling from a GPT-style model.

Linus

@thesephist

1 year

Built a token-wise likelihood visualizer for GPT-2 over the weekend. There are some interesting patterns and behaviors you can easily pick up from a visualization like this, like induction heads and which kinds of words/grammar LMs like to guess.

14

88

784

4

28

227

Linus

@thesephist

5 months

entering a group chat should feel like this

11

14

224

Linus

@thesephist

10 months

New tool for thought just dropped

18

7

220

Linus

@thesephist

3 months

Anthropic's rigor in research, their long-term principled foresight and short-term prioritization, thorough reports, and (perhaps most obviously) class-leading research communication and collaboration frequently has me in awe. Need more orgs like this.

3

18

217

Linus

@thesephist

1 year

Interesting concept msft is calling "Token healing" in their Guidance project (seems similar to LMQL): Simple clever workaround for the "don't end your prompts with a whitespace" problem. Surprised I haven't seen it before.

3

25

217

Linus

@thesephist

6 months

This is a really interesting way to visualize QKV attention! I don't think I've seen it anywhere else. The embeddings as visualized here are kind of useless but combined with sparse autoencoder-based features from more recent work, might be interesting? source: chemBERTa paper

4

31

214

Linus

@thesephist

9 months

this is a masterclass in data visualization. one of my favorite things i've seen for visualizing training dynamics of toy neural networks. so cool!

1

29

214

Linus

@thesephist

1 month

Think i found a better (gradient-based) way to edit stuff in latent space. Far more precise and steerable than previous methods of just moving embeddings in different directions or adding vectors together 👇

10

16

214

Linus

@thesephist

3 years

I would have been pretty excited for a Clubhouse that felt like Tumblr or Reddit but instead it's just turning into a LInkedin in audio form.

12

3

209

Linus

@thesephist

4 years

MIT kids be like I'm taking course 3.14159265

1

10

209

Linus

@thesephist

4 years

Currently in my twitter notifications.

9

6

198

Linus

@thesephist

3 months

some slides that didn't make it into the final draft, but I think are worth thinking about

16

15

204

Linus

@thesephist

3 years

Experimenting with an idea: Chrome extension that summarizes long articles for me before I spend a lot of time reading it.

11

10

204

Linus

@thesephist

2 years

It seems like almost everyone is building something on GPT3 these days. But few have ever looked at its parameters. I spent the last year studying all 175B parameters of GPT-3. Here are my favorite 6B 🧵 (1/6000000001)

4

5

200

Linus

@thesephist

1 year

Good user interfaces don't just lower barrier to entry. They present mental models that align with underlying software behaviors, so users can contend with complexity when necessary. Chat has to be the start, not the end, of AI UI. Talk to Nick if you're working in this space!

Nick Arner

@nickarner

1 year

New blog post - LLM Powered Assistants for Complex Interfaces

13

45

343

3

11

196

Linus

@thesephist

10 months

Todays research reading 🍷

9

7

197

Linus

@thesephist

5 months

the macos emoji picker is the single most frustrating piece of software ever devised by humankind

17

3

200

Linus

@thesephist

6 months

Levine doesn't miss

3

5

198

Linus

@thesephist

4 years

@whrobbins I would argue this is a side effect of a more fundamental character trait: knowing very clearly and to a high degree of confidence what it is they want to do with their time/resources, independent of the in-vogue pursuit of the voices around them.

0

9

191

Linus

@thesephist

1 month

I feel like a good AI app shouldn't "write for you" or "search for you" any more than a good drawing app "sketches for you." Inventing better media and tools, not replacements and prosthetics.

11

9

199

Linus

@thesephist

2 years

NEW PROJECT — 🦄 YC Vibe Check 🦄 YCVC is a semantic search engine over *every YC company ever*. Type in an idea, vertical, or problem space and see every YC co that's worked on it, and even some stories about them online. 🔗 💻

8

11

195

Linus

@thesephist

4 years

Ever been coding and thought, "Boy, this would be way better if code felt like a tabloid magazine page with clickbaity headlines!" Well have I GOT a PROGRAMMING LANGUAGE for YOU 🎉 Launching "Tabloid," a clickbait language 🔥 💻

9

40

195

Linus

@thesephist

7 months

do u even hydrate dawg

19

6

193

Linus

@thesephist

2 months

i can't be the only one who just has one giant tldraw file with every diagram i've ever made on it in case i need to edit it again

22

7

193

Linus

@thesephist

1 month

google colab would simply be an unstoppable tool if only there was a persisted storage mechanism that did not require using your personal Google Drive to store absolutely everything

9

3

190

Linus

@thesephist

4 years

* sits down for a date * "I have gpt3 early access"

7

6

188

Linus

@thesephist

2 years

the male urge to make a new rich text editor

12

8

182

Linus

@thesephist

1 year

If I see another GPT3 writing app called GhostWriter I'm gonna lose it

18

1

185

Linus

@thesephist

6 months

Sharing our next step in AI today! 𝗡𝗼𝘁𝗶𝗼𝗻 𝗔𝗜 𝗤&𝗔 🪄—ask questions, search, and synthesize info in your Notion. Our team took SOTA in LLMs and pushed beyond it for the last months. We're really proud of where it is today & where we're headed👇

Introducing Q&A: get instant answers to your questions in Notion

Q&A is now available in beta. Leveraging information across your Notion workspace, Q&A can instantly pull answers to questions to help you spend more time doing and less time searching.

www.notion.so

11

8

187

Linus

@thesephist

2 years

Everyone in the group chat gets blocked by @pmarca call that a block chain

8

6

183