We don't talk enough about how insane aluminum foil is
Imagine telling some ancient person we have so much abundance in our time that we use very thin metal as a disposable paper-like wrapping and it costs essentially nothing
Wait so better reaction times are correlated to higher IQ??
How much of being smart is just your neurons firing faster?
Can we just overclock this meat CPU?
My friend works for a large nonprofit
They just received a $20K donation and are having a 110 person, 90 minute meeting discussing how they’re gonna spend it
The total value of these 110 people’s 90 minutes is $10K
Had my American mind blown when visiting a friend in Germany
There was a 300 year old building they'd gutted and put in a Burger King
I said "Alex, this is awful! They turned this 300 year old building into a fast food place!"
He said "Grant, it's *only* 300 years old"
Founder of Trader Joe's set up a bonus system where overperforming store managers are the highest compensated employees of the business — more than those at HQ
He wanted to avoid any incentives for them to want to get into the bureaucracy
Big tech companies should take note IMO
@SurrealistShip
I’m doing what? Idk man I am just pressing random buttons really fast. It’s called wavedashing? Oh my guy, what’s his name, Falcrow? I’m shield tilting now? Really? Where do you come up with these crazy names haha. Oh wow, 20-0, this game is fun.
A couple years ago I started installing these dog bag dispensers on all the telephone poles around our house
I restock them as we walk our dog, takes no time and maybe $20/yr of bags
It's easy to add to the commons, just do it, nobody will stop you
There was a great line I couldn't quite remember from "One Hundred Years of Solitude"
Search engines & ChatGPT couldn't recall it either from my bad description
Wrote a little 20 line program to read the whole book and found it in a minute, cost 25 cents
Wild times we live in
@_brendand_
I think it's interesting because they could *comprehend* it immediately -- they had metal foil back then, too (copper, gold, tin), it was just insanely expensive
A favorite load balancing technique at AWS is "the power of two random choices"
On the left, nodes are chosen and used at random
On the right, 2 nodes are chosen at random, but only the minimum is used
This simple technique balances load very well
The male-coded version of astrology is the guys who think they can see simple patterns in stock price graphs
"Oh yeah that's a double-mountain followed by a head-and-shoulders, time to buy"
There's a lot of talent locked up in risk-averse FAANG staff engineers
Weird to me more bigtechs don't offer in-house startup programs with these people
"We'll pay salary and take equity, try your idea for a year, if it doesn't work, go back to your old job"
The new GPT model, gpt-3.5-turbo-instruct, can play chess around 1800 Elo.
I had previously reported that GPT cannot play chess, but it appears this was just the RLHF'd chat models. The pure completion model succeeds.
See game & thoughts below:
GPT4 cannot play chess.
Expected to get a bot up to ~1000 Elo and totally failed -- but negative results are still interesting!
Even given verbose prompting, board state descriptions, CoT, etc, it loses badly to depth=1 engine.
Code & notes below.
It's weird to me more self-avowed communists don't set up actual communes
It's actually really easy to mostly exit modern society and form your own parallel thing, just look at the Amish
Farmland is like $3K/acre, easy to pool resources and form your own little town if you want
GPT can iteratively write, debug, and test programs to accomplish arbitrary goals.
Pictured: GPT reading snippets of HTML from HN and building a headline scraper in Python, overcoming bugs by simply reading the errors and self-judgments and hypothesizing to itself.
Thread ↓
I knew a kid in elementary school who would assign a daily ranking to other kids
Some kids cared about the rankings, giving him status, which made more kids care about his rankings, and soon enough he was the highest status kid on the playground
Child psychology is wild
My most toxic belief is thinking I could turn a batch of highschool graduates into L5 FAANG engineers in about 2 years of full-time work
One year to learn all the CS fundamentals, another year for software engineering
University overrated, apprenticeship learning underrated
Added arbitrary context free grammar constraints to llama.cpp
Can now plug in any llama.cpp compatible model and give an exact grammar spec: JSON, etc
Excited to use with more powerful local models as they are released
Thanks
@ggerganov
& friends for such a wonderful project.
GPT can use a web browser to answer questions.
When embedded in a REPL environment and prompted to strategize and monologue, agent-like behavior emerges. The agent can solve multi-step problems that involve going to pages, following links, reading the next page, etc.
Thread ↓
My all-time favorite datastructures & algorithms interview question:
Design a key-value datastructure with methods: insert, delete, and getRandom
All methods must be faster than O(N)
There are at least 3 solutions, and it's a real problem I actually had to solve at work
Python won because you can download IDLE, type print("Hello world") and you're now a programmer
As a kid, I tried to "download C++"
What's GCC? What's a linker? How to point my IDE to it? As a kid, you don't even know what words to google.
So Python it was
A year ago we spent Christmas in the hospital as my wife had a serious health issue
Doctors misdiagnosed and mistreated her for weeks, but eventually a specialist figured it out
Months later, GPT4 came out and correctly diagnosed given only the notes from the first doctor
I'm still open to the idea that all LLMs can do is interpolate, but that we also don't have a good grasp on what it even means to interpolate between concepts in billion dimensional space
Maybe all the ideas we could ever have are already inside the convex hull of existing ideas
My favorite way to think about this is with space-filling curves
For example, the picture shows a 1D line is able to encode 3D coordinates
If you train an autoencoder to do this and visualize the output it will look roughly like this
It’s wild that models encode more features (things like random street names, people, concepts, etc) than they have parameters.
But a parameter is just a single number.
How is this even possible?
@TrentonBricken
and
@_sholtodouglas
on superposition:
@_nmilo
It’s a medical nonprofit so this corresponds to patients the medical staff isn’t seeing while they attend the meeting which directly translates to revenue
"Wow amazing result!"
I know right? And there's only a 5% chance it's due to statistical error.
"That's so impressive! You're so much better than other researchers!"
Yeah! 19 other labs tried this experiment and failed — losers!
Being able to apply bulk ad-hoc intelligence to tasks like this makes so many things feasible that weren't before.
Doing things like academic research where you need to crawl through huge tomes is going to get so fun and efficient.
Voice commanded essay copilot
I always wanted the ability to lean back in my chair and talk an essay into existence, rambling as needed, letting the AI organize my thoughts.
Transcription:
@ggerganov
's whisper.cpp
App:
@TauriApps
Backend: GPT 3.5 with new functions API
GPT can execute fairly complicated programs as long as you make it print out all state updates. Here is a linear feedback shift register (i.e. pseudorandom number generator). Real Python REPL for reference.
I'm gonna be so disappointed if humanity doesn't discover any better ML techniques and AGI winds up just being GPT-19 with 100T parameters and 10 billion token context window or something dumb like that
It would probably work, but would just be such a let-down
The world wasn't created 4000 years ago
It was created at midnight on January 1st 1970
This explains the "wtf happened in 1971" graphs -- it wasn't the gold standard, it was the simulation we live in stabilizing. The graphs before 1970 aren't real.
Added a full LLM type system and context-free grammar to Python. Includes all typescript types, regexes, and arbitrary grammars. Makes coding programs with LLM calls woven in super easy.
Thanks
@ggerganov
for llama.cpp and
@abetlen
for llama-cpp-python that enable this.
Every few weeks I feel this urge to buy some solar panels
Not to hook up to the grid or put on my roof or anything — in Seattle all our power is already hydro
There is just something intrinsically neat about owning these magic wafers that produce power, I can't explain it
Star Trek technology always seemed ridiculous, but the universal translator suddenly seems plausible: when you encounter an alien ship, just quickly exchange a petabyte of corpus data, train an LLM on the ship supercomputer in a few milliseconds, have your chat
@q_star_shaman
Been doing this for 3 years and empirically this doesn't seem to happen
The typical user is someone who used their last bag on a walk and their dog decided to go a second time
The craziest thing about watching this portrayal of a struggle session during the Cultural Revolution is that I know people in Seattle who would 1000% love to do this to techbros/landlords/hedge funds/whatever
I don't understand why more >$100MM wealthy people don't commission more books
Like you could pay a good indie author a decent salary to write books about whatever you want
I have a ton of story ideas but no time to write, I would commission soooooo many books
Most large software orgs are probably leaving millions of $$ of developer productivity on the table by not hiring 1 guy whose sole responsibility is making the test suite run faster
Nobody gets promoted for this task, so nobody does it — should just make it its own role
First time seeing ChatGPT give me 2 possible responses in this style (I did not press the "regenerate response" button). When did this functionality get added?
I assume it detected I had corrected it multiple times in a row to trigger this mode. Good way to gather RL data.
@knowclarified
I think raw reaction time is probably lower level
That is, for an IQ task, you probably want "speed to put the task into a useful mental framework that you can then use to solve the problem"
And raw processing speed is a subcomponent of that
@levelsio
@Uber
You'll want to set your payment method to cash anyway because a lot of Uber drivers there don't have their app configured to accept card at all (because they want to avoid paying taxes)
You really want your high-agency, talented people near the front lines
Somewhat related, but the more I think about it, the more I think
@sebasbensu
basically nailed it in this post
I wish we lived in a high-trust society
At a ramen shop in Japan a few years ago, rainstorm starts. Old lady owner won't let us leave without giving us an umbrella. Language barrier, but the vibe was "just bring it back tomorrow"
Can you imagine trusting strangers like that?
Solved by getting the LLM to simulate the execution of a program that carefully prints out every state mutation and logic operator
Mapping onto A,B,C,D made it work better due to tokenization (OP said this was fine)
Claude beats GPT for this one
Mandatory clarifications and thoughts:
1. This isn't a tokenizer issue. If you use 1 token per symbol, GPT-4 / Opus / etc. will still fail. Byte-based GPTs fail at this task too. Stop blaming the tokenizer for everything.
2. This tweet is meant to be an answer to the following…
A lot of the appeal of programming is that the computer is always right
There is nobody to blame for problems except yourself
All issues are skill issues — that clarity is nice
Big news hidden in the OpenAI GPT4 public availability announcement: a gpt-3.5-turbo-instruct model! Been wanting this for a long time.
gpt-3.5-turbo is chat-tuned which makes it much harder to use than davinci for tasks that are better structured as completions.
Wanted to update my pfp with my short haircut, but not change the general layout because people recognize you by the thumbnail
Seemed like a perfect application of AI image editing
I tried 5 different popular apps and none came close to
@dingboard_
at ease of use or outcome
Talked to a neuroscience researcher friend who leads a lab here in Seattle.
I knew his team had been using ChatGPT a lot since it was released, but was interesting to catch up now that it's been a few months.
Some notes below about use outside of tech for those curious.
Ok so the whole "captchas are getting harder" thing is kind of funny now, but really what does the internet do in 6-12 months when all captchas are truly solved
Is everyone gonna have to have some proof-of-human private key? Iris scan?
??? what are we gonna do? seems looming
@eriktorenberg
@realGeorgeHotz
has been making an incredible demo of hiring at
@__tinygrad__
by simply posting open-to-the-public, paid bounty tasks and hiring from the pool of successful individuals
What could this model look like for education? Or post-education, or whatever we call it.
Has anyone slapped monte-carlo tree search on top of an LLM yet, with each token selection representing a branch? Use logprobs as the initial node bias and train a discriminator to score the leaf nodes?
Excited to share what I've been working on at
@GetRowZero
for the last 2 years
We've created the world's fastest spreadsheet engine, about 100-1000x faster than Excel or Google Sheets
All using your familiar Excel skills!
Try it at rowzero[dot]io
It will never stop being funny to me that my 35 year old piece of shit Japanese mini truck is better than modern trucks in every way that matters
- 6.5ft bed
- Sides fold down for flatbed
- Easy to maneuver on a jobsite
- Trivial to maintain/repair
- Better fuel economy
The Seattle Fire Department is the
#1
opposition to adding any traffic calming in the city
The number of people who die in car crashes versus fires differs by 100x, so in the grand scheme of things, the fire department is actually net killing people
Cities should deputize neighborhood locals to do stuff like fill potholes
You know how many retired grandpas would love to be the pothole king of their neighborhood?
You don't even need to pay these guys, just supply the materials & offer training
Cheaper & faster
It was a good lesson though — in life you’ll encounter people who want to create organizations/legibility so that they can be on the top of that hierarchy
The benefit of plain N^2 transformers is you correlate *all* the tokens, and won't tend to miss much.
Abandoning N^2 means making heuristic choices about which tokens to correlate. Figuring out a good heuristic is a million (billion?) dollar question.
LongNet: Scaling Transformers to 1,000,000,000 Tokens
paper page:
Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity,…
I'm surprised some pro-natalism billionaire hasn't funded the creation of an actually good and healthy dating app designed to actually get people into committed relationships and off the hedonic treadmill
Probably dating apps just need to be operated as a public service. Absolutely massive amounts of consumer surplus and any attempt to capture nontrivial amounts of that surplus is insanely destructive
@ByrneHobart
The cosine similarity of those two tokens in the latent space is pretty small, easy mistake for a neural network to make early in the training run
GPT4 cannot play chess.
Expected to get a bot up to ~1000 Elo and totally failed -- but negative results are still interesting!
Even given verbose prompting, board state descriptions, CoT, etc, it loses badly to depth=1 engine.
Code & notes below.
Altman always says "we were surprised nobody built ChatGPT first".
Folks made chatbots with GPT3, they just cost a fortune to run, of course nobody released a free website.
Pictured: my friend trying to jailbreak my Twilio SMS GPT3 bot over a year ago
For months I’ve loudly asked for a great conversational assistant. OpenAI finally shipped it.
And, just like ChatGPT, the developer community dropped the ball. There is no secret-sauce here. It’s just Whisper -> GPT -> TTS. Devs had these tools since March
The new model readily beats Stockfish Level 4 (1700) and still loses respectably to Level 5 (2000). Never attempted illegal moves. Used clever opening sacrifice, and incredibly cheeky pawn & king checkmate, allowing the opponent to uselessly promote.
I'm pretty sure the top 25% or so of public school kids would be better off if you just left them alone to read the textbooks and help each other as needed
It turns out textbooks are written by smart people and tend to be much better than the median teacher for certain students
@BorisMPower
Interestingly, in the games it lost again higher rated Stockfish bots, even after GPT made a bad move, it was still able to *predict* Stockfish's move that takes advantage of GPT's blunder. So you could probably get a >2000 Elo GPT by strapping on a tiny bit of search.
@charlieboardman
One of the best business books I have ever read, absolutely packed full of great business/marketing/management lessons, and fascinating history/stories
Since this experience, whenever I make friends with a new doctor, I quiz them to see if they would have correctly diagnosed the issue
Around 50% get it right, and to the ones who get it right, it's obvious and they're astounded anyone with a medical license could get it wrong
The impact AI will have on the field of history is underrated.
AI will read every cuneiform tablet, monastic scroll, 17th century diary, understand every ancient language natively, and be able to synthesize a coherent view of history from every point of view.
@levelsio
@fede_intern
@Uber
Yes all these people are talking about the case where you would have brought USD cash in to exchange for the gray market rate (which is what all locals do)
The ATM rate is about 15% worse than the gray market rate (it used to be like 3x worse a few months ago, but Milei fixed)
@nabeelqu
He had that dawg in him
But really you see this with a lot of the FIRE guys -- many people who have the dedication to build a business and amass enough wealth to retire by mid-30s are also the kind of guy who can't sit still and needs to build
Riffing on this idea a bit, I think you'll see a resurgence of languages more akin to prolog where you codify the properties you want some code to have, and the runtime uses intelligence to figure out the optimal implementation
I have a long-standing prediction that software will be written in increasingly low level languages due to the cost of development and maintenance drops with AI. high level prompts, low level code, best possible performance and customizability.
Every codebase tends to be extended to the point of incomprehensibility
If people can comprehend it, they will extend it, and will keep doing so until it's not comprehensible -- in the long run, all codebases become incomprehensible
Peter Principle but for code
Carpenters if they were like junior software developers:
“I went to add a shelf but found the house is framed with pine instead of cedar, so anyway I tore down the house and am rebuilding it”
In conclusion, I now totally believe
@BorisMPower
's claim about 1800 Elo for GPT4.
I think the RLHF'd chat models do not do this well, but the base/instruct models seem to do much better.
It's a shame there isn't more research put into non-lethal guns
Fundamentally I think people have the right to defend themselves from assault
But I wish there were more options that were just as effective as a gun but less lethal
Why no full-auto self-targeting tasers??
Was a TA for algorithms class. Made an automated grader: student uploads program, run 100 test cases, score is % correct
Once a student got 99%, missed a small edge case
I'd email students to offer hints. He responds "99% is fine, thanks though"
Broke my perfectionist brain
@alexgraveley
It's funny how people have their own "this one thing will teach you everything you need to know" (and I totally believe all of them)
For me, it's "write a disk-backed, log-structured merge tree KV store with REST API" -- this one project will teach you literally everything