v Profile Banner
v Profile
v

@iavins

3,898
Followers
1,730
Following
288
Media
3,423
Statuses

breaking databases @tursodatabase . W1 '21 @recursecenter excited about databases, storage engines and message queues

b tree
Joined April 2010
Don't wanna be here? Send us removal request.
Pinned Tweet
@iavins
v
2 years
For the past few months, I have been learning about internals of databases. I found many excellent articles on writing compilers, but I could not find many practical resources for databases. So I wrote one. CaskDB is the project I wish I had started with.
7
73
440
@iavins
v
1 month
fun fact: SQLite is the most deployed and most used database. There are over one trillion (1e12) SQLite databases in active use. It is maintained by three people. They don't allow outside contributions.
59
1K
14K
@iavins
v
8 months
the complexity of building a new web browser from scratch
Tweet media one
173
865
6K
@iavins
v
7 months
I always wondered how adblockers for YouTube worked, this post nicely explains it. Also, the cat-and-mouse chase between YouTube and adblockers. A fascinating read
Tweet media one
55
805
6K
@iavins
v
20 days
CPU during disk I/O
30
173
2K
@iavins
v
1 month
The developers -
Tweet media one
6
32
1K
@iavins
v
1 month
It's pretty much everywhere -
Tweet media one
1
22
1K
@iavins
v
1 month
Open-Source, not Open-Contribution -
Tweet media one
3
32
1K
@iavins
v
1 month
inspired from this fun fact -
@___frye
frye
2 months
here’s a fun fact: almost all modern computer systems depend on a single time zone database that gets updated when local laws change it’s maintained by two people
Tweet media one
Tweet media two
135
2K
32K
2
17
742
@iavins
v
2 years
@LaurelCoons Damn, where can I learn / read more about such patterns?
7
0
563
@iavins
v
1 month
How do they cook? There are over 600 lines of test code for every line of code in SQLite. Tests cover 100% of branches in the library.
Tweet media one
6
23
454
@iavins
v
8 months
Here is a fascinating story of how researchers teamed up with SQLite core developers to make it faster using Bloom filters! Let's also dive into database internals and understand how databases implement joins. 📄Paper: SQLite - Past, Present, and Future (2022)
5
60
369
@iavins
v
8 months
from Drew DeVault's post The reckless, infinite scope of web browsers
5
15
283
@iavins
v
2 years
Tweet media one
7
11
227
@iavins
v
8 months
If you want something better than Bloom filter, here are some alternatives. Found this answer which provides research progress of probabilistic data structures Bloom filter -> Cuckoo filter -> XOR filter
Tweet media one
5
31
226
@iavins
v
6 months
Did you know that if SQLite performs better, then the lifetime of your mobile increases? But how? Let's uncover that from the paper: SQL Statement Logging for Making SQLite Truly Lite. 1/n
1
32
217
@iavins
v
2 months
Are you an influencer? And people are bothering you with your empty Github contributions? Look no further, use Rockstar! Rockstar generates real looking Github contribution graph so that you can focus on your business. See how it looks on my profile:
Tweet media one
18
7
218
@iavins
v
2 months
Day 1 with Devin:
Tweet media one
@glcst
Glauber Costa
2 months
I just fired @iavins and hired Devin in his place. AMA
10
0
54
2
12
205
@iavins
v
17 days
This is an excellent post! @thorstenball neatly explains why text editors don't use string rather use rope, gap buffers etc, why zed uses rope and then how they implemented rope on top of a B+Tree, called SumTree
Tweet media one
@thorstenball
Thorsten Ball
20 days
Here it is: a new Zed Decoded post. It's about the two data structures at the heart of Zed: the Rope & the SumTree.
Tweet media one
8
26
252
1
21
166
@iavins
v
9 months
So, thats how I joined @tursodatabase :) It feels surreal
@glcst
Glauber Costa
9 months
Most of the career advice I see on this platform is not applicable to systems level software (OS, compilers, etc). As I promised a cpl of weeks back, here's some tips from my career on how to develop your own career, including a real life story of how we hired @iavins :…
12
41
279
12
9
156
@iavins
v
1 year
in 2021, Notion sharded their single 'monolithic' db into 32 physical dbs, each with 15 logical shards Their post on process is excellent with a systematic approach to sharding, with schema design, capacity planning, and technical details of migration 🔗
Tweet media one
0
31
155
@iavins
v
7 months
What’s the biggest SQLite database you have seen in the wild?
Tweet media one
7
17
150
@iavins
v
9 months
Andy's Intro to Database Systems now uses @duckdb too
Tweet media one
2
6
143
@iavins
v
1 month
SQLite vs Postgres benchmarks when both run locally
Tweet media one
11
11
140
@iavins
v
9 months
I am excited to share that on September 8th, I'll be unwrapping the fascinating story of SQLite. I am going to present the 'SQLite: Past, Present, and Future' paper at @papersweloveblr 1/2
6
26
140
@iavins
v
2 months
It literally stands for Remote Dictionary Server. Yes. It is a fancy hash table. A very good one.
@ImLunaHey
luna
2 months
Tweet media one
47
28
448
3
3
140
@iavins
v
3 months
TIL PRQL - Pipelined Relational Query Language, pronounced “Prequel”. It compiles to SQL and claims that it makes writing complex SQL queries simple and intuitive
Tweet media one
16
11
136
@iavins
v
8 months
Paper: How does the performance of a graph database such as Neo4j compare to the performance of a relational database such as Postgres? "Postgres outperformed Neo4j in almost every query under various settings" oof
@iavins
v
8 months
I found this neat article which shows how to turn Postgres into a graph database. I wonder how it fares with larger workloads. Are there any benchmarks/blogs where someone replaced Postgres with Neo4j / Dgraph, etc.?
Tweet media one
4
2
46
4
7
134
@iavins
v
8 months
I sat with a friend to read this paper on SQLite, it took us only three hours. That is sufficient to get most of the ideas! 🚀 📄 details of the meetup in the next tweet 1/2
@iavins
v
9 months
I am excited to share that on September 8th, I'll be unwrapping the fascinating story of SQLite. I am going to present the 'SQLite: Past, Present, and Future' paper at @papersweloveblr 1/2
6
26
140
3
10
127
@iavins
v
7 months
Dua Lipa recommending edge servers to reduce p99 latency
6
11
122
@iavins
v
10 months
I can’t believe it’s already been a month since I joined the fantastic team at @tursodatabase I will be working on databases full time now. Super excited 🚀🚀
24
5
119
@iavins
v
8 months
Thank you everyone for the amazing support. Can't believe CaskDB crossed 1000 stars already! Amazed and humbled🤗 Whats CaskDB? It's an educational project which aims to teach you writing a persistent key value store from scratch
Tweet media one
6
11
118
@iavins
v
1 month
SQLite is good enough for 98% of use cases.
@buffys
1 month
what opinion will you defend like this?
Tweet media one
1K
155
2K
10
6
115
@iavins
v
8 months
found this super cool decade old video of Solomon Hyke showing docker to the public for the first time in a five minute lightning talk at a PyCon. and then it changed everything... 📺
2
11
112
@iavins
v
8 months
It is hard, but it is not impossible. All it takes is discipline, determination and dedication. You can do it! I locked myself in a room, and I got this contribution graph in under two minutes using rockstar -
Tweet media one
9
9
112
@iavins
v
1 year
This is amazing! A highly visual guide to understand how our code allocates and frees memory by @samwhoo 🔗
Tweet media one
1
25
109
@iavins
v
8 months
Paper: Cuckoo filters, an alternative to Bloom filters, which improve upon in three ways 1. support for deleting items dynamically 2. better lookup performance and 3. better space efficiency for applications requiring low false positive rates(ε < 3%) 📄
Tweet media one
2
19
105
@iavins
v
2 years
@b0rk - Designing Data-Intensive Applications by @martinkl - Database Internals by @ifesdjeen - Writing An Interpreter/Compiler In Go by @thorstenball - Algorithms Design Manual by @StevenSkiena
0
9
98
@iavins
v
14 days
Now there is a whole page explaining why SQLite uses Bytecode
Tweet media one
@iavins
v
15 days
DRH explaining why SQLite compiles SQL to byte code 👇
1
0
31
1
14
99
@iavins
v
9 months
Weekend paper recommendation: SQLite: Past, Present, and Future 📄
@iavins
v
9 months
I am excited to share that on September 8th, I'll be unwrapping the fascinating story of SQLite. I am going to present the 'SQLite: Past, Present, and Future' paper at @papersweloveblr 1/2
6
26
140
4
16
97
@iavins
v
3 months
My patch with the fixes is now merged! This is my first contribution to the libsql core :)
@glcst
Glauber Costa
3 months
C is not just a bad language. It is a demonic instrument, that leaves scars on your soul that not even time can heal. We had a small bug recently where one extension that works with SQLite stopped working with libSQL. One of our engineers, @iavins , spent a whole day debugging…
Tweet media one
123
228
2K
7
0
95
@iavins
v
1 year
Postgres is too posh for Sudha Murty, so she uses this instead:
Tweet media one
3
3
91
@iavins
v
1 month
Yes, they did have a Christian Code of Conduct adapted from The Rule of St. Benedict. But it was controversial and they modified it after two years, to Code of Ethics
1
2
92
@iavins
v
2 months
wrote my first sqlite extension: sqlite> .load ./greet sqlite> select greet('World'); Hellooo, World!
Tweet media one
2
2
92
@iavins
v
1 month
okay, no Soundcloud, but follow me on WhatsApp / Telegram for more database fun facts Telegram - WhatsApp -
2
3
92
@iavins
v
3 months
Tweet media one
@glcst
Glauber Costa
3 months
C is not just a bad language. It is a demonic instrument, that leaves scars on your soul that not even time can heal. We had a small bug recently where one extension that works with SQLite stopped working with libSQL. One of our engineers, @iavins , spent a whole day debugging…
Tweet media one
123
228
2K
0
3
89
@iavins
v
7 months
💀
Tweet media one
6
6
84
@iavins
v
2 months
To write a database from scratch, first, you must write a compiler from scratch.
@eatonphil
Phil Eaton
2 months
Writing your own compiler is awesome Writing your own database is also awesome
37
34
625
6
0
84
@iavins
v
8 months
cursed TIL
Tweet media one
7
0
84
@iavins
v
1 year
TIL T-Tree 📄 A Study of Index Structures for Main Memory Database Management Systems (1986) -
Tweet media one
1
9
81
@iavins
v
9 months
This blog post by matklad has some excellent resources on learning / programming
2
14
80
@iavins
v
7 months
TIL @penberg ! 📄The Impact of Thread-Per-Core Architecture on Application Tail Latency -
Tweet media one
@withoutboats
srrrse
8 months
“Thread per core” arguments conflate two stories: - I have carefully optimized my tasks to guarantee balanced work among threads and therefore can avoid the synchronization costs of work stealing - wahh wahh wahh sharing state in a thread safe way is hard I don’t wanna
6
8
174
2
10
78
@iavins
v
8 months
Tweet media one
3
3
78
@iavins
v
7 months
Tweet media one
3
5
75
@iavins
v
1 year
I tried to learn C earlier but couldn’t make progress. C is everywhere, from systems stuff to databases. SQLite is in C, Postgres is in C! I am picking it up again and hoping to understand the internals. If you have any advice, do share!
Tweet media one
28
3
75
@iavins
v
6 months
I would like to understand write-amplification in B Tree vs LSM Tree. Is there any survey/research paper explaining the same? or a blog post? I found a few articles online, but none covered all the aspects and missed nuance. e.g. batching in LSM tree, WAL in B tree etc.
9
11
70
@iavins
v
7 months
Why does C use the & for pointers?
Tweet media one
5
13
69
@iavins
v
2 months
Just use a fridge. It's the best database, ever.
Tweet media one
@yacineMTB
kache
2 months
you should not be allowed to use a database if you can't build one from scratch
121
26
768
2
8
67
@iavins
v
2 years
Check this programming challenge site where you will build a TCP server - nice.
2
12
67
@iavins
v
7 months
Pointers in Python, because why not?
Tweet media one
11
7
66
@iavins
v
8 months
oof this tweet blew up. I don't have Soundcloud, but you may find my project interesting where I am trying to build a database from scratch github -
@iavins
v
2 years
For the past few months, I have been learning about internals of databases. I found many excellent articles on writing compilers, but I could not find many practical resources for databases. So I wrote one. CaskDB is the project I wish I had started with.
7
73
440
1
2
64
@iavins
v
2 years
CMU Database Talks is back with a great line up. Excited about @neondatabase , @jorandirkgreef , @benbjohnson , and SplinterDB talks!
Tweet media one
2
11
65
@iavins
v
8 months
One more alternative to Bloom filter: Split Block Bloom Filter 📄 Best part: you can read TigerBeetle’s open source implementation 👇
@jorandirkgreef
Joran Dirk Greef
8 months
@iavins If you want something that’s faster to construct than XOR / Binary Fuse Filters (e.g. if write-intensive), and simple to intuit/implement as a middle ground, plus significantly faster than Bloom Filters… …then take a look at the Split Block Bloom Filter in Apache Impala, which…
2
8
56
0
6
65
@iavins
v
8 months
Next paper - 📄 Looking Ahead Makes Query Plans Robust Query optimisation is an NP-Hard problem. So, people find ways to make query execution better. In this paper, the researchers introduce Lookahead Information Passing where they use Bloom Filters for faster execution 1/3
@iavins
v
8 months
I sat with a friend to read this paper on SQLite, it took us only three hours. That is sufficient to get most of the ideas! 🚀 📄 details of the meetup in the next tweet 1/2
3
10
127
2
6
61
@iavins
v
1 year
I am trying out Fly's distributed systems coding challenge. But my code was so bad that it crashed Jepsen itself 💀 But hey, achievement unlocked I guess? When Jepsen breaks, it also says sorry politely.
Tweet media one
0
2
63
@iavins
v
9 months
I am never bored doing Rust. I am either busy fighting the borrow checker or reading about some new drama everyday🍿
7
4
63
@iavins
v
2 years
@largedatabank @phil_eaton @borjasotomayor @justinjaffray These are great! Some more top of my head: - Database Internals by @ifesdjeen - Let's Build a Simple Database - This is classic. Though incomplete, still a great resource - Architecture of a Database System by Stonebraker - (1/2)
2
14
61
@iavins
v
8 months
Those are just extremes. Due to the scaling difficulties of Postgres/MySQL, people made Citus/Vitess. Also, many NoSQL databases today are ACID compliant. They aren’t mutually exclusive, NoSQL + ACID is perfectly fine.
@arpit_bhayani
Arpit Bhayani
8 months
Run away from system design instructors who say, "Use a NoSQL database because SQL doesn't scale" 🤦‍♂️ It shows the sheer immaturity and lack of practical experience of the instructor because crude generalization doesn't exist in computer science and software engineering. Always…
26
24
372
5
3
62
@iavins
v
1 year
A few days ago, @penberg posted about the Hekaton MVCC paper. I hit a wall implementing it. I tried my best to figure it out and kept doubting myself because I did not know that papers could have errors! Later, I confirmed the error with the authors:
4
5
61
@iavins
v
1 year
I enjoyed reading this post by Philip O’Toole, which goes through the journey, design and implementation of rqlite (which came out nine years ago!) It details consensus algorithm upgrades, scaling read performance, reducing disk usage etc.
Tweet media one
1
5
59
@iavins
v
12 days
John Cena being long on databases 👇
@JohnCena
John Cena
12 days
Strive for more relational and less transactional.
809
3K
13K
2
4
58
@iavins
v
8 days
TIL Verus > Verus is a tool for verifying the correctness of code written in Rust. Developers write specs of what their code should do, and Verus statically checks that the executable Rust code will always satisfy the specifications for all possible executions of the code
Tweet media one
2
8
56
@iavins
v
7 months
Reminder that P99 Conf is happening today and tomorrow. They have some excellent talks lined up. Looking forward to the talks by @sarna_dev , @glcst , @gwenshap , and matklad ( @TigerBeetleDB )🚀
6
7
55
@iavins
v
3 years
@PierreZ @therealdatabass @ifesdjeen @intensivedata @martinkl @CMUDB @ifesdjeen 's book is just too good 💯💯💯 I would also recommend Berkeley's CS 186. It also has lecture videos (recorded in 2018). The assignments are open, unlike CMU's course. Also, the assignment uses Java, more approachable than C++, imo.
4
11
55
@iavins
v
1 year
This is an excellent and informative post by @boyter on how to start a Go project in 2023 Covers initial setup, testing, profiling, linters and other tooling:
1
6
53
@iavins
v
1 year
Noria is one of my fav papers. The idea of partial data flow for caching is simply mind blowing. Don’t miss this talk! (starts in 40 mins)
@TigerBeetleDB
TigerBeetle
1 year
Next up in Systems Distributed premieres: join @alanamarzoev and TigerBeetle live in the chat as Alana shares what makes caching so hard in application development! Monday April 17th at 4PM UTC
Tweet media one
0
10
40
1
5
53
@iavins
v
1 year
I released a Go version of CaskDB 🥞
@iavins
v
2 years
For the past few months, I have been learning about internals of databases. I found many excellent articles on writing compilers, but I could not find many practical resources for databases. So I wrote one. CaskDB is the project I wish I had started with.
7
73
440
1
3
52
@iavins
v
9 months
This fantastic paper covers the historical perspective, how it fares with analytics workloads, comparisons with @duckdb and some super cool tricks to make SQLite faster. Join me! - 2/2
3
6
53
@iavins
v
1 year
one thing I enjoy about programming is writing long comment explanations with ascii art (from caskdb)
Tweet media one
2
1
51
@iavins
v
2 years
I watched Andy's lectures on Transactions and found this crazy story of how a bitcoin exchange lost 896 BTC in a day because they used Mongo, which did not support transactions (at that time)! They shut down soon after 😲😲
Tweet media one
3
7
50
@iavins
v
4 months
Is it possible to write deterministic software in languages with automatic GC? e.g. Go
13
5
50
@iavins
v
8 months
Thank you to everyone who showed up, that too on the Friday eve. I had great fun discussing with you all. I met many amazing people and had great discussions; please DM / email me to continue the conversation. This was my first ever public talk; any feedback is welcome.
@iavins
v
9 months
I am excited to share that on September 8th, I'll be unwrapping the fascinating story of SQLite. I am going to present the 'SQLite: Past, Present, and Future' paper at @papersweloveblr 1/2
6
26
140
4
1
47
@iavins
v
7 months
Here is a reminder from Andy to take databases seriously or else you will end up in jail :p
2
3
49
@iavins
v
3 months
TIL which converts C gibberish to English
Tweet media one
5
3
49
@iavins
v
8 months
Found this excellent visualisation and explainer of probabilistic filters:
Tweet media one
1
6
47
@iavins
v
8 months
I found this neat article which shows how to turn Postgres into a graph database. I wonder how it fares with larger workloads. Are there any benchmarks/blogs where someone replaced Postgres with Neo4j / Dgraph, etc.?
Tweet media one
4
2
46
@iavins
v
9 months
SQLite 1.0 was released 23 years ago, on 17th August 🥳
2
3
46
@iavins
v
10 days
Tweet media one
@zack_overflow
zack (in SF)
12 days
Whenever someone says they think Rust is a very low-level language I show them this:
Tweet media one
136
68
1K
2
3
46
@iavins
v
4 months
Language I dislike: Rust Language I begrudgingly respect: Rust Language I think is overrated: Rust Language I think is underrated: Rust Language I love: Rust Language I hate: Rust Language I dream of writing in: Rust
@penberg
Pekka Enberg
5 months
Language I dislike: Bash Language I begrudgingly respect: COBOL Language I think is overrated: Brainfuck Language I think is underrated: Java Language I like: Rust Language I love: C Language I dream of writing in: My own (some day)
7
1
35
4
3
45
@iavins
v
2 months
The MySQL video series by Aaron is one of the best out there. Consider hiring him:
@aarondfrancis
Aaron Francis
2 months
I no longer have a job! I was part of a big layoff at @PlanetScale today. I'll miss working there, it was a great experience. Feeling bummed, kinda embarrassed, but also slightly optimistic? Trying to figure out what's next. I'd love to hear any ideas ❤️
588
146
3K
1
3
44
@iavins
v
2 months
TIL Go can only read upto 1GB per Read call
Tweet media one
2
2
44
@iavins
v
2 months
This happened :( If any journalists are working on a story about AI taking human jobs, I am open to discussing the matter.
@glcst
Glauber Costa
2 months
I just fired @iavins and hired Devin in his place. AMA
10
0
54
6
0
44
@iavins
v
3 months
The book has taught me so much and I still learn something new every time. The perfect book for a database fan boi like me. I wish more and more such books come out and databases to get same love as compilers
@ifesdjeen
Alex P
3 months
When I wrote Database Internals back in ~2018, my main goal was to make the field more approachable and less intimidating. Everyone should be welcome to enter, there's so much work to do here! Over the last year, so many people have read it. I've seen at least 3 book reading…
8
73
583
2
2
42
@iavins
v
10 days
Rust made me realise that progamming is not easy.
@rakyll
Jaana Dogan ヤナ ドガン
11 days
C++ is complex because it doesn't lie about the complexity of the world.
42
55
723
5
0
43
@iavins
v
1 year
Writing a Simple Garbage Collector in C This seems approachable, covers implementing a memory allocator like malloc, and mark and sweep garbage collector. For simplicity, it is stop the world kind, meaning your code halt while the gc is running 🔗
@iavins
v
1 year
The 2nd ed of The Garbage Collection Handbook: Art of Automatic Memory Management is available to preorder. I have yet to read the first edition (1996), but some people have recommended this to me If writing a garbage collector is your jam, then get it!
0
3
20
0
2
43
@iavins
v
7 months
a tip about Little's law
Tweet media one
@abhi9u
Abhinav Upadhyay
7 months
While I was writing the GPU article, I had to review Little's law which comes from queueing theory. I came across an old book which explains queuing theory in the context of computer systems. it is not too math heavy, you can read through it without too many equations. As it's…
5
24
258
2
5
42
@iavins
v
18 days
This tweet was reviewed by @ThePrimeagen lol
Tweet media one
@iavins
v
1 month
fun fact: SQLite is the most deployed and most used database. There are over one trillion (1e12) SQLite databases in active use. It is maintained by three people. They don't allow outside contributions.
59
1K
14K
4
1
42