SeattleDataGuy Profile Banner
SeattleDataGuy Profile
SeattleDataGuy

@SeattleDataGuy

22,830
Followers
8,362
Following
1,138
Media
16,174
Statuses

Data Engineer/Data Science Consultant and All Around Data Guy | Angel Investor - Now In Denver

Denver, CO
Joined January 2017
Don't wanna be here? Send us removal request.
@SeattleDataGuy
SeattleDataGuy
5 years
Every group project ever...where would we be without Arya #AryaStark #BranStark #JonSnow #Sansa #GOTS8E3
Tweet media one
20
736
3K
@SeattleDataGuy
SeattleDataGuy
5 months
I created a guide for data engineers, software engineers, data scientists, analysts, and other technical professionals who want to start consulting. It has templates for proposals, decks, and emails as well as links to books, YouTube channels, and videos you should look into…
Tweet media one
4
137
926
@SeattleDataGuy
SeattleDataGuy
2 years
1/ 15 Data Analytics Buzzwords You Need To Know Analytics Engineer = A BI Engineer who uses dbt Data Mesh = We tried an EDW and it was taking too long so we went back to the siloed data approach but now we opened it up to the entire company
13
154
753
@SeattleDataGuy
SeattleDataGuy
6 months
If you work in data, you will have a SQL interview round. It doesn't matter if you're a data scientist or data engineer. SQL is everywhere. Here are some quick tips from beginner to advanced you should know and be able to talk about in a SQL interview. Beginner - Be able to…
Tweet media one
3
116
620
@SeattleDataGuy
SeattleDataGuy
3 months
Data engineering terms you should know. Even if you're not a data engineer! ETL/ELT - Another common concept is ETL/ELTs which letters stand for Extract, Transform, and Load. These patterns can often be built into DAGs but represent a process of pulling data from source systems…
Tweet media one
4
115
566
@SeattleDataGuy
SeattleDataGuy
1 year
I genuinely don’t understand why so many people hate SQL
76
36
564
@SeattleDataGuy
SeattleDataGuy
2 years
Data engineering is fulfilling but here are some harsh truths - You're going to be on a migration project every 2 yrs-ish - A lot of DE work is SQL and not big data frameworks - We tend to be the middle child between SWEs and DS - Reconciling will become your least favorite word
12
47
433
@SeattleDataGuy
SeattleDataGuy
2 years
How To Start Your Next Data Engineering Project 1. Find a data set you like(preferably an API or raw source/no manicured data sets) 2. Write down a quick plan of questions you want to answer 3. Pick a tool for ingestion, storage and display 4. Go
6
86
390
@SeattleDataGuy
SeattleDataGuy
1 year
Everyday is day one
Tweet media one
10
53
382
@SeattleDataGuy
SeattleDataGuy
6 months
Data engineering projects are a great way to show off your skills on your resume. But thinking of a data engineering project idea is hard. So here are 7 great examples and other resources you can use to inspire your next data engineering project. 1. How To Start Your Next Data…
3
63
368
@SeattleDataGuy
SeattleDataGuy
7 years
I died at 'PHP: What if we wanted to make #SQL injection even easier' Thank you @nixcraft #Coding Need to make one for #datascience #funny
Tweet media one
12
261
357
@SeattleDataGuy
SeattleDataGuy
3 months
After you’ve worked as a data engineer or data analyst for a few years you’ll likely realize that there are a whole lot of projects and skills that aren’t covered in a Python or SQL tutorial. You’ll be asked to take on migrations from one data warehouse to another. You’ll need…
50
34
359
@SeattleDataGuy
SeattleDataGuy
1 month
Data engineering is a difficult role to break into. So, I created a 100 Days Of Data Engineering Guide! It involves free videos, articles, projects, and other challenges. Yes, 100 days is NOT enough to become a "full-fledged data engineer." But it'll be a great way for you…
4
55
343
@SeattleDataGuy
SeattleDataGuy
2 months
Data engineering is fulfilling but here are some harsh truths - You're going to be on a migration project every 2 yrs-ish - A lot of DE work is SQL and not big data frameworks(unless its SQL abstracting away a big data framework) - We tend to be the middle child between SWEs…
10
40
306
@SeattleDataGuy
SeattleDataGuy
8 days
Junior Data Engineer Mindset: "The end-user wants their data in real-time for the dashboard so I am going to make sure my data pipeline updates up to the millisecond" Senior+ Data Engineer Mindset: "The end-user wants their data in real-time, so I asked them how often they plan…
5
43
301
@SeattleDataGuy
SeattleDataGuy
2 years
Three certainties of life 1. Death 2. Taxes 3. All data ends up in Excel
9
35
234
@SeattleDataGuy
SeattleDataGuy
5 months
How to annoy a data person. 1. Ask for an ad-hoc data request that is urgent. Then never bring it up again. 2. Assume the data you are asking for is easy to get. It never is. 3. Use the phrase "At my last place this was very easy". Perhaps it was easy at your last job. Also,…
11
52
230
@SeattleDataGuy
SeattleDataGuy
6 months
If you're looking for Black Friday deals on data engineering courses, guess what, you can learn data engineering for free! What's even better is you can do it through building your own project! Here are the basic steps. 1. Find a data source, preferably with an API, you can go…
2
45
217
@SeattleDataGuy
SeattleDataGuy
1 year
@swyx A wise person once told me, legacy just means it works in production
3
12
210
@SeattleDataGuy
SeattleDataGuy
3 months
Data quality is one of the most essential investments you can make when developing your data infrastructure. If you're data is "real-time" but it's wrong, guess what, you're gonna have a bad time. So how do you implement data quality into your pipelines? On a basic level you'll…
Tweet media one
2
34
203
@SeattleDataGuy
SeattleDataGuy
6 months
When leading a data team, it’s far too easy to say yes to everything. Yes, to every adhoc request. Yes, to every dashboard request. Yes, to every automated workflow request. Yes, to every new metric…. But...
Tweet media one
3
32
199
@SeattleDataGuy
SeattleDataGuy
5 months
If you're trying to break into data engineering, then it's easy to get overwhelmed by all the terms, technologies and best practices. That's why many of my Youtube videos are meant to help provide an intro to many of these subjects and technologies. So if you're trying to break…
2
43
190
@SeattleDataGuy
SeattleDataGuy
2 months
SQL is unavoidable if you work in data. In fact @LukeBarousse showed this when he analyzed well over 1 million jobs from indeed(image below)! Even with assistants like ChatGPT, I still need to write a lot of my own SQL. And whether you're a data scientists or data engineer…
Tweet media one
4
25
188
@SeattleDataGuy
SeattleDataGuy
22 days
Data engineering is a difficult role to break into. There are WAY TOO MANY tools, solutions and skills that DEs are expected to know. You don't have to learn everything all at once. But if you are looking to read on some data engineering topics, here are 7 great…
1
34
188
@SeattleDataGuy
SeattleDataGuy
5 months
A common path to becoming a data engineer is to start as a data analyst. This will give you exposure to the data-sets and processes that data engineers build. It will also expose you to: - Working with data warehouses and data lakes - SQL - Automating processes with Python and…
Tweet media one
0
32
186
@SeattleDataGuy
SeattleDataGuy
6 months
Everyone wants a shortcut to becoming a data engineer or data scientist. They want to know what the fewest steps required to make +250k a year. The truth is. There isn't an easy path. Learning to program and write SQL well is hard and it shouldn't be rushed. It's not a…
8
25
166
@SeattleDataGuy
SeattleDataGuy
6 months
Data modeling should start before you write code. Not while you’re already trying to create objects with an ORM. Or while you're writing SQL. In the last two weeks I have seen a lot of scary data modeling situations. I am talking about data in 1st normal form level scary.
3
12
160
@SeattleDataGuy
SeattleDataGuy
25 days
Companies are still expecting data scientists to basically be data engineers...even 10 years after the data science hype started. They'll hire a data scientist or machine learning PhD prior to hiring a data engineer and tell them to research and analyze data that isn't…
Tweet media one
6
23
160
@SeattleDataGuy
SeattleDataGuy
2 months
When I was first learning SQL there were several "AH-HAH" moments that helped take me to the next level. Many of these didn't come from books but instead were necessities because of the problems I was solving at the time. Here are a few of those key lessons. 1. Understanding…
6
11
159
@SeattleDataGuy
SeattleDataGuy
6 months
If you want to get a job as a data engineer, don’t start by learning Airflow or Snowflake. Learning tools and solutions before understanding the basics is like learning a language by learning phrases instead of understanding how to pronounce the characters that makeup said…
3
39
160
@SeattleDataGuy
SeattleDataGuy
2 years
1/If you work in data, you will have a SQL interview round. It doesn't matter if you're a data scientist or data engineer. #SQL is everywhere. Here are some quick tips from beginner to advanced you should know and be able to talk about in a SQL interview.
2
30
154
@SeattleDataGuy
SeattleDataGuy
1 month
I created a guide for data engineers, analysts, managers, and directors who are running data teams(or will be running a data team one day). I have listed some reliable books, articles, templates as well as YouTube videos you should look into as a data lead. Now, there are a…
3
24
154
@SeattleDataGuy
SeattleDataGuy
1 year
Tech Industry 5 years ago: SQL is dead SQL:
3
20
129
@SeattleDataGuy
SeattleDataGuy
2 years
THEY'RE CALLED DATA CONTRACTS THEY'RE API-LIKE AGREEMENTS BETWEEN THE SOFTWARE ENGINEERS WHO OWN THE SERVICES AND THE DATA CONSUMERS THAT RELY ON THEM. IT'LL ALLOW THE SWES TO WORRY LESS ABOUT BREAKING PRODUCTION DATA PIPELINES AND HELP THE DATA TEAM MOVE AWAY FROM FIXING IN SQL
Tweet media one
4
15
124
@SeattleDataGuy
SeattleDataGuy
1 month
If you're working as a data scientist, a data engineer, or a data analyst there are many lessons you probably wish you would have known before you started. Here are some things I wish I knew before I became a data engineer. 1. Don’t let the hype distract you - There are a lot…
6
18
120
@SeattleDataGuy
SeattleDataGuy
1 month
Airflow remains a popular choice when teams need to pick an open-source orchestration solution.But it is often set up the wrong way. Here are a few common mistakes DevOps and Data teams make when they spin up Airflow. 1. The DAG folder is in the same Repo as the main Airflow…
Tweet media one
2
15
122
@SeattleDataGuy
SeattleDataGuy
2 years
In all seriousness.. How many times has SQL come out on top? BigQuery tried to make it's own version of SQL and had to quickly add in standard SQL Hadoop has Presto and Hive Spark has SparkSQL Sorry, your new query language or processing engine will always go back to SQL
9
21
122
@SeattleDataGuy
SeattleDataGuy
5 months
If you work in data, you can't only rely on years of experience to help grow your career. You'll need to put yourself in situations where you can grow exponentially. This means: 1. Finding projects that challenge you at work 2. Looking through code bases from more experienced…
2
25
121
@SeattleDataGuy
SeattleDataGuy
4 months
Running a successful data team is hard. You're often pressured to deliver quickly sometimes feeling as if you aren't given the proper amount of time to consider data modeling or quality. Other times you're having to fend off the 100th "Hey I Have A Quick Ask" request. All of…
1
13
117
@SeattleDataGuy
SeattleDataGuy
3 months
There are many reasons new data teams and strategies fail. Here are a few I have seen over the past few years while I have been consulting. ML First, Analytical Processes Never - I have now had 3-4 discussions with companies that are running towards machine learning, without…
Tweet media one
1
12
115
@SeattleDataGuy
SeattleDataGuy
6 months
I predict that the need for data engineering and data analytics consultants will continue to grow over the next few years. I bet on this trend when I quit Facebook to start my own consulting company. And now I am seeing a growing interest in others who want to go down a similar…
2
8
113
@SeattleDataGuy
SeattleDataGuy
5 months
Companies tend to do things in the wrong order when it comes to data teams. Even 10 years after the data science hype started, most companies still hire a data scientists or machine learning PhDs prior to hiring a data engineer. Then they start to wonder why they aren't getting…
Tweet media one
3
19
109
@SeattleDataGuy
SeattleDataGuy
2 years
Behind the scenes of the "What is a Data Engineer?" article with @GergelyOrosz On the left was my version of the image... On the right was after @Pragmatic_Eng finished editing. Thanks again for the collab!
Tweet media one
Tweet media two
2
12
102
@SeattleDataGuy
SeattleDataGuy
6 months
Interviewing for a data engineering position requires preparing, studying, and long, all-day interviews. This is why I put together a data engineer interview study guide to help keep track of my progress when I was interviewing(shared below).
1
17
99
@SeattleDataGuy
SeattleDataGuy
17 days
There are a lot of "easy fixes" in data that aren't the right solution. They either add unnecessary tech debt or don't really improve the situation. Here are 3. 1. Let's just fix it in SQL - It can seem really easy to fix business logic in the SQL layer rather than from the…
Tweet media one
6
11
98
@SeattleDataGuy
SeattleDataGuy
5 months
A lot of companies out there fail new data scientists and data engineers. I read an article 9 months ago from someone who said they are leaving data science. One of the drivers was poor mentorship. I could relate to this section so strongly. When I first started, I thought I…
Tweet media one
4
15
96
@SeattleDataGuy
SeattleDataGuy
2 years
A meeting every other 30 minutes is the worst kind of torture. You can't really get any technical work done, but its enough time to stress that you should be getting technical work done
7
9
90
@SeattleDataGuy
SeattleDataGuy
2 years
CEO: Quiet quitters and people with two jobs are ruining America Also that same CEO: Leading advisor of 2 start-ups, on the board of 5 companies, running 2 podcasts and writing a book
1
7
91
@SeattleDataGuy
SeattleDataGuy
2 years
7 Words data engineers hate... 1. Backfilling 2. Reconciling 3. Oncall 4. Migration 5. Deadline 6. Adhoc 7. Change-requests
9
8
85
@SeattleDataGuy
SeattleDataGuy
5 months
It can be hard to justify hiring a data engineer. Hiring a data scientist or analyst is generally easier because their work is immediately tangible. Many data scientists and analysts are also great at building SQL and notebooks workflows that can take data from Data Lakes and…
4
13
83
@SeattleDataGuy
SeattleDataGuy
6 months
Learning how to data model using the basic Sales or Orders example star schemas is like learning data science using the IRIS data set. It works great as a toy example. But you'll quickly find out it's not that simple. Data modeling in real life doesn't fit into a perfect…
7
8
84
@SeattleDataGuy
SeattleDataGuy
19 days
Data modeling is key skill data engineers need to know. In particular dimensional modeling. Whether you eventually decide to go into one big table or you just need to go through and think about your data model. You'll likely put together a dimensional model. So here is a…
0
6
80
@SeattleDataGuy
SeattleDataGuy
1 year
1/ How to annoy a data person in 2023. 1. Ask for an ad-hoc data request that is urgent. Then never bring it up again. 2. Ask for the data from a dashboard to be exported into Excel 3. Assume the data you are asking for is easy to get. It never is.
2
13
75
@SeattleDataGuy
SeattleDataGuy
3 months
Learning how to data model using the basic Sales or Orders example star schemas is like learning data science using the IRIS data set. It works great as a toy example. But you'll quickly find out it's not that simple. Data modeling in real life doesn't fit into a perfect…
3
11
75
@SeattleDataGuy
SeattleDataGuy
6 months
Many companies rely on a single data person to manage their entire data infrastructure. Truthfully, sometimes its just a single IT person doing it all! This is not bad per se... But 33% of my projects in the last year came from what I call key person dependency issues. A…
Tweet media one
5
16
73
@SeattleDataGuy
SeattleDataGuy
6 months
1) One of the most common mistakes you’ll make as a data engineer or data scientist early on in your career is not truly understanding the business requirements. The business will come to you and ask for a real-time dashboard.
Tweet media one
1
6
70
@SeattleDataGuy
SeattleDataGuy
14 days
When setting up your team's data infrastructure there unforeseen issues that can arise. Will you pick a tool that's too expensive in the long run? Will it actually have all the features you need? But one thing that gets discussed less is the key person dependency problem. In…
Tweet media one
4
6
68
@SeattleDataGuy
SeattleDataGuy
6 months
There are a lot of "easy fixes" in data that aren't the right solution. They either add unnecessary tech debt or don't really improve the situation. Here are 3. 1. Let's just fix it in SQL - It can seem really easy to fix business logic in the SQL layer rather than from the…
Tweet media one
1
7
66
@SeattleDataGuy
SeattleDataGuy
2 years
2/Data Scientist = a Schrodinger’s cat of: an analyst, or a Stats PhD Reverse ETL = LTE Data Quality Monitoring = A bunch of automated SQL statements Low Code/No Code = Click, click, shoot how do I configure the underlying environment. Forget it, I am going back to Python
3
6
61
@SeattleDataGuy
SeattleDataGuy
2 years
Here are some things I wish I knew before I became a data engineer. 1. Don’t Let the Hype Distract You 2. Create maintainable systems 3. Source Of Truth Is Less Of A Destination And More Of A Process 4. Save Your SQL - You will use it again 5. Don’t Say Yes To Every Request
3
6
63
@SeattleDataGuy
SeattleDataGuy
5 months
Sometimes, no matter how hard you’re trying to get a job the market just isn’t right. Especially if you're trying to get your first job. I recall when I was first trying to get my first job…it took nearly a year and honestly, if it wasn’t for my dad literally pointing to a job…
2
10
62
@SeattleDataGuy
SeattleDataGuy
2 years
What you thought data engineering was going to be: - Building distributed systems - Building real-time pipelines - Optimizing very complex sql queries What it ended up being: - Migrations - Adhoc data requests - Oncall rotations - Cron and SFTP
1
6
63
@SeattleDataGuy
SeattleDataGuy
1 year
Imagine Microsoft not allowing you to download Google Chrome...
3
6
56
@SeattleDataGuy
SeattleDataGuy
1 month
Databricks ended 2023 by pulling in $1.6 billion or more than 50% over the prior year. Perhaps that's why during the last two months my intro to Databricks video has been gaining the most views out of any video. More than any of my data roadmap or cliche popular videos. What are…
6
3
53
@SeattleDataGuy
SeattleDataGuy
6 months
1) If you work on a data science or data engineering team, then cost reduction is likely a major point of discussion. I have managed to save well over 500k for my clients this year alone!
Tweet media one
2
2
54
@SeattleDataGuy
SeattleDataGuy
2 months
In my first year of consulting in 2017, I was lucky and was able to match the income that I was making as an FTE. However, the second year I barely scratched $10k. Far from enough to consider quitting my job. That's when I realized, I needed to learn how to create prospect…
1
3
49
@SeattleDataGuy
SeattleDataGuy
5 months
What does it take to get promoted as a data engineer or data scientist? I have posted about this generally in the past before, but I'd like to do something different! I'd love to hear about your stories. What got you promoted, and what skills, projects, or behaviors were…
0
7
51
@SeattleDataGuy
SeattleDataGuy
6 months
What is expected at different seniority levels of data engineering? Honestly, of any engineering? Here are some points I have collected after working for a few different companies. Also, I would love to hear your thoughts! Junior - You're able to complete tasks assigned to…
Tweet media one
1
7
50
@SeattleDataGuy
SeattleDataGuy
2 years
Someone posted this in the discord... #SQL
Tweet media one
4
4
47
@SeattleDataGuy
SeattleDataGuy
6 months
Are you considering becoming a data engineer? Or maybe you need to get up to speed on data engineering quickly? Here is a free mini-course. You'll learn about DE key concepts and so much more. 1. What Is Data Engineering - Why Is DE Important?
1
9
49
@SeattleDataGuy
SeattleDataGuy
2 years
@jteelms @cokiehasiotis This is more a reference to programmers who make 300k a year but still live with nearly nothing.
@jjirsa
jjirsa
3 years
🤷‍♂️
Tweet media one
0
0
8
1
0
47
@SeattleDataGuy
SeattleDataGuy
2 years
3/Self Service Analytics = Self-service after a data engineer or BIE spends 3 hours writing the query Modern Data Stack = Everything is SQL now, No more excel Source Of Truth = I used to trust this, until I learned about Truth V2
1
5
47
@SeattleDataGuy
SeattleDataGuy
1 year
1/What is expected at different seniority levels of data engineering? Honestly, of any engineering? Here are some points I have collected after working for a few different companies. Also, I would love to hear your thoughts!
1
6
46
@SeattleDataGuy
SeattleDataGuy
2 years
Still true
Tweet media one
3
7
46
@SeattleDataGuy
SeattleDataGuy
2 years
6/Data Lake = File server we just dump all our files in. Hope this brought you a chuckle please fee free to add a few more!
5
4
46
@SeattleDataGuy
SeattleDataGuy
1 year
I didn't get a blue checkmark..but recently I got an orange one!
Tweet media one
3
2
44
@SeattleDataGuy
SeattleDataGuy
2 years
1/If you're working as a data engineer, a data analyst, or a data scientist there are many lessons you probably wish you would have known before you started. Here are some things I wish I knew before I became a data engineer.
1
10
43
@SeattleDataGuy
SeattleDataGuy
2 years
This is the way
Tweet media one
0
1
42
@SeattleDataGuy
SeattleDataGuy
1 year
Over 4,000 unique viewers attended the State Of Data Conference yesterday and at its peak we had 400 concurrent viewers 🎉 There were some definite lessons learned and I look forward to the next one!
2
0
42
@SeattleDataGuy
SeattleDataGuy
2 years
Tips I have for people getting ready to interview for data engineer jobs: - Always be applying, before you're ready - Ask recruiter what you should study - Verbally outline your answer prior to white-boarding - Make your resume good enough, then stop, it's just busy work
1
4
41
@SeattleDataGuy
SeattleDataGuy
1 year
| ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄| Delete more code |________________| \ (•◡•) / \ / —— | | |_ |_
3
1
40
@SeattleDataGuy
SeattleDataGuy
2 years
1/How can you improve your SQL🧵? Here are some quick ways you can take your SQL to the next level. #sql #dataengineering
2
6
38
@SeattleDataGuy
SeattleDataGuy
2 years
Since I know @amitisinvesting you like tracking jobs with foundry. Here is a data engineering role paying 250-300k with multiple references to Foundry $pltr
Tweet media one
4
2
36
@SeattleDataGuy
SeattleDataGuy
2 years
5/Democratizing Data = Let’s give everyone access and see how many competing narratives we can get from the same data Lakehouse = We couldn’t setup a Data Lake so we salvaged whatever we could and told our users it looks now like the previous DWH.
2
4
35
@SeattleDataGuy
SeattleDataGuy
2 years
Just heard the intern I was an intern manager for got hired back to FB for 2022. And the analyst I was working with to become a DE inside at FB became a DE. 😢 They grow up so fast.
2
1
35
@SeattleDataGuy
SeattleDataGuy
2 years
There is no such thing as a "quick" 30 minute call to pick anyone's brain.
3
1
32