Dagster @dagster Twitter profile | Pikagi

Pikagi

Dagster

@dagster

4,223

Followers

139

Following

300

Media

1,409

Statuses

Ship data pipelines with extraordinary velocity. Dagster+: GitHub: Slack:

☁️

https://t.co/1Q3pxn0mgI

Joined May 2019

Don't wanna be here? Send us removal request.

Pinned Tweet

@dagster

Dagster

21 days

Thanks to everyone who joined us for the launch! Dagster+ redefines data orchestration by: • reducing spend via cost observability • eliminating error-prone point-solutions • enabling data teams to operate autonomously Get started for free here:

Tweet card media

The enterprise orchestration platform that puts developer experience first, with fully serverless or hybrid deployments, native branching, and out-of-the-box CI/CD.

0

0

2

Last Seen Profiles

@PacerAthletics

@Adam_Fenwick

@NOAR_UK

@helepoleo

@spinsteelballs

@alicenting

@Milabug_Kick

@MoneyFreedom12

@grahammctavish

@NigelGBenn

@KishouDayo

@JennyHuosew

@NeoCranium

@luardds

@Crumpiiii

@powyizu

@BrankoGnjatovi3

@Cxc258444637306

@rohit_manas

@fkeverycountry

@jandakembangstw

@bierman_luke

@jakekillerx

@Charlotte_S94

@PassNewHijab

@Darkness1749695

@YumeYuzuha

@MattZaffino12

@wangdrunk

@Proj_Beloved

@YiIMiz

@LuhYello

@fcsdathletics

@forencos_japan

@CoronelDelicia

@ciaomack

@dagster

Dagster

2 years

Can @duckdb replace our cloud data warehouses or data lakes? @floydophone and @s_ryz build a data lake from scratch with @duckdb @ApacheParquet and S3.

Tweet card media

Build a poor man’s data lake from scratch with DuckDB | Dagster Blog

DuckDB is so hot right now. Learn how to build a data lake from dbt using DuckDB for SQL transformations, along with Python, Dagster, and Parquet files.

2

27

132

@dagster

Dagster

2 years

With @coalesceconf just around the corner, we are pleased to share a new Dagster feature: using @duckdb , @getdbt , and @plotlygraphs with Dagster software-defined assets.

Tweet media one

2

6

86

@dagster

Dagster

1 year

Why would somebody fake GitHub stars? Because they influence serious, high-stakes decisions, including which projects get adopted and which startups get funded. But fake ☆s are not all that hard to spot. Here we share a Dagster project to do just that.

Tweet card media

Tracking the Fake GitHub Star Black Market with Dagster, dbt and BigQuery | Dagster Blog

It's easy for an open-source project to buy fake GitHub stars. We share two approaches for detecting them.

4

15

75

@dagster

Dagster

2 years

Introducing Dagster 1.0! 🎉 After 463 releases and with over 200 contributors, we are proud to release Dagster 1.0. Data teams who want to access what’s unique about Dagster now have a stable foundation to build on. Check it out!

Tweet card media

Introducing Dagster 1.0: Hello | Dagster Blog

Announcing Dagster 1.0. - a stable foundation for building the orchestration layer for modern data platforms.

1

11

73

@dagster

Dagster

5 months

One Data engineering team saved $30K a year on their Fivetran bill by swapping out their data platform's database-to-database data movement tasks with Dagster's embedded ELT functionality. On Tuesday, @pdrmnvd will walk us through this approach.

Tweet media one

2

5

58

@dagster

Dagster

2 months

Experience the future of data orchestration with us as we unveil the next generation of Dagster Cloud. New features, more collaboration, trusted data delivery, and cost management await. Mark your calendars! April 17th at 12 PM EST. Register here:

Dagster+ - expect more from your orchestration solution.

Dagster+ redefines the capabilities of the orchestration layer, incorporating data quality, advanced insights, branch deployments with change tracking, and major data reliability/quality features.

3

10

57

@dagster

Dagster

1 year

We are very pleased to announce our Series B. Elementl - the company behind Dagster - has raised an additional $33M in capital to continue building out the open-source solution, the community, and the commercial adoption for Dagster Cloud.

Tweet media one

1

6

54

@dagster

Dagster

2 years

We're excited to share that we are partnering with Dennis Hume (Data Engineer at @getdutchie ), and @corise_ to build a course on #DataEngineering with Dagster! The course launches on August 15th –sign up here: #DataOrchestration

Tweet card media

Uplimit - Data Engineering with Dagster

A well built data platform allows for fast iteration and safe deployments. This course will teach you how to design, build, and maintain a data platform that supports a wide range of data tasks.You...

1

3

51

@dagster

Dagster

10 months

dbt is one of the most commonly used technologies in data transformation. But how you you fully leverage dbt in a modern data pipeline? In our upcoming release - Dagster 1.4 - we will be introducing new capabilities to supercharge your dbt work... Learn more on Aug 2nd 🧵 1/7

Tweet media one

1

6

49

@dagster

Dagster

9 months

While most data engineers working on Dagster are fully conversant with Python, others welcome an intro or maybe a refresher on Python basics. For these folks, we are building out Python primers specific to data engineering. We just published chapters 5 and 6. Here is a recap:

Tweet media one

1

9

48

@dagster

Dagster

2 years

We're big fans of @getdbt . We're even bigger fans when you can seamlessly interleave dbt models with Python and other tools. With software-defined assets, you can: 🌳 Declare data lineage across tools 🕐 Schedule jobs to ensure data assets are fresh

Tweet card media

Orchestrating Python and dbt with Dagster | Dagster Blog

How asset-focused orchestration bridges the gap between some of data's most popular tools.

1

8

46

@dagster

Dagster

3 years

We’re proud to announce 0.13.0 of Dagster. We’ve made dramatic improvements to our core APIs, completely revamped our UI, and brought renewed clarity to our mission.

0

9

46

@dagster

Dagster

9 months

Elementl, the company behind the Dagster project, has been renamed Dagster Labs. In the following blogpost, @schrockn and @floydophone briefly share the thinking behind the change. tl;dr: it's simpler.

Tweet card media

Introducing Dagster Labs | Dagster Blog

In the spirit of simplification, the company formerly known as Elementl is now doing business as Dagster Labs.

2

10

46

@dagster

Dagster

9 months

We should organize a conference... what would we call it? @Octolis_app @zentaskai @snowplow @SQreamTech @krakenfx @pictoryai @chainguard_dev @AirbyteHQ @argoproj

Tweet media one

21

6

44

@dagster

Dagster

2 years

1/ Today is Dagster Day! Find out how Dagster lets you ship data pipelines with extraordinary velocity. We start in 30 minutes, and you can join us here: . We will thread updates here as we go.

Tweet media one

2

20

44

@dagster

Dagster

4 months

Dagster 1.6 is now available. Entitled "Back to Black" it offers many enhancements to the UI including - you guessed it - dark mode. But there is a lot more to this release... 🧵

Tweet media one

1

7

38

@dagster

Dagster

30 days

While we're at it... dlt is now part of our Embedded ELT! Enjoy expanded data ingestion from APIs & systems in a seamless, Pythonic approach that complements Sling's database and file system replication for efficient pipeline development. Learn more:

Tweet card media

Expanding the Dagster Embedded ELT Ecosystem with dltHub for Data Ingestion | Dagster Blog

We now have an officially supported dlt integration.

0

7

39

@dagster

Dagster

2 years

It's integration season 🍁 We've been shipping non-stop this month, and it's only the beginning. Try out our revamped integrations with @AirbyteHQ , @ApacheAirflow , and @duckdb (with @noteable_io and @getdbt coming soon 👀)

Tweet media one

1

5

34

@dagster

Dagster

1 year

As seen on Reddit...

Tweet media one

2

5

33

@dagster

Dagster

2 years

Notice something different? That's right. @dagsterio is now simply @dagster . Many thanks to @du_griff , a true gentleman.

Tweet media one

1

2

30

@dagster

Dagster

2 years

Introducing @floydophone , Elementl's new Head of Engineering.

Tweet card media

My Path to Elementl: Pete Hunt | Dagster Blog

Pete Hunt discusses what caused him to make the leap from Twitter to Elementl.

1

4

30

@dagster

Dagster

1 year

Learn how to prompt ChatGPT to answer technical questions about your documentation! @floydophone shows us how to use Dagster to power a chatbot trained on your latest support docs using @LangChainAI and @OpenAI .

Tweet card media

Build a GitHub Support Bot with GPT3, LangChain, and Python | Dagster Blog

In this tutorial, we tap into the power of OpenAI's ChatGPT to build a GitHub support bot using GPT3, LangChain, and Python.

3

2

30

@dagster

Dagster

2 years

☁️😶‍🌫️☁️ 🙏 Zero-downtime deployments 🙏 Offloaded operations burden 🙏 Enterprise auth and granular permissions ... and more! Introducing Dagster Cloud, our enterprise grade data orchestration platform. Like Vercel, but for your data pipelines 👀

Tweet card media

Introducing Dagster Cloud | Dagster Blog

Dagster Cloud, the enterprise orchestration platform that puts developer experience first, with fully serverless or hybrid deployments, is now here.

2

6

29

@dagster

Dagster

2 years

Together with the community, Dagster is powered by a core team of contributors from Elementl. Today, we're excited to announce our $14MM series A. Read more about our future timeline 👀☁️☁️

Tweet card media

Why Elementl and Dagster: The Decade of Data | Dagster Blog

Announcing our $14M Series A led by Index Ventures, alongside Sequoia Capital, Slow Ventures, Coatue, Amplify Partners, OSS Capital, and others.

0

5

29

@dagster

Dagster

3 months

Viewing data work as asset creation, or "thinking in assets" results in clearer data lineage, easier maintenance, and better transparency in data pipeline development. @tims_tangents walks through this approach in our latest blog post. Read now:

Tweet card media

Thinking in Assets | Dagster Blog

How to develop data pipelines using Software-defined Assets.

0

3

28

@dagster

Dagster

2 years

If you’re looking to learn how to design, build, and maintain a data platform, look no further than Dennis Hume’s 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝗴𝘀𝘁𝗲𝗿 course launching on CoRise on November 21st!

1

2

29

@dagster

Dagster

5 months

Something exciting is coming to the Dagster UI

@iamjoshbraun

Josh Braun

5 months

👀🌙🌃

2

0

19

2

6

29

@dagster

Dagster

11 months

With the launch of the Cloud solution, we revisit our popular "Poor Man's Data Lake" project, switching from local @duckdb to @motherduck . "a huge usability improvement on top of S3 and Parquet, and it’s much easier to collaborate using Motherduck rather than vanilla DuckDB."

Tweet media one

1

2

29

@dagster

Dagster

1 year

We are creating a guide to support data engineers who may be new to Python. Today we publish part 3: Best Practices in Structuring Python Projects. We dive into 9 key best practices, provide examples of folder structure and review the role of key files.

Tweet card media

Best Practices in Structuring Python Projects | Dagster Blog

We cover 9 best practices and examples on structuring your Python projects for collaboration and productivity.

1

5

29

@dagster

Dagster

2 years

this post was made by Big Complexity Gang

Tweet media one

1

1

27

@dagster

Dagster

2 years

What Big Chocolate Chip doesn't want you to know: software-defined cookies (SDC) solves this. Strap yourselves in this 🧵 (1/597)

@sethrosen

Seth Rosen

2 years

If you've been paying attention there is a huge unbundling happening in the chocolate chip cookie space. A thread 1/x

Tweet media one

31

85

839

1

2

27

@dagster

Dagster

1 year

Discover Dagster in this 10 minute overview with @lopp_sean and learn how progressive data engineering teams deliver high quality data assets faster and with greater control.

Tweet card media

Dagster Data Orchestration 10 min walkthrough

Dagster is a cloud-native data orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, ...

www.youtube.com

0

1

25

@dagster

Dagster

3 months

The Dagster Labs team is growing! Here are our team photos from our Jan 2023 vs. Jan 2024 offsites. Welcome to all the new team members. We have big plans for 2024 - come join us! We will be adding more roles as the year rolls on:

1

0

26

@dagster

Dagster

1 year

Building better data analytics pipelines starts with managing the complexity inherent in todays data environments. The complexity resides in the data, but also in the plethora of systems, stakeholders, schedules, versions, and compute environments you have to juggle.

Tweet media one

2

3

26

@dagster

Dagster

3 months

If you're a dbt user, you probably remember the new pricing plans that will come into effect this year. Get ahead of the upcoming pricing changes by learning more on how you can use Dagster to orchestrate your dbt runs Watch our detailed guide here:

Tweet card media

Supercharge dbt: You might not need dbt Cloud!

Are you looking for potential alternatives to dbt Cloud? Join the Dagster team for a unique showcase of new dbt functionality included in Dagster 1.4. which ...

www.youtube.com

1

4

26

@dagster

Dagster

1 month

The Dagster GitHub repo has officially reached 10,000 stars! We appreciate our growing community + the collective effort of the contributors, users, and supporters who believe in making data orchestration more approachable, reliable, and productive. Here's to the next 10K.

Tweet media one

0

3

25

@dagster

Dagster

1 year

Is your team frustrated by the limitations of @ApacheAirflow ? Join us on Feb 8th as we share best practices for smoothly transitioning to Dagster. It's a robust tool that provides a fantastic developer experience and fosters collaboration across teams.

Tweet card media

Dagster Airflow event - Feb 2023

Easily migrate off Apache Airflow and onto Dagster with specific tooling and guides. Deliver high-quality, manageable data pipelines faster. Escape DAGs and imperative programming, and embrace data...

1

8

25

@dagster

Dagster

2 years

Would you like to see Dagster in action? Join us on Thursday, Nov 3rd at 9 AM PST for a live demo of Dagster. @OwenKephart will show how to create a pipeline using software-defined assets, then will hold a live Q&A. Sign up below!

Tweet card media

Dagster

0

0

25

@dagster

Dagster

8 months

Last month @pdrmnvd shared a bird-centric MDS pipeline at MDS Fest '23. Pedram showcased a wide range of free technology for data ingestion, data storage, data transformation, data orchestration, and data visualization. [1/2]

Tweet media one

1

3

24

@dagster

Dagster

10 months

Dagster 1.4 — “Material Girl” — is now live. The release includes dagster-dbt enhancements which we will demo on Aug 2nd. 1.4 also evolves asset materializations, giving you more fine-grained control and observability over when, why, and how your computations run. 1/6

Tweet media one

1

5

25

@dagster

Dagster

4 months

Dagster Pipes and External Assets are quickly gaining adoption and will prove to be a game changer in how people think about data orchestration.

Tweet card media

Introducing Dagster Pipes | Dagster Blog

A new protocol and toolkit for integrating and launching compute into remote execution environments from Dagster.

@pdrmnvd

pedram

4 months

Love seeing the community build cool shit together. Here’s a repo by Philip Orlando showcasing how Dagster and R can work together through a Dagster Pipes integration and retoculate. Gotta bring R to the people.

Tweet media one

1

3

19

0

1

23

@dagster

Dagster

2 years

We have exciting news: Dagster 1.0 and Dagster Cloud will be released on August 9 at 9AM PST / 6PM CEST. Join us for Dagster Day and learn about our promises for API stability in our open-source framework, as well as the launch of our hosted offering.

Tweet card media

Dagster Day: Announcing Dagster 1.0 and Dagster Cloud | Dagster Blog

The release of Dagster 1.0 and the GA launch of Dagster Cloud represent major milestones in the evolution of our orchestration solution.

0

5

23

@dagster

Dagster

2 months

Build generative AI steps into your pipelines with our new dagster-openai integration! You can now use OpenAI's powerful LLMs in your data pipelines for smarter automation, streamlined tasks, and cost-effective insights. Explore the details:

Tweet card media

Integrate OpenAI calls into your pipelines | Dagster Blog

The new dagster-openai integration lets you tap into the power of LLMs in a cost-efficient way.

0

2

23

@dagster

Dagster

1 year

As @DSJayatillake publishes his 3rd and final installment of 'Dabbling with Dagster,' here is a recap, ICYMI. His conclusion? "I wholeheartedly recommend Dagster over Airflow." David is the Head of Data @metaplane and wrote this series independently. And we love it! [1/4]

1

3

23

@dagster

Dagster

1 year

dbt is central to many Dagster projects, so it’s no surprise that we have focused on making @getdbt models easy to integrate into Dagster pipelines to centralize observability and run metadata. And now we are adding Declarative Scheduling for dbt!

Tweet media one

1

2

23

@dagster

Dagster

1 year

Looking to migrate away from Apache Airflow? With Dagster, we provide built-in utilities to help you seamlessly transition away from the legacy platform. Here are four new resources and an upcoming event to help teams migrate to Dagster! 🧵 1/6

Tweet media one

1

1

22

@dagster

Dagster

2 years

Imagine if you had one pane of glass for your data team. Imagine if you could understand the lineage of your data assets, all in one place.

Tweet media one

0

1

22

@dagster

Dagster

1 year

For data engineers new to Python, Python Packages can be a bit of a head-scratcher. @elliot_j_g wrote a handy Python primer as an intro to learning Dagster. In Part 1, he covers modules, packages, __init__.py, pip, and relative/absolute imports.

Tweet card media

Python Packages: a Primer for Data People (part 1 of 2) | Dagster Blog

The foundation of a solid Python project is mastering modules, packages and imports.

1

5

21

@dagster

Dagster

8 months

We will be hosting our Fall Launch Week from October 6th to the 13th. Each day, we will announce and showcase new features and capabilities on the Dagster platform. The theme of Launch Week is "Escaping the Modern Data Trap". Why? [1/3]

Tweet media one

1

3

21

@dagster

Dagster

1 year

📢 Join us Dec 7th at 9 AM PST / 5 PM GMT for a special virtual 𝗗𝗮𝗴𝘀𝘁𝗲𝗿 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆 𝗠𝗲𝗲𝘁𝗶𝗻𝗴. We'll share updates on the Dagster project, news on integrations and partners, new faces on the Elementl team, and hold a live Q&A.

Tweet card media

Dagster Community Day | Dagster

Join us for a live event featuring Dagster project announcements, product demos, and Q&A with the core team.

1

2

20

@dagster

Dagster

3 years

The moment is finally here... 🥁 0.10.0, "Edge of Glory" has been released 🙏 Hope Lady Daga is proud of us ☺️

Tweet card media

Release 0.10.0 The Edge of Glory · dagster-io/dagster

0.10.0 The Edge of Glory Major Changes A native scheduler with support for exactly-once, fault tolerant, timezone-aware scheduling. A new Dagster daemon process has been added to manage your sched...

0

4

19

@dagster

Dagster

2 years

Three main reasons to choose Dagster over Airflow: 1. Dagster is designed for end-to-end productivity 2. Dagster supports a declarative, asset-based approach to orchestration 3. Dagster is cloud- and container-native @s_ryz and @schrockn make the case.

Tweet card media

Dagster vs. Airflow | Dagster Blog

Looking for an Apache Airflow alternative? See why data teams choose Dagster for data orchestration in this five-part Dagster vs Airflow comparison guide.

1

4

20

@dagster

Dagster

10 months

In data pipelines, we often have processes that 'fan-out' - an operation that results in many identical downstream tasks. A pipeline with a 'fan-out' step may require a scale-up of computing power, with each sub-task run in isolation from the others.

Tweet card media

Fan-Out in Data Engineering | Dagster Glossary

Fan-out in data engineering refers to a pipeline design in which one operation is broken into - or results in - many parallel downstream tasks.

1

7

20

@dagster

Dagster

4 years

We’re thrilled to announce a new integration between Dagster and Great Expectations ( @expectgreatdata ). GE enables Dagster users to build data quality checks directly into their pipelines, making it easier to catch data issues early. More on our blog:

Tweet card media

Great Expectations for Dagster | Dagster Blog

We’re thrilled to announce a new integration between Dagster and a fellow open-source project, Great Expectations (GX).

0

3

18

@dagster

Dagster

1 year

The upcoming Dagster 1.3 release will bring major ergonomic improvements to the config and resource systems by using Pydantic for specifying schema and validation.

Tweet card media

Community Memo: Pythonic Config and Resources | Dagster Blog

Major ergonomic improvements are coming to Dagster's config and resources systems, including a Pydantic frontend.

2

1

18

@dagster

Dagster

2 years

Following the release of Dagster 1.0 and the launch of Dagster Cloud, we are delighted to present core concepts, demos, and best practices in #dataengineering at four upcoming conferences!

Tweet media one

1

4

19

@dagster

Dagster

1 year

Would you like more structure in your ML experiment tracking? In under 5 mins, @GusCavanaugh runs us through some best practices in building a ML pipeline, tracking experiments in @MLflow and using @github actions as a CI tool.

Tweet card media

Quick ML Pipeline using MLflow, Dagster, and Github Actions

This is a quick ML pipeline using:- MLflow for experiment tracking- Dagster for orchestration- Github for version control- Github Actions for CIAs I've been ...

www.youtube.com

0

3

19

@dagster

Dagster

4 months

"Upsert" is a basic data engineering operation: update a record in a database or file, or create it if it does not yet exist. And yet, upserting has some nuance when it comes to performance and interpretation. Learn more about upsert:

0

3

19

@dagster

Dagster

3 months

For #dataengineering practitioners, embracing Domain-Specific Languages (DSL) is not just about technical efficiency but also about being able to scale, simplify, standardize, and democratize #data processes. Learn more about DSLs in our latest blog:

Tweet card media

Standardize Pipelines with Domain-Specific Languages | Dagster Blog

By implementing DSLs, data teams can open their data platform to many more users without compromising on standards.

0

3

19

@dagster

Dagster

2 years

Dagster 0.15.0 "Cool for the Summer" has been released! Featuring... 🌟 Software-defined assets are marked fully stable! 👀 A new partitions and backfills experience in Dagit ⤵️ Top-level inputs can be passed to jobs And more! Check out the recap 👇🏼

Tweet card media

Dagster 0.15.0: Cool for the Summer | Dagster Blog

In 0.15.0, software-defined assets are now marked fully stable and are ready for primetime.

1

6

19

@dagster

Dagster

6 months

ngrok's Data Platform runs on #Dagster . Here's how they manage an influx of about 20TB of net new data each month in over 400 Dagster assets and ~170 dbt models. Stack: @apachekafka @getsentry @AirbyteHQ @kubernetesio @ApacheIceberg @dag

Tweet card media

ngrok blog: How ngrok uses Dagster to run our data platform

Explore how ngrok leverages Dagster to orchestrate its data platform, ensuring efficient data management and insights generation.

0

3

19

@dagster

Dagster

2 years

Dagster Cloud launches today! We just posted a submission to @ProductHunt here: - We look forward to the feedback as we release Dagster Cloud to the world!

Tweet card media

Dagster - Product Information, Latest Updates, and Reviews 2024 | Product Hunt

The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

www.producthunt.com

0

1

19

@dagster

Dagster

2 years

To bundle, or not to bundle, that is the question (though you already know our answer) Live coverage of Bundlegate continues Tuesday, March 15th. See the showdown at @AtlanHQ

Tweet media one

2

6

18

@dagster

Dagster

3 years

We're hosting our fourth Dagster Community Meeting, tomorrow, Jan 12, at 9 AM PST (UTC-7)! We have three presentations lined up from the core team, running through some features in our 0.10.0 release "Edge of Glory" 👇🏼👇🏼👇🏼

1

7

18

@dagster

Dagster

4 years

We often get the question "Should I use Dagster or dbt? They both have dependency graphs". We view them as complementary tools. So the answer is "both."

Tweet card media

Dagster and dbt: Better Together | Dagster Blog

People sometimes ask us — should I use Dagster, or should I use dbt? We view Dagster and dbt as complementary technologies, not competing technologies.

0

3

18

@dagster

Dagster

1 year

A software engineer’s commodity is a code change. To be a productive data engineer, you need to master changes: how these affect the program and others on the team. @alex_langenfeld walks us through a practice called “Stacked Diffs” or “Stacked PRs.”

Tweet card media

Getting Stuff Done: a Guide to Productive Software Engineering | Dagster Blog

To be a more productive software engineer you need to master changes, how these affect the program and others on the team.

1

3

18

@dagster

Dagster

4 years

Happy to announce of latest release, 0.7.0. We've made a ton of progress on a bunch of fronts, working with users with real scalability needs.

Dagster 0.7.0: Waiting To Exhale

We at Elementl are pleased to announce version 0.7.0 of Dagster, codenamed “Waiting To Exhale”. Our last release, 0.6.0, expanded Dagster…

1

3

17

@dagster

Dagster

11 months

In this updated tutorial we migrate the Poor Man’s Data Lake away from S3 and Parquet files into a single system. it’s straightforward and we realize all of the benefits of Motherduck without touching our business logic.

Tweet card media

Revisiting the Poor Man’s Data Lake with MotherDuck | Dagster Blog

See how much easier you can collaborate using DuckDB’s high-powered cloud version MotherDuck to build a one-system data lake.

1

2

18

@dagster

Dagster

3 years

Ever wonder how other teams build their data platforms? Join us at 9:00 - 10:00 AM Pacific Time Tuesday, February 9, 2021 to meet Dennis Hume from @Drizly and @kantrn from @geomagical_labs and learn about their production Dagster setups.

Tweet card media

Dagster Monthly Community Meeting Featuring Drizly and Geomagical Labs

Dagster community meeting is on the second Tuesday of every month at 9:00 - 10:00 AM Pacific Time (PT). The next meeting is on 2/9/2021.

www.eventbrite.com

0

6

18

@dagster

Dagster

3 years

Madagnna's new remastered single, 0.11.0 "Lucky Star" has been released! 🌟

Release 0.11.0 Lucky Star · dagster-io/dagster

0.11.0 Lucky Star Major Changes MySQL is now supported as a backend for storages. You can now run your Dagster Instance on top of MySQL instead of Postgres. See the docs for how to configure MySQL...

3

5

17

@dagster

Dagster

2 years

Bordeaux? In this economy?

@_abhisivasailam

Abhi Sivasailam

@_abhisivasailam

2 years

Achievement unlocked: converted 7 execs to @getdbt + @dagsterio + @fivetran in under 2 glasses of 🍷. I'd like my commission in Bordeaux if you don't mind.

6

2

62

0

0

17

@dagster

Dagster

1 year

Weekly Release Highlights: 1.1.11✨ ☝️ One command `dagster dev` to run both UI and daemon in the same process during local dev. 🏎️ Utility to cache compilations from @getdbt Cloud jobs, which allows dbt assets to be loaded faster. 📜 New example for the branching I/O manager.

Tweet media one

3

0

17

@dagster

Dagster

4 months

Learning Dagster opens many opportunities. For example, would you like to be a Mutineer? Here's your chance. - an Australia-based SaaS company specializing in marketing mix modeling - is looking for a Staff Software Eng. with Dagster & dbt experience to…

Tweet card media

Mutinex GrowthOS | Marketing Mix Modeling Platform

Mutinex GrowthOS is the marketing mix modeling platform that helps marketers make better investment decisions

2

0

17

@dagster

Dagster

2 years

👀 See a preview of what we have in store for Dagster Day - join us on August 9 at 9AM PT to learn about our release of Dagster 1.0 and Dagster Cloud!

1

3

16

@dagster

Dagster

21 days

We're LIVE. Meet Dagster+. We'll be covering the event in this thread, but you can join the event here:

Tweet card media

Introducing Dagster+

The Dagster Labs team introduces you to Dagster+, a data orchestration platform that pushes the boundaries of traditional orchestration.With Dagster+ you can...

www.youtube.com

1

3

17

@dagster

Dagster

4 months

Stored procedures are a critical concept data engineers need to master. In this Dagster Glossary entry, we provide an overview and a Python/Postgres example of these precompiled and stored SQL statements and procedural logic.

Tweet card media

Stored Procedure | Dagster Glossary

Stored procedure is a database programming concept that refers to precompiled & stored SQL statements and procedural logic within a database management system.

1

1

17

@dagster

Dagster

21 days

Today's the day. Today, we present to you Dagster+. New capabilities that embed data reliability, accelerate dev cycles, optimize costs + enrich metadata insights await. The virtual event starts in 1 hour. Can't wait to see you there. Join here:

Tweet card media

Dagster+ - expect more from your orchestration solution.

Dagster+ redefines the capabilities of the orchestration layer, incorporating data quality, advanced insights, branch deployments with change tracking, and major data reliability/quality features.

0

6

17

@dagster

Dagster

10 months

We 😍 to see Dagster users riffing on our tutorials and examples. Here, Marco William Silva shows how to run an ETL script on Dagster with software engineering best practices.

1

5

16

@dagster

Dagster

1 year

Are you attending Data Council in Austin in March? Come pre-game with us courtesy of the modern data stack dream team: @BrooklynData , @_hex_tech @HightouchData @AirbyteHQ and yours truly.

Tweet media one

0

3

16

@dagster

Dagster

2 years

Define your data assets and their dependencies in code. We'll keep their materializations in storage up-to-date. With software-defined assets, Dagster now brings a declarative model to data orchestration.

Tweet card media

Introducing Software-Defined Assets | Dagster Blog

Software-Defined Assets are a new abstraction that allows data teams to focus on the end products, not just the individual tasks, in their data pipeline.

1

3

15

@dagster

Dagster

2 years

The Data team at @zephyr_ai is revolutionizing cancer treatment through bioinformatics and predictive analytics. They shared with us their journey of building ML pipelines on Dagster.

Tweet media one

1

2

16

@dagster

Dagster

3 months

Are you done with Airflow? If you’ve ever had to install Kubernetes locally just to test a simple pipeline, or have resorted to the push-to-prod-and-pray method, it’s time to take a look at how Dagster’s configuration and resource systems allow you to develop locally and ship…

Tweet media one

2

2

15

@dagster

Dagster

1 year

We've already supported generating software-defined assets from @getdbt Core. Starting with next week's release, you can generate software-defined assets from dbt Cloud. In one place, understand the lineage of your dbt Cloud models along with your other data assets.

0

2

16

@dagster

Dagster

11 months

If you are at this year's #DataAISummit be sure to catch @s_ryz 's live presentation on "The Future of Data Orchestration: Asset-Based Orchestration" on the 28th @ 1PM. Sandy's demo includes @AirbyteHQ , @getdbt and of course @databricks .

Home | Databricks

www.databricks.com

0

3

16

@dagster

Dagster

1 month

#DataCouncil - see you there! with @motherduck @hashboardhq @the_cube_dev

Tweet media one

0

0

16

@dagster

Dagster

1 year

Dagster offers an integration with @fivetran making it easy to chain a Fivetran sync with upstream or downstream steps in your #ELT or #ETL workflow. By pairing these systems, you gain observability, lineage, and all the benefits of maintainable, testable code.

1

2

16

@dagster

Dagster

3 years

Why Dagster over Airflow? We show the advantages of using Dagster for: 🧪 Developing and testing computations 📦 Deploying and executing pipelines 🔍 Monitoring computations and observing data assets Read our case 👇

Tweet card media

Dagster vs. Airflow | Dagster Blog

Looking for an Apache Airflow alternative? See why data teams choose Dagster for data orchestration in this five-part Dagster vs Airflow comparison guide.

0

6

16

@dagster

Dagster

3 years

Dagster 0.10.0 will be 🔥🔥🔥 In our most recent community meeting, we showcased features in our upcoming release on January 14, 2021 Here’s our sneak preview (with time stamp links!) 👇👇👇

2

1

16

@dagster

Dagster

1 month

Still buzzing from last night's Data Council Pre-Party! Big thanks to our co-hosts @motherduck , @the_cube_dev , @hashboardhq for helping make the event possible + memorable, and to everyone who came, had drinks, talked, + brought their data passion to life. Until next time!

Tweet media one

Tweet media two

Tweet media three

1

1

16

@dagster

Dagster

1 year

Many ETL workflows start off as scripts. After building these scripts, you need to put them into production. You want things like... ⏱️ Event or schedule-based triggers 🔍 Observability into your computations and data and more! Dagster provides all of this out-of-the-box.

Tweet media one

1

2

16

@dagster

Dagster

1 year

We decided to drive Dagster to a 100%-typed public interface. This turned out to be somewhat difficult. Elementl's Sean Mackesey shares lessons learned in introducing typing to a large Python codebase.

Tweet card media

Adding Types to a Large Python Codebase | Dagster Blog

What we learned when we introduced dynamically typed code to a large Python codebase, bringing Dagster's public API to 100% type coverage.

1

5

15

@dagster

Dagster

2 years

Would you like to make use of the full Dagster emoji set in other Slack channels? Well now you can. You will find them all here: Just download the emoji pack, then look for the `add emoji` button at the bottom of Slack's emoji selection window.

Tweet media one

3

1

13

@dagster

Dagster

1 year

Denoise, Discretize, Fragment, Hash, Impute, Memoize, Normalize, Pickle, Prep, Reshape, Shred, Shuffle, Tokenize, Validate... We have developed a new resource that explains 60+ #dataengineering terms and provides working #python code samples for each.

Tweet card media

Data Engineering Glossary and Definitions with code examples

A glossary of data engineering terms including useful code examples.

1

8

15

@dagster

Dagster

11 months

Seafowl lets you deliver query results for data visualizations, dashboards and notebooks in milliseconds. The team at Splitgraph published a neat guide on using Dagster to orchestrate these interactive analytical applications:

Tweet card media

Using Dagster with Seafowl

Import the result of your data pipeline into Seafowl easily using Dagster!

www.splitgraph.com

0

8

15

@dagster

Dagster

3 years

What did @Mapbox get when incrementally adopting Dagster for their address conflation engine? 🕹 A tight dev/test cycle when building new data pipelines 💰 50% savings in testing infrastructure costs Read their story below!

Incrementally Adopting Dagster at Mapbox | Dagster Blog

At Mapbox, we've adopted Dagster without breaking compatibility with our legacy Airflow systems -- and with huge gains to developer productivity.

0

3

13

@dagster

Dagster

2 years

We believe the role of the orchestrator is to ensure that physical assets in the data warehouse match the logical assets that are defined in code. You can build this reality with software-defined assets. @s_ryz explains our approach at @DataCouncilAI !

Tweet card media

Rethinking Orchestration as Reconciliation: Software Defined Assets...

ABOUT THE TALKThis talk discusses software-defined assets, an approach to orchestration and data management that makes it drastically easier to trust and evo...

www.youtube.com

1

1

15

@dagster

Dagster

2 years

New asset collab dropping soon... I wonder where it's backed 🤔📖✨

Tweet media one

0

1

15

@dagster

Dagster

1 year

Level up from running basic ELT/ETL jobs to managing a data platform for your analytics workloads! With Dagster, you can develop locally and deploy to prod with confidence. Join us tomorrow (May 10) for the benefits and practice of orchestrating data analytics.

Tweet media one

4

5

15

@dagster

Dagster

1 year

Level up from running basic ELT/ETL jobs to managing a data platform for your analytics workloads! With Dagster, data teams and stakeholders can collaborate around shared data assets. Join us May 10 as we run through the benefits and practice of orchestrating data analytics.

Tweet media one

1

4

14

@dagster

Dagster

1 month

We switched from a managed ETL provider to our own embedded ELT library within Dagster. The result? We saved $40K, escaped vendor lock-in, and took control of our data processes. Discover how this move can benefit you too in our latest blog:

Tweet card media

Sling Out Your ETL Provider with Embedded ELT | Dagster Blog

How we saved $40k and gained better control over our ingestion steps.

0

2

14