๐จ New website, new brand, new event-driven workflows. Take it for a spin:
At Prefect, we make complex workflows simpler, not harder. How, you might ask?๐งต
It's our most-requested feature ever... and now the Prefect UI is open-source!ย ๐
๏ฟผ
Runย `prefect server start`ย to get going - it's really that easy.
The open-source Prefect platform just got an upgrade so big, we had to create two new repos just to hold it! ๐๐
Run `prefect server start` to get going - it's really that easy.
Thanks for being part of our community!
We couldn't have reached this milestone without you, and we can't wait to show you what's next.
Happy engineering!
We have Liftoff!
Prefect is launching four key initiatives this quarter:
๐ Expansion
๐ Product
๐ค Partnerships
๐ฌ Community
Join us (and Marvin) to see how we're transforming data engineering.
Introducing simple, success-based pricing for Prefect Cloud!
โ Only successful tasks count; never pay for retries or failures
๐ Every plan includes 10,000 free tasks / month
๐ Automatic discounts as usage scales
Even Marvin's excited about it.
ONE THOUSAND! ๐
It's been less than a year since we launched our Slack community with 20 members and 1 simple rule:
Be nice.
Thank you for making Prefect the best data community around - we can't wait to show you what's coming next.
Join us here:
Curious about
@vuejs
?
In this Friday's livestream,
@____nicholas
will give
@lauralorenzdev
(and you!) an introduction to Vue development by building a brand new tile for our open-source UI's dashboard.
All experience levels are welcome!
Love
@expectgreatdata
?
Wish you could get validation reports in the Prefect UI?
Now anyone can create a native UI integration for any library with only one line of code - just like we did for Great Expectations.
Check out our blog to see how easy it is.
Say ๐ to Orion, our second-generation workflow engine!
It's DAG-free, has an observable rules engine, and delivers an incredible developer experience from test to production.
You might just love your workflows again.
Want to make your complex workflows simpler?
Schedule pure Python using a simple
@flow
decorator. Develop locally, deploy to prod to build, observe, and react to data pipelines.
Get started with workflow orchestration with this simple tutorial:
We're excited to share that we raised $32M from Tiger Global and
@BessemerVP
!
We're so grateful to have such an incredible community of users, advisors, and investors, all working together to build a new kind of workflow system.
Happy engineering!
What a day!
Since launch, Prefect became the
#12
trending repo on GitHub, and reached
#2
among all Python projects!
Thank you for your support as we build a new kind of workflow tool.
Who wants coffee??
To celebrate 1,000 followers, we're giving away 3 tins from our friends
@CompassCoffeeDC
, plus some exclusive Prefect gear!
On 7/22 we'll randomly select one of our followers to win.
To enter, just like this tweet.
๐โ๏ธ๐งข
We have an official Prefect 2.0 release, and it's graduating to beta!
๐ฉโ๐ป Finally, Code as Workflows
๐ Apache 2.0, top to bottom
๐ค Ephemeral API
โฐ๏ธ Streaming use cases
๐ป Brand new UI, included
More on how Prefect 2.0 helps you orchestrate your stack:
We're excited to announce that we acquired Slate Data!
Slate's Chicago-based team of amazing client-facing engineers has joined Prefect's customer success and integrations teams, so our users will benefit from their presence immediately.
Welcome aboard!
We โค๏ธ
@dask_dev
, and you should too!
In our latest post,
@markov_gainz
shows how easy it is to deploy your Prefect flows on a distributed Dask cluster.
(spoiler alert: set two environment variables and you're done)
It's the biggest release we've ever done:
Prefect Server and UI are graduating to standalone open-source projects! ๐พ
Upgrade to 0.13.0 and get four months of performance improvements, features, and UX upgrades with one command:
`prefect server start`
Verified by Twitter? So 2008. Verified by Prefect is the new blue check. Today launches our new Premier Technology Partner Program within our partner ecosystem, read on ๐
We launched the new, open-source Prefect platform exactly one month ago.
Since then, weโve shot through:
โญ๏ธ 2150 GitHub stars
๐ฆ 2180 Twitter followers
๐ค 795 Slack members
Itโs incredible to see the community grow at this rate. Thanks for your support!
You might have noticed that the internet broke last Friday.
...unless you use Prefect Cloud.
Our CTO
@markov_gainz
explains how the Prefect Hybrid Model allows our SaaS users to mitigate failure, even during global internet outages:
๐ Prefect Cloud just got twice as good!
๐ In the last 6 months, the average team doubled its usage, so we're doubling our free tier to keep up.
๐ Every plan now includes 20,000 free runs every month!
๐ค More people than ever can use Prefect Cloud - and we're ready for them!
Prefect Core is now Prefect 1.0! This release is the product of 4 years of invention and refinement. After rigorous testing, we published a 1.0 release candidate and invited feedback from our community a month ago. 1.0 is a rock solid, stable release:
We're hiring!
Check out our open positions, and if you don't see one that's a prefect fit, shoot us an email with your ideal role and we'll see what we can do!
- Head of Growth
- Front end engineer
- Back end engineer
- DevOps engineer
#MozillaLifeboat
Today weโre introducing KV Store: managed metadata for Prefect Cloud flows!
Use it to track useful information across runs:
๐ข the last processed value
๐ค easily share state across flows
โ๏ธ custom configuration
Learn more from the lead developer here:
Prefectโs never gonna let you down.
The newest example from
@dask_dev
:
- runs a Prefect ETL pipeline
- in a Dask cluster
- to make a GIF (because the internet)
Try it yourself with
@mybinderteam
:
Today, we're announcing a scheduler for the internet!
Prefect Scheduler is a free workflow system that supports any execution environment, powered by Prefect Cloud. It's the product we've always wanted to build.
Join the waitlist now!
Do you work for a
๐ startup,
๐ academic institution,
๐ or non-profit?
Weโre running Lighthouse Partner programs for these 3 groups that include free access to Prefect Cloud.
Please email hello
@prefect
.io to apply (startups must be VC-backed and Series A or earlier). Thanks!
Itโs ๐day!
Reply with an open-source project youโve contributed to or RT to thank an open-source contributor, and weโll send you a pizza.
REALLY (til 4pm ET ๐)
Thereโs no catch. Just pies for pi.
Use the form in our bio to send us your address to get your โZA!
We've posted 3 new case studies that you'll have to read to believe:
-
@SymphonyRM
saw efficiency gains of 99%
- Slate Data saw a 1700x increase in productivity
-
@clearcover
saved days over building and testing Airflow DAGs
Check them out at
And just like that, Prefect 2.0 is out of beta! Thanks to all of our contributors, community members and the Prefect team.
Prefect Cloud 2.0 is also generally available today - sign up at to start coordinating your dataflow!
Two years ago, we launched the Prefect Slack Community with 20 people.
Today, we welcomed our ๐10,000th๐ member!
Thank you for joining this amazing community of data engineers. Prefect wouldn't be where it is without such positive and enthusiastic contributors! ๐ฅ
Itโs not everyday you wake up and see your name in Forbes! Amazing companies were listed on Americaโs Best Startup Employers 2022, proud to be named among them.
Weโve just released a HUGE new version of Prefect Cloud:
๐จ enjoy a fully-redesigned UI
๐ visualize flows and run progress with new schematics
๐ค monitor all connected Prefect Agents
๐ต๏ธโโ๏ธ jump straight to error logs from any dashboard
โฐ keep an eye on late runs
Wondering what we've been up to with
@Microsoft
?
Check out the first steps of how we're working together to support our users, and join us at
#MSBuild
to learn more!
Say ๐ to Prefect 0.14.9!
This is a very special release, driven entirely by the Prefect community.
It also includes
@fivetran
's very first workflow integration! Please give them a warm welcome and let us know how we should continue collaborating.
Say your
@dask_dev
cluster was a chicken.
With Prefect 0.12.5's new resource manager API, you can control its entire lifecycle, from ๐ฅ to ๐, from inside your flow.
Pro tip: it also works with non-chickens like Spark clusters, containers, or databases.
Prefect and Airflow share inspiration, but take very different approaches to certain aspects of workflow management.
Weโve put together a document that highlights key differences and why those decisions were made.
Here's an excellent interview of our friend
@mrocklin
by
@unbalancedparen
about what
@dask_dev
is, why itโs awesome, and what other tools leverage it. Prefect even got a shoutout!
Toss a rock, and there's a 30% chance you'll hit a Prefect engineer (so... please don't throw too hard).
We're ecstatic to see more and more industry surveys that reflect Prefect's rapid adoption among mature data organizations.
The new flow run page is here!
In the advanced options, you'll find the new "run config" feature (Prefect 0.14+).
Now you can change your execution environment, configuration, labels, or even infrastructure on a per-run basis.
Weโre excited to be a partner sponsor at
@PyCon
, along with many of our favorite partners!
Looking for a Prefect opportunity? Come chat with
@jlowin
and
@markov_gainz
at the job fair this Sunday from 1-3pm ET!
Excited to share the biggest release of Prefect 2.0 to date. Highlights include:
๐ Flow Run retries
๐ Configurable notifications
๐ฆ Flow packaging and deployment UX improvements
and... drum roll please...
๐ฒ BLOCKS...!
Read the full update here:
"Prefect is a lot of things. We think it is the best possible tool for dataflow automation. We use Prefect for data modeling, ETL, ML, basically anywhere that data automation is needed."
Learn how
@gabcoyne
and Slate use Prefect and
@getdbt
together!
"If you think Airflow is a great tool, you might find that Prefect is an even greater tool."
Awesome to hear our friends
@mrocklin
and
@hugobowne
on with
@TobiasMacey
- thanks for making us your "Pick"!
Thanks to
@Zach_Angell
for the first community addition to our task library: support for AWS Lambda!
Do you want to contribute to an open source project? Would you like a new car?
At least one of those is possible at
Another Thursday, another release! Prefect 2.8.7 is live with
โ๏ธ Tons of engine performance and reliability improvements
๐บ Results, a visual representation of output from your tasks & flows, are live in the UI
๐ Flow run graph now supports subflows + thousands of nodes
We were so excited about the Server + UI launch, we forgot to say ๐ to Prefect 0.13.0!
๐ซ New Server + UI compatibility (of course)
๐ธ Webhook flow storage
๐ซ Cancel in-flight work (on
@dask_dev
)
๐ Improved version-locks for global run-once semantics
Prefect Cloud surged in November โ usage jumped 50% and we crossed โจhalf a billionโจ tasks processed for our users.
Happy engineering, Prefectionists!
This week we're excited to welcome Allyson Lubimir and Mariia Kerimova to the Prefect engineering team!
Watch this space, things are about to move very quickly...
Weโre proud to support
@boss_librarian
โs Digital Citizenship program at
@JATrojans
by donating 15 laptops to the school library in recognition of October being anti-bullying month.
We love the program K.C. runs and the impact it has and look forward to working together more!
Prepare 2 b excited with our latest release:
๐ฅ๏ธ Windows support for 2.0
โจ๏ธ Support for type-checking
โจ Oh, and a new UI ๐คฉ
More on all of that here:
Interested in deploying
@PrefectIO
Server on
@Microsoft
Azure AKS?
Join us this Friday 3/19 at 4ET/1PT for a livestream walkthrough on deploying and upgrading your Prefect Server!
Say ๐ to Prefect Orion 2.0a4!
This release is all about executors:
โคต๏ธ Each subflow can have its own executor
โจ Expanded support for Dask clusters
๐ Lots of CLI improvements
Thanks to our community members that helped prioritize this roadmap!
Say ๐ to Prefect 0.15!
It's a whole new flow run experience:
โจ Brand new flow run CLI
๐ค Subflow result passing
๐ต๏ธ Agentless execution
๐ New Cloud auth patterns
Thanks to all who contributed over many months of work!
Did you know Prefect has two different Python APIs?
The *Functional API* will be familiar to anyone coming from a data science background.
Data engineers may be more comfortable with the *Imperative API*.
Hate choices? Use both!
Our new Interactive API is out of beta.
Now you can leverage the full power of Prefect Cloud's GraphQL API -- without leaving your browser!
๐จโ๐ป Interact with all of your Prefect data
โ Take advantage of GraphQL autocomplete
๐ Easily browse the schema and documentation
We turn 4 this week! We're so excited to be leaving toddlerhood behind as we embark on our next chapter.
Take a look what we accomplished over this past year:
All that ^ and we haven't even had our first day of kindergarten yet!
It's
#InternationalWomensDay
, and no one embodies
#GirlPower
like our COO
@Smoses_Supposes
. She bulldozed a new way of work creating an inspiring culture and will go to extreme heights - not just for a climb - but to make someones wild dream become a Prefect reality. TY Sarah.
Listen up:
@jlowin
was invited on to the
@DataEngPodcast
!
Tune in to hear how we're eliminating negative engineering with best-practice design and awesome technologies like
@dask_dev
.
Interested in learning the basics of MLOps? Prefect is a part of the
@DataTalksClub
ML Zoomcamp!
๐ to join the upcoming live cohort or follow along with the self-paced curriculum.
Thank you to part of our sales organization who teamed up for yesterday's beach cleanup!
While we are laser-focused on delivering value to our community and users, Prefect will always be committed to making the world a better place. ๐
Looking to get started with
@dask_dev
?
@lauralorenzdev
and
@jsignell
from
@saturn_cloud
just posted an excellent beginner's guide to some of the most common questions, like:
๐บ Where to host your cluster?
๐ How many workers?
๐งต How many threads?
Want a chance to win one a Prefect keyboard? We'll be raffling them at our meetups tomorrow night in San Jose and Washington DC!
Sign up โฌ๏ธ
DC:
San Jose:
"You can obtain a highly available, scalable, distributed system that will make the orchestration of your data pipelines for ETL & ML much more enjoyable and will free up your time to work with data and generate value out of it."
Awesome tutorial, Anna!
Collaborators are live on Prefect Cloud 2!
Life is better with friends, and beta users can add up to 3 collaborators to their workspace for free. Give it a go at .
More at
Marvins collaborating pictured courtesy of DALL-E Mini ๐ฆ
Sure, we all love automation. But sometimes you just need a human in the loop.
Here's how easy it is to make your Prefect flows pause for manual approval: