blob wave Hi everyone! I'm Eunice from Elementl, the team behind Dagster. If you're interested in getting on our list for swag, please fill out this Google form (1 - 2 min). Thanks for being such a kickass community!

🎉 One more ping for anyone interested in getting on our list for swag (only takes 1 - 2 minutes)! Thank you!

📰 Our eighth Dagster Community Meeting is on Tuesday, July 13 at 9AM PST! Register for an invite here! 🎙️ Presentations • 🤖 @carlson (Thinking Machines) will show off how their data consultancy firm uses Dagster for machine learning workflows. • D We just released Dagster 0.12.0 “Into the Groove” last week, which introduces a new set of experimental APIs that consolidate and improve many Dagster core concepts. @sandy (Dagster core team) will go over these recent improvements to showcase the new elegance and power of these APIs!

📆 Our eighth Dagster Community Meeting is starting in 2 minutes. Join us on Zoom to hear from @carlson at Thinking Machines and about new features in the 0.12.0 release!

📰 The Dagster 0.12.0 blog post is out!   It highlights and elaborates on a few of the more impactful and highly-anticipated features of the release: Pipeline Failure Sensors blob detective*, Solid-Level Retry Policies* 🔃*,* and Easier Unit Tests 🧪. If any of that interests you, check it out here! https://dagster.io/blog/dagster-0-12-0-into-the-groove Thanks as always to all the members of the community who helped make this happen through your feedback and contributions!

📰 Our very own @sandy will be speaking at Subsurface tomorrow on July 22, 2021, at 10:15 AM PST! The conference is geared towards data architects and data engineers looking to build next generation cloud data lakes. If this is of interest to you, register here! His presentation is titled “Dagster: An Orchestrator for the Full Data Lifecycle”. You can check the full details of the talk here.

"Why can't you nest a pipeline inside a pipeline?  What's the difference between a mode and a preset?  What's a solid?" As we look towards a 1.0 release, we want a set of core APIs that will stand the test of time.  In the last couple months, we've collected feedback from users on the least intuitive aspects of Dagster and used that feedback to plan a set of final changes to the core APIs. These changes are significant and impactful, including major renames (for example, 

@solid

 will become 

@op

) , unification of concepts such as modes and presets, and separation of DAG definitions from their resources and default config. Yesterday we published a blog post describing these changes and why we're making them, as well as a migration guide with side-by-side examples.  We believe these APIs, and the corresponding Dagit changes, are a huge step forward for testing, elegance, deployment, and verbosity.  While currently experimental in Dagster 0.12.x, they'll serve as the foundation of Dagster starting in 0.13.0.  We need your feedback to make this awesome. We’d love for you to try them out, and then to tell us how we can improve.

📈 🚀🥂🙏🏻: Over 2,000 members of the Dagster Slack community! Last week we reached a new milestone: we have over 2,000 members in our community! I want to thank everyone for participating in our community and maintaining its positive and supportive atmosphere. I also want to thank the core Dagster team and the rest of the community for doing such an amazing job fielding all the questions in #dagster-support, #dagster-kubernetes, and other channels. We consistently get feedback that this is one of the most responsive and helpful Slack communities. This takes an incredible amount of work and we’re all in your debt. But we’re just getting started and there’s so much more to do. We are actively hiring for a Developer Relations Lead to further invest and grow the community. We consider this a senior role. You will be a critical leader in the project, but you have the ability to shape product roadmap and company trajectory. If you are interested in this position please apply or DM me directly! I might be a bit biased, but it’s a joy to work here and the team is incredibly talented. It’s a fantastic opportunity to have a lot of impact.

📰 Our ninth Dagster Community Meeting is tomorrow, Tuesday, September 14 at 9AM PST! Join us on Zoom! Register for a calendar invite here. 🎙️ Presentations • @Alessandro Marrella from Earnest Research will present on how Earnest Research uses Dagster to power their machine learning pipelines. • @owen from Elementl will present on Dagster logging improvements that allow configuring log handlers at the instance level, as well as logging to the Dagster log using the standard Python logging invocations.

Dagster 0.12.12 introduced a pair of experimental APIs for building Dagster jobs via “software-defined assets”. Dagster is advancing in some big ways. Next week, we’ll release 0.13.0, which lays down a new foundation in the form of the graph, job, and op APIs. But this is only the start of our roadmap. On top of this foundation, we’ve been working on some big new capabilities. One of these is software-defined assets. It revolves around two APIs -

@asset

and

build_assets_job

- that enable constructing Dagster jobs that puts assets at the forefront. A software-defined asset is a description, in code, of how to compute a particular asset - e.g. a table, ML model, or report. It inverts the typical relationship between data products and data pipelines. Instead of defining a graph of ops and recording which assets those ops materialize, you define a set of assets, each of which knows how to compute its contents from upstream assets. Taking a software-defined asset approach has a few main benefits: • Write less code - each asset knows about the assets it depends on, so you don't need to use

@job

/

@pipeline

to wire up dependencies between your ops. • Track cross-job dependencies via asset lineage - Dagit shows you the parents and children of assets, even if they live in different jobs. This helps locate the sources of problems and the consequences of changing or removing an asset. • Learn when you need to take action on an asset - In a unified view, Dagster compares the assets you've defined in code to the assets you've materialized in storage. You can catch when an asset is stale and needs to be regenerated. • dbt-native orchestration - software-defined assets match the mental model of tools like dbt, making it sensible for Dagster to ingest and orchestrate graphs of dbt models. A dbt model is essentially a software-defined asset. For a fuller description of the new APIs, including code examples, take a look the Github discussion. For a fuller example, take a look at this asset-based version of the Hacker News demo jobs. We're very excited about this direction, but it is still early days. We're looking for early design partners to work with us to make this a reality. If you want to take advantage of this approach and help shape the future of the Dagster product, please reach out to either me or Nick directly.

D 👌 pnkdaggy <!channel> We're excited to announce 0.13.0! Dagster 0.13.0 "Get The Party Started" We’re proud to release version 0.13.0 of Dagster. This foundational change in the Dagster ecosystem brings dramatic improvements to our core APIs, and a completely revamped UI and design language. 0.13.0 represents the most significant change in the project since its initial release. New Core APIs The job, op, and graph APIs (first announced in this blog post) now represent the stable core of the system, and replace pipelines, solids, composite solids, modes, and presets as Dagster’s core abstractions. All of Dagster’s documentation - tutorials, examples, table of contents - is in terms of these new core APIs. Pipelines, modes, presets, solids, and composite solids are still supported, but are now considered “Legacy APIs”. We will maintain backcompatibility with the legacy APIs for some time, however, we believe the new APIs represent an elegant foundation for Dagster going forward. As time goes on, we will be adding new features that only apply to the new core. All in all, the new APIs provide increased clarity - they unify related concepts, make testing more lightweight, and simplify operational workflows in Dagit. For comprehensive instructions on how to transition to the new APIs, refer to the migration guide. Improved Python Logging Capabilities You no longer have to pass a context object around to do basic logging. Many updates have been made to our logging system to make it more compatible with the python logging module. You can now capture logs produced by standard python loggers, set a global python log level, and set python log handlers that will be applied to every log message emitted from the Dagster framework. Check out the docs here! Dagit Redesign Dagit has received a complete makeover. This includes a refresh to the color palette and general design patterns, as well as functional changes that make common Dagit workflows more elegant. These changes are designed to go hand in hand with the new set of core APIs to represent a stable core for the system going forward. Attached are some screenshots of the newly designed UI.

<!channel> Hey everybody! We are so thrilled to unveil our new website -- along with @schrockn's blog post announcing Dagster 0.13.0 "Get The Party Started" -- that we couldn't restrain ourselves from spamming the channel. 😜 The elegant new core APIs that will serve as a foundation for Dagster's 1.0 release are now joined by a revamped UI and visual language, and a clear new statement of our mission: To empower every organization to build a productive, scalable data platform. We're so thankful for your trust and participation as members of this community & we couldn't be more excited to be working on these complex and important problems with all of you. Thank you!! 🙏 https://dagster.io/blog/dagster-0-13-0-a-new-foundation

📰 Our tenth Dagster Community Meeting is tomorrow, Tuesday, November 11 at 9AM PT. Join us on Zoom! To register for a calendar invite, fill out this form. 🎙️ Agenda: • 0.13.0 recap - quick summary of Dagster's latest major release. • Coming next - what's coming to Dagster in the near future. • Software-defined assets - next-level orchestration that puts assets at the forefront. As always, we'll have some time for general Q&A. We're looking forward to seeing you there!

📰 We've recently released a new blog post, focusing on the improvements to the Dagster logging system included in 0.13.0! With our most recent work, you can... 🪵 Create Dagster logs without access to a

context

object 📦 Configure instance-wide log handlers for your job runs 🤝 Capture external python logs into the Dagster event stream Check out the post here!

<!channel> elementl + dagsir We’re thrilled to announce Elementl’s Series A. For those of you who don’t know Elementl is the company behind Dagster. Our 14MM USD Series A was led by Index Ventures by Mike Volpi, who is on the board of Elementl, along with Cockroach Labs, Elastic, Confluent, Starburst, and other leading open core companies. We deliberately chose an investor who understand the value of open-source communities, but also has the experience of building awesome companies that can sustain those projects and communities. Joining that investment are top firms such as Sequoia Capital, Slow Ventures, Coatue, Amplify Partners, OSS Capital, Neo, and others. We’ll also be announcing broader early access to Dagster Cloud in the coming weeks. This is important for a couple reasons. One is that we think the service can provide enormous value right now easing operational burden, deployment, and building advanced capabilities targeted to enterprises that are easier and more efficient to develop and deploy centrally. Stay tuned to hear more about that. But more broadly, it is important for every stakeholder to know that there is a durable business that exists that will support Dagster for the long-term. Many of you have placed enormous bets on Dagster, both in terms of your time, careers, and energy, in addition to the infrastructure at your organization. We want to honor that investment with a commitment to building a project and a business so that Dagster fulfills its promise of being a foundational layer for the modern data ecosystem. As part of our rollout we will also be articulating our principles around the relationship between Elementl the company and Dagster the OSS project. Our goal is to build an awesome business, but in a way where we articulate values so that everyone in the community understands the dividing line between the open-source project and our proprietary products. Developers tend to not trust companies that don’t articulate this barrier well, and rightly so. Those companies end up shifting what is open and what is not based on short-term competitive or business pressure. Our goal is to lay out a framework that you believe makes sense, is fair, enables us to build a successful business, but where you can predict our decision-making going forward. Please read more about our funding and the future of the project and company here: https://dagster.io/blog/decade-of-data Here’s my tweetstorm on twitter; https://twitter.com/schrockn/status/1460686324256555009 Please feel free to share broadly!

Hello <!channel>, We’ve known for a long time that for many enterprises, the stability, security, and scale of hosted services are key when adopting open source technologies.  So we knew that in 2021, we would need to build a hosted service to extend the capabilities of open-source Dagster, simplify deployment, and help more organizations manage the complexity of their data platforms. And over the last several months, we’ve been in closed beta with early design partners -- that’s some of you! We’re now very excited to announce the Early Access program for Dagster Cloud. With Cloud, you never have to upgrade a database or worry about backwards compatibility with existing code. You can manage multiple teams in isolated execution environments without heavyweight provisioning. And Cloud comes with enterprise SSO integrations and built-in granular RBAC. We see Dagster Cloud as complementary to open source. Getting big enterprises to run their data platforms on Dagster will only make our community stronger and our project more robust.  For smaller users who don’t want or need the full flexibility and pluggability of our open-source deployment options, Cloud can be a simple way to onboard and get the benefits of Dagster without the burden of infrastructure management -- which will help us grow the open-source community to users who find the existing deployment paths too frictionful. Sign up here for early access. Feel free to DM me or @max directly if you have questions about whether Dagster Cloud might be right for you. We’ve also posted a tweetstorm describing the product. Please like and RT!

👋 Hello everybody! The Dagster team is going to be making a surprise visit to Data Council in Austin, TX on January 26-28, and we'd love to hear from you! Will you be attending? Do you want to come to a Dagster meetup? (Or just hang out in the hallways?) Fill out our (very) quick survey! https://docs.google.com/forms/d/1cNqvh1CYjrTCL0D4BFm1c0NcCDMOLl_J0G5oosI3mto/edit

👋🏻 Hello all! The Dagster Core team is taking a well-deserved break until Jan 3. We’ll be checking slack occasionally to ensure there aren’t any crises, but don’t expect the level of responsiveness you’ve come to expect from the core team until we come back. Thank you all for all your support over the past year! We have an exciting roadmap ahead of us and you’ll also see our team growing as well. Everyone from Elementl and the Dagster Core team wishes you happy holidays and joyful new year. See you in 2022!

Longtime community member @dhume is speaking about Dagster at a Big Data Madison virtual meetup next week! Check it out! https://www.meetup.com/BigDataMadison/events/283167742/

📰 Our eleventh Dagster Community Meeting is tomorrow, Tuesday, February 8 at 9AM PT. Join us on Zoom! To register for a calendar invite, fill out this form. 🎙️ Here's the agenda: • @mrdavidlaing, Staff Data Engineer @ VMware, will talk about about using monitoring to drive the creation of high quality data assets with Dagster. • @owen, Software Engineer @ Elementl, will demo recent work on software-defined assets, showing how they interact with ingest tools (like Fivetran or Airbyte), as well as dbt. As always, we'll have some time for general Q&A.  We're looking forward to seeing you there!

<!channel> 🎉 👌 D 0.14.0 is out! Dagster 0.14.0 “Never Felt Like This Before” We’re proud to release version 0.14.0 of Dagster. This version introduces much more mature version of software-defined assets_,_ new integrations, a new homepage for Dagit, and a wide set of other features and improvements. Software Defined Assets Software-defined assets, which offer a declarative approach to data orchestration on top of Dagster’s core job/op/graph APIs, have matured significantly. Improvements include partitioned assets, a revamped asset details page in Dagit, a cross-repository asset graph view in Dagit, Dagster types on assets, structured metadata on assets, and the ability to materialize ad-hoc selections of assets without defining jobs. Users can expect the APIs to only undergo minor changes before being declared fully stable in Dagster’s next major release. For more information, view the software-defined assets concepts page here. Factory Floor View Dagit has a new homepage, dubbed the “factory floor” view, that provides an overview of recent activity for all the jobs. From it, you can monitor the status of each job’s latest run or quickly re-execute a job. The new timeline view reports the status of all recent runs in a convenient gantt chart. Dagster-Airbyte A new Airbyte integration (dagster-airbyte) allows you to kick off and monitor Airbyte syncs from within Dagster. This contribution includes a resource implementation as well as a pre-built op for this purpose, and we’ve extended this library to support software-defined asset use cases as well. Dagster-Pandera A new Pandera integration (dagster-pandera) allows you to use Pandera’s validation library to wrap dataframe schemas in Dagster types, enabling runtime data validation of Pandas dataframes in Dagster ops/assets. Additionally, Dagit displays Pandera schema information using a new 

TableSchema

 API. Metadata on Dagster Types Dagster Types can now have attached metadata. This allows 

TableSchema

 objects to be attached to Dagster Types via 

TableSchemaMetadata.

 A Dagster Type with a 

TableSchema

 will have the schema rendered in Dagit. Op Selection in Subgraphs Op selection now supports selecting ops inside subgraphs. For example, to select an op 

my_op

 inside a subgraph 

my_graph

, you can now specify the query as 

<http://my_graph.my|my_graph.my>_op

. This is supported in both Dagit and Python APIs. ECSRunLauncher The ECSRunLauncher is no longer considered experimental. You can bootstrap your own Dagster deployment on ECS using our docker compose example or you can use it in conjunction with a managed Dagster Cloud deployment! Asset Observations The new `AssetObservation` event enables recording metadata about an asset without indicating that the asset has been updated. Op Events Without Yield Dagster now supports logging 

AssetMaterialization

, 

ExpectationResult

 and 

AssetObservation

 events via 

OpExecutionContext.log_event

. Previously, Dagster only supported yielding these events within the body of an op. Auto-start Sensors and Schedules Sensors and schedules now include a 

default_status

 parameter. If this parameter is set to 

RUNNING

, sensors and schedules will default to running as soon as they are loaded in your workspace. Some screenshots of the new UI:

We also published this post about the direction of the project. There was a lot of discussion around an article called “The Unbundling of Airflow” and it was in line with a bunch of our thinking, so we decided to engage in the conversation directly. I also tweeted about it. Please like and RT! 🙏🏻🙏🏻🙏🏻

We are excited to invite the Dagster Community out for a great night of community, data, and drinks! Join us on Tuesday, March 22, from 6:00 pm - 9:00 pm for a pre-Data Council 2022 happy hour at Terrace 59 at Speakeasy. You'll have plenty of time to take in the Beautiful Austin downtown views, connect with colleagues who care as much about data as you do and meet the team building Dagster. Space is limited! Please reserve your spot now. We look forward to raising a glass together! 🥳 🎉 dagsir https://www.eventbrite.com/e/dagster-happy-hour-tickets-298727731687

Do you enjoy using Dagster? Join us as a developer advocate! ✨Develop product education materials ✨Represent Dagster in the developer community ✨Engage with users ✨Influence product direction on behalf of the community As the developer advocate, you will create product education materials that will help our users to smoothly onboard to Dagster and succeed in learning about the project and best practices, adapting Dagster to their custom infrastructure, evaluate Dagster for their organization, or anything in between. You will also be one of the early members of the dev rel team and will have the opportunity to help us shape the team as it grows. See full job posting and apply here! Feel free to also ping me directly if you're interested. 🙂

Hey all! We shipped a new Community page on the Dagster website this week! dagster yay It's a place for us to share publicly about upcoming events and feature some of the amazing things you've all built with Dagster. Check it out » dagster.io/community and let us know if you have a project or event you’d like for us to feature there! PS: We’ve got a lot more fun stuff planned for this space in the future too. 👀

📣 New channel alert! Join the #events channel to post about, hear about, or discuss upcoming events in the space. Because we've added it as a default channel going forward, I've retroactively added folks to it; however, feel free to opt out at any time! 🙂 Here's a question to get things going:

📰 The next Dagster Community Meeting is happening Tuesday, April 12 at 9AM PT. Join us on Zoom! To register for a calendar invite, fill out this form. 🎙️ Here's the agenda: • @marcos, K12 Analytics Engineer, will present on how K12 education leverages analytics engineering using Dagster • @Daniel Blinick, Software Engineer @ Immunai, will present on how Immunai uses Dagster to tackle bio-tech data engineering challenges • @yuhan from Elementl will demo a new community space, Dagster GitHub Discussions dagsir ❓ As always, we'll have some time for general Q&A. 😄 We're looking forward to seeing you there!

🎉 Hi all! We’re opening up GitHub Discussions as a public knowledge base complement to our Slack community. This is a great place to explore ideas, share what you’ve made, and discuss best practices / design patterns with the community members and the Dagster core team! Here are some highlights: • excited Companies / Projects using Dagster is a place for folks to showcase how your companies/projects are using Dagster. • dagsir Dagster-related Job Openings is a place for companies looking for people with Dagster experience. • 💬 The How-to examples category aims to collect code snippets, examples, and design discussions for accomplishing specific use cases with Dagster. 📝 Check out this welcome message to explore more categories! We’ll also be surfacing FAQ and valuable discussions in Slack to this pubic space for content discoverability. Hope you’d find it helpful that questions and answers become more publicly searchable and accessible. 🧪 This is something new we are experimenting with. If you’d like to see any changes or have feedback to our GitHub Discussions, please feel free to ping me directly! hat tip

Reminder that the community meeting is tomorrow! 😄 https://dagster.slack.com/archives/CCCR6P2UR/p1649193211954679

For those of you who missed today's Community Meeting, ! 🌈 Additionally, We'd like your feedback! Have the community meetings felt useful to you? Is there something in particular you'd like to see at future community meetings? Let us know!