For all of you following dagster on Twitter, you will notice we have changed our handle: we are now simply @dagster. The account stays the same, and if you were following @dagsterio you will still be getting all our updates.

📰 Hi all, we created a channel (#faq-read-me-before-posting) that highlights the structure of our Slack workspace. For new users and existing users, it contains some links to get started with Dagster and the Slack. Also, it highlights all of our channels in one place and separates them by category (e.g. Dagster Cloud users, integration specific use-cases, or just general interest). This new channel is meant to be read-only, so please don’t post there! Let us know if you have any questions :daggy-love:

We’ll be highlighting new channels in the upcoming weeks that are dedicated to integrations that we’re actively working on to be fully compatible with the Dagster framework out-of-the-box. In addition to library support, we’ll be providing step-by-step guides to use the key features of the integration in tandem with Dagster. One of these new channels is #dagster-airbyte, with integration development being led by our very own @ben! Here’s some of his recent efforts in making the integration software-defined asset compatible. We can generate the corresponding assets in Dagster as you manage your connections in Airbyte. It’s pure magic ✨ If you’re interested in using this integration, join the channel to receive weekly updates and discuss your use-cases :next-level-daggy:

:alert: Another channel highlight! Many of you have been familiar with orchestration, even before Dagster. For those of you who already working with an existing orchestration platform, like Airflow, we’ve been making incredible strides to make an incremental migration path. You can incrementally adopt Dagster and realize the value and potential of our programming model to manage your data assets. In #dagster-airflow, @Joe has led the development of three methods to help you on this path to incremental migration: 1. ➡️ You can orchestrate Dagster runs from Airflow by using the

DagsterCloudOperator

or

DagsterOperator

operators in your existing Airflow DAGs. 2. 🤯 You can automatically create a Dagster job from an existing Airflow DAG - we’ll infer all the dependencies for you. 3. 👀 You can take advantage of Dagster’s programming model to create robust, testable, and reusable jobs to materialize your data assets - we’ll infer the Airflow DAG for you. Join the channel for updates and to discuss your Airflow migration questions :dagster-spin:

Our next open demo and live Q&A is on Thursday, Join us at 9:00 AM PST for the demo or at 9:30 AM if you just want to join the Q&A. Join us: https://dagster.io/dagster-demo-signup

Many thanks to @David Jayatillake for leading the way in creating a Mastodon server specific for the data community. We are happy to be part of it as @dagster@data-folks.masto.host You can find us there and across the Mastodon network.

🕴️:elementl: ➡️:computercat: 😛lanet-daggy: Hello all! Dagster is a growing and thriving community, and the project is backed by a company, Elementl. Up until now I’ve been the CEO of Elementl. That changes today. @Pete Hunt––who was our Head of Engineering––is assuming the role of CEO, and I am moving to the CTO role. I am incredibly excited about this change. I will get to intensely focus on product, engineering, and evangelism of Dagster knowing that the operations and execution at the company are in Pete’s immensely capable hands. You can expect more activity from me in both the codebase and in Slack! We’ll be announcing this news more broadly next week but we wanted this to be the first public forum to push this out! -- Nick

:dagsir: Hi folks, we will be downgrading our Slack plan on Wednesday, November 9th. This plan change shouldn’t affect most of our day-to-day conversations nor our support workflow, i.e. we will still be using Slack as a main community space to provide more hands-on support and make announcements about events and releases, and we’ll keep using GitHub Discussions as the public knowledge base complement to Slack. What this will affect is that we will only be able to view the most recent 90 days of messages and files. To retain the messages and files for longer than 90 days and keep the discoverability of them, we’re introducing discuss.dagster.io: • It is a place where we sync all the public conversations to in real-time. • You can use the search bar inside the site to find older conversations. • The contents will also be Google-searchable.

We would like to welcome our 5,000th Slack community member!

:next-level-daggy: We’re giving another channel highlight! Recently, we partnered with Noteable on an integration on top of our existing dagstermill infrastucture. You can now manage Noteable notebooks as software-defined assets, and orchestrate their execution in tandem with the other assets in your data platform! @jamie, alongside the Noteable team, led the development of this integration. We just pushed out a blog post about this - check it out! Join the channel #dagster-noteable to receive updates about this integration as well as to discuss your use cases with orchestrating your notebooks 📖

DAGSTER COMMUNITY DAY: DEC 7TH @ 9:00 AM PST There has been a lot going on with Dagster: new features, community activity, partnership integrations, Elementl team expansion…. We are happy to announce a special Dagster community day on Dec 7th to share all the updates. Sign up here and join us online. https://dagster.io/community-day

The Dagster team will be out of office Thursday 11/24 and Friday 11/25 for Thanksgiving. Our Slack responsiveness will be limited until we return to work on Monday 11/28.

:alert: :dbt: Another channel highlight! We recently revamped our dbt integration to support software-defined assets for dbt Cloud. Essentially, we’re bringing parity to our support for dbt Core and dbt Cloud. Read our step-by-step guide to get started! For example, you can now understand your dbt Cloud job’s model dependencies, all within Dagster! And we’re still cooking up more features for this integration, like partition support and the ability to materialize individual models from a dbt Cloud job. Join the channel #dagster-dbt to receive updates about this integration as well as to discuss your use cases with dbt ⚙️

📰 Hi all! We’re starting a newsletter that will highlight developments, content, and events that are happening within our Dagster community. To subscribe, sign up here: https://dagster.io/newsletter-signup We’ll be sending out the first newsletter later today 👀

Our next open demo and live Q&A is on Thursday, Join us at 9:00 AM PST / 5:00 PM GMT for the demo or at 9:30 AM PST / 5:30 PM GMT if you just want to join the Q&A. Join us: https://dagster.io/dagster-demo-signup

We have the final details for next week’s Dagster Community Day. Here are the links for adding the event to your calendar: Add to Google Calendar Add to Microsoft Outlook Add to Office 365 Add to Yahoo Calendar (With iCal file attached below)

Dagster Community Day starts in 10 minutes: https://dagster.io/community-day

<!here> we are live!

📢 Announcing a webinar next week in collaboration with Noteable! Fresh from PyData NYC, we are pleased to bring you this great session on debugging Notebooks: Data engineers waste a lot of time troubleshooting long-running pipelines and know only too well the frustration of minor errors consuming hours of work. In this practical tutorial we will demonstrate an innovative solution for dramatically shortening testing cycles and reducing the number of reruns required, boosting developer/practitioner productivity, and reducing frustration on the team. A productionalized notebook integrated with an orchestration platform provides an excellent balance of reproducibility, flexibility, and intent in a way that will be quickly consumable. This tutorial is valuable to data scientists and data engineers. This setup makes it easy to take notebooks from exploratory to production, but even easier to debug and ensure quality over time. This tutorial will show how you can achieve: • Time-saving in initiating jobs: Allowing users to seamlessly transition an exploratory workflow created within a Noteable notebook, into a productionalized scheduled workflow in Dagster. • Time and Cost Saving for debugging failed runs: Allowing users to immediately dive into a live running notebook at the point of failure, with all of the in-memory state preserved. This saves the users’ time, as well as saves companies’ compute costs by not requiring debugging to re-execute previous steps of the workflow. Join Noteable’s CTO & Co-Founder *@Matthew Seal*and Elementl’s @jamie for this virtual event. https://dagster.io/events/noteable-dagster-dec-2022

If you missed Wednesday’s Community Day event, fret not…. We have an entire recap for you right here: https://dagster.io/events/community-day-dec-2022

❄️ New channel highlight! ❄️ We’re making improvements to our Snowflake integration! Check out the new getting started guide to learn how to: • set up your connection to Snowflake • store assets as tables in Snowflake • load existing tables as SourceAssets • load tables in downstream assets We’ve got lots of updates planned, including improved partitions support and handling more data formats. Join the channel #dagster-snowflake to get updates about new features, ask questions, and provide feedback!

Today’s demo will start in ~ 1 hour (9AM PST/12PM EST/5PM GMT) - join us on the hour for the demo, or jump on ~20 mins after the start if you want to join the live Q&A. https://us02web.zoom.us/j/81542945555?pwd=U3BTeGxFU2szalorNWdXb3hENEhXdz09

The Dagster team will be out of office Friday 12/23, Monday 12/26, and Monday 1/2. Our Slack responsiveness will be limited on those days and the surrounding weekends. Additionally, we’ll have no scheduled Thursday releases on 12/22 and 12/29. We’ll resume weekly releases on Thursday 1/5. Happy Holidays and Happy New Year!

Just a quick note that we will NOT be holding the regular bi-weekly demo on Dec 28th. We will resume the demos on Jan 11th. Thanks,

Happy new year to the Dagster Community! We are planning an event next month (Feb 2023) focused on migrating from Apache Airflow to Dagster. If you have made this transition yourself and would be open to sharing your perspectives with the community, please let me know. We are looking to share best practices, challenges, pitfalls, and how-to tips.

🕴️🏽 Happy Friday y’all! This is long overdue, but we now have a LinkedIn page for Dagster (it’s an affiliate page of our existing Elementl page!) You can finally tag us in LinkedIn posts 🤣 On LinkedIn, we’ll be sharing Dagster content from our core team, and reposting content from our community, and promoting upcoming events (such as our February 2023 event focused on discussing strategies for migrating Apache Airflow to Dagster!) Make sure to give us a follow to keep up to date 🙂

There are two upcoming ML conferences looking for presentation proposals. IF you are using Dagster for ML and would like to collaborate on a submission to talk, please let me know. Submissions for both are due next Sunday (Jan 15th). The conferences are: • MLDM: International Conference on Machine Learning and Data Mining -> July 15-19, 2023 in NYC • MLConf -> March 30, 2023 in NYC

Support & Release Update For Next Week The Dagster team will be going on an offsite next week, from 1/23 until the 1/27. During that time, we will still be responsive on Slack, but it might take us a little longer than usual to reply. Additionally, we'll have no scheduled Thursday release on 1/26. We'll resume weekly releases on Thursday the 2/2. Thank you for your patience!