Many thanks to @David Jayatillake for leading the way in creating a Mastodon server specific for the data community. We are happy to be part of it as @dagster@data-folks.masto.host You can find us there and across the Mastodon network.

🕴️:elementl: ➡️:computercat: 😛lanet-daggy: Hello all! Dagster is a growing and thriving community, and the project is backed by a company, Elementl. Up until now I’ve been the CEO of Elementl. That changes today. @Pete Hunt––who was our Head of Engineering––is assuming the role of CEO, and I am moving to the CTO role. I am incredibly excited about this change. I will get to intensely focus on product, engineering, and evangelism of Dagster knowing that the operations and execution at the company are in Pete’s immensely capable hands. You can expect more activity from me in both the codebase and in Slack! We’ll be announcing this news more broadly next week but we wanted this to be the first public forum to push this out! -- Nick

:dagsir: Hi folks, we will be downgrading our Slack plan on Wednesday, November 9th. This plan change shouldn’t affect most of our day-to-day conversations nor our support workflow, i.e. we will still be using Slack as a main community space to provide more hands-on support and make announcements about events and releases, and we’ll keep using GitHub Discussions as the public knowledge base complement to Slack. What this will affect is that we will only be able to view the most recent 90 days of messages and files. To retain the messages and files for longer than 90 days and keep the discoverability of them, we’re introducing discuss.dagster.io: • It is a place where we sync all the public conversations to in real-time. • You can use the search bar inside the site to find older conversations. • The contents will also be Google-searchable.

We would like to welcome our 5,000th Slack community member!

:next-level-daggy: We’re giving another channel highlight! Recently, we partnered with Noteable on an integration on top of our existing dagstermill infrastucture. You can now manage Noteable notebooks as software-defined assets, and orchestrate their execution in tandem with the other assets in your data platform! @jamie, alongside the Noteable team, led the development of this integration. We just pushed out a blog post about this - check it out! Join the channel #dagster-noteable to receive updates about this integration as well as to discuss your use cases with orchestrating your notebooks 📖

DAGSTER COMMUNITY DAY: DEC 7TH @ 9:00 AM PST There has been a lot going on with Dagster: new features, community activity, partnership integrations, Elementl team expansion…. We are happy to announce a special Dagster community day on Dec 7th to share all the updates. Sign up here and join us online. https://dagster.io/community-day

The Dagster team will be out of office Thursday 11/24 and Friday 11/25 for Thanksgiving. Our Slack responsiveness will be limited until we return to work on Monday 11/28.

:alert: :dbt: Another channel highlight! We recently revamped our dbt integration to support software-defined assets for dbt Cloud. Essentially, we’re bringing parity to our support for dbt Core and dbt Cloud. Read our step-by-step guide to get started! For example, you can now understand your dbt Cloud job’s model dependencies, all within Dagster! And we’re still cooking up more features for this integration, like partition support and the ability to materialize individual models from a dbt Cloud job. Join the channel #dagster-dbt to receive updates about this integration as well as to discuss your use cases with dbt ⚙️

📰 Hi all! We’re starting a newsletter that will highlight developments, content, and events that are happening within our Dagster community. To subscribe, sign up here: https://dagster.io/newsletter-signup We’ll be sending out the first newsletter later today 👀

Our next open demo and live Q&A is on Thursday, Join us at 9:00 AM PST / 5:00 PM GMT for the demo or at 9:30 AM PST / 5:30 PM GMT if you just want to join the Q&A. Join us: https://dagster.io/dagster-demo-signup

We have the final details for next week’s Dagster Community Day. Here are the links for adding the event to your calendar: Add to Google Calendar Add to Microsoft Outlook Add to Office 365 Add to Yahoo Calendar (With iCal file attached below)

Dagster Community Day starts in 10 minutes: https://dagster.io/community-day

<!here> we are live!

📢 Announcing a webinar next week in collaboration with Noteable! Fresh from PyData NYC, we are pleased to bring you this great session on debugging Notebooks: Data engineers waste a lot of time troubleshooting long-running pipelines and know only too well the frustration of minor errors consuming hours of work. In this practical tutorial we will demonstrate an innovative solution for dramatically shortening testing cycles and reducing the number of reruns required, boosting developer/practitioner productivity, and reducing frustration on the team. A productionalized notebook integrated with an orchestration platform provides an excellent balance of reproducibility, flexibility, and intent in a way that will be quickly consumable. This tutorial is valuable to data scientists and data engineers. This setup makes it easy to take notebooks from exploratory to production, but even easier to debug and ensure quality over time. This tutorial will show how you can achieve: • Time-saving in initiating jobs: Allowing users to seamlessly transition an exploratory workflow created within a Noteable notebook, into a productionalized scheduled workflow in Dagster. • Time and Cost Saving for debugging failed runs: Allowing users to immediately dive into a live running notebook at the point of failure, with all of the in-memory state preserved. This saves the users’ time, as well as saves companies’ compute costs by not requiring debugging to re-execute previous steps of the workflow. Join Noteable’s CTO & Co-Founder *@Matthew Seal*and Elementl’s @jamie for this virtual event. https://dagster.io/events/noteable-dagster-dec-2022

If you missed Wednesday’s Community Day event, fret not…. We have an entire recap for you right here: https://dagster.io/events/community-day-dec-2022

❄️ New channel highlight! ❄️ We’re making improvements to our Snowflake integration! Check out the new getting started guide to learn how to: • set up your connection to Snowflake • store assets as tables in Snowflake • load existing tables as SourceAssets • load tables in downstream assets We’ve got lots of updates planned, including improved partitions support and handling more data formats. Join the channel #dagster-snowflake to get updates about new features, ask questions, and provide feedback!

Today’s demo will start in ~ 1 hour (9AM PST/12PM EST/5PM GMT) - join us on the hour for the demo, or jump on ~20 mins after the start if you want to join the live Q&A. https://us02web.zoom.us/j/81542945555?pwd=U3BTeGxFU2szalorNWdXb3hENEhXdz09

The Dagster team will be out of office Friday 12/23, Monday 12/26, and Monday 1/2. Our Slack responsiveness will be limited on those days and the surrounding weekends. Additionally, we’ll have no scheduled Thursday releases on 12/22 and 12/29. We’ll resume weekly releases on Thursday 1/5. Happy Holidays and Happy New Year!

Just a quick note that we will NOT be holding the regular bi-weekly demo on Dec 28th. We will resume the demos on Jan 11th. Thanks,

Happy new year to the Dagster Community! We are planning an event next month (Feb 2023) focused on migrating from Apache Airflow to Dagster. If you have made this transition yourself and would be open to sharing your perspectives with the community, please let me know. We are looking to share best practices, challenges, pitfalls, and how-to tips.

🕴️🏽 Happy Friday y’all! This is long overdue, but we now have a LinkedIn page for Dagster (it’s an affiliate page of our existing Elementl page!) You can finally tag us in LinkedIn posts 🤣 On LinkedIn, we’ll be sharing Dagster content from our core team, and reposting content from our community, and promoting upcoming events (such as our February 2023 event focused on discussing strategies for migrating Apache Airflow to Dagster!) Make sure to give us a follow to keep up to date 🙂

There are two upcoming ML conferences looking for presentation proposals. IF you are using Dagster for ML and would like to collaborate on a submission to talk, please let me know. Submissions for both are due next Sunday (Jan 15th). The conferences are: • MLDM: International Conference on Machine Learning and Data Mining -> July 15-19, 2023 in NYC • MLConf -> March 30, 2023 in NYC

Support & Release Update For Next Week The Dagster team will be going on an offsite next week, from 1/23 until the 1/27. During that time, we will still be responsive on Slack, but it might take us a little longer than usual to reply. Additionally, we'll have no scheduled Thursday release on 1/26. We'll resume weekly releases on Thursday the 2/2. Thank you for your patience!

👋🏻 <!here> just a reminder that our Airflow Migration event starts in 5 minutes. We think it will have valuable content for all community members, not just ones that are actively seeking to migrate, since I’m sure many of you field questions about why Dagster and not Airflow are a regular (if not constant) basis. We’ll be active in the chat as well. We’re also announcing 1.0 of our dagster-airflow library, which assists migrations. We can ingest unmodified Airflow DAGs into Dagster and execute them natively. It sounds like witchcraft––which it is––and also sounds impossible––which it is not. The demo alone is worth the price of admission. Also all of this content will be available offline! Thanks, — Nick

👋 Happy Thursday everyone! The Dagster team will be out of office Monday 2/20. Our Slack responsiveness will be limited on that day.

:daggy-celebrate: New channel highlight! We partnered with Weights &amp; Biases on an integration and released

dagster-wandb

in 1.1.18 (release highlights). You can now orchestrate your MLOps pipelines (e.g. launch jobs in W&B) and maintain ML assets (e.g. W&B Artifacts and Model Registry) with Dagster. Join the channel #dagster-wandb to receive updates about this integration as well as to discuss your use case with orchestrating your ML pipelines :wandb:

Following a number of requests by people either looking to offer freelance services or teams looking for freelancers, we have introduced a #gigs-freelance channel. Any full-time positions should still be posted in #jobs . Enjoy!

:daggy-celebrate: New channel highlight! This week’s release includes a new BigQuery I/O manager! If you’re using BigQuery, join #dagster-bigquery to get updates on new features, ask questions, and give feedback!

Our next open demo and live Q&A is tomorrow - Thursday 9th, Join us at 9:00 AM PST / 5:00 PM GMT for the demo or at 9:30 AM PST / 5:30 PM GMT if you just want to join the Q&A. Join us: https://dagster.io/dagster-demo-signup