<!here> we are live!

📢 Announcing a webinar next week in collaboration with Noteable! Fresh from PyData NYC, we are pleased to bring you this great session on debugging Notebooks: Data engineers waste a lot of time troubleshooting long-running pipelines and know only too well the frustration of minor errors consuming hours of work. In this practical tutorial we will demonstrate an innovative solution for dramatically shortening testing cycles and reducing the number of reruns required, boosting developer/practitioner productivity, and reducing frustration on the team. A productionalized notebook integrated with an orchestration platform provides an excellent balance of reproducibility, flexibility, and intent in a way that will be quickly consumable. This tutorial is valuable to data scientists and data engineers. This setup makes it easy to take notebooks from exploratory to production, but even easier to debug and ensure quality over time. This tutorial will show how you can achieve: • Time-saving in initiating jobs: Allowing users to seamlessly transition an exploratory workflow created within a Noteable notebook, into a productionalized scheduled workflow in Dagster. • Time and Cost Saving for debugging failed runs: Allowing users to immediately dive into a live running notebook at the point of failure, with all of the in-memory state preserved. This saves the users’ time, as well as saves companies’ compute costs by not requiring debugging to re-execute previous steps of the workflow. Join Noteable’s CTO & Co-Founder *@Matthew Seal*and Elementl’s @jamie for this virtual event. https://dagster.io/events/noteable-dagster-dec-2022

If you missed Wednesday’s Community Day event, fret not…. We have an entire recap for you right here: https://dagster.io/events/community-day-dec-2022

❄️ New channel highlight! ❄️ We’re making improvements to our Snowflake integration! Check out the new getting started guide to learn how to: • set up your connection to Snowflake • store assets as tables in Snowflake • load existing tables as SourceAssets • load tables in downstream assets We’ve got lots of updates planned, including improved partitions support and handling more data formats. Join the channel #dagster-snowflake to get updates about new features, ask questions, and provide feedback!

Today’s demo will start in ~ 1 hour (9AM PST/12PM EST/5PM GMT) - join us on the hour for the demo, or jump on ~20 mins after the start if you want to join the live Q&A. https://us02web.zoom.us/j/81542945555?pwd=U3BTeGxFU2szalorNWdXb3hENEhXdz09

The Dagster team will be out of office Friday 12/23, Monday 12/26, and Monday 1/2. Our Slack responsiveness will be limited on those days and the surrounding weekends. Additionally, we’ll have no scheduled Thursday releases on 12/22 and 12/29. We’ll resume weekly releases on Thursday 1/5. Happy Holidays and Happy New Year!

Just a quick note that we will NOT be holding the regular bi-weekly demo on Dec 28th. We will resume the demos on Jan 11th. Thanks,

Happy new year to the Dagster Community! We are planning an event next month (Feb 2023) focused on migrating from Apache Airflow to Dagster. If you have made this transition yourself and would be open to sharing your perspectives with the community, please let me know. We are looking to share best practices, challenges, pitfalls, and how-to tips.

🕴️🏽 Happy Friday y’all! This is long overdue, but we now have a LinkedIn page for Dagster (it’s an affiliate page of our existing Elementl page!) You can finally tag us in LinkedIn posts 🤣 On LinkedIn, we’ll be sharing Dagster content from our core team, and reposting content from our community, and promoting upcoming events (such as our February 2023 event focused on discussing strategies for migrating Apache Airflow to Dagster!) Make sure to give us a follow to keep up to date 🙂

There are two upcoming ML conferences looking for presentation proposals. IF you are using Dagster for ML and would like to collaborate on a submission to talk, please let me know. Submissions for both are due next Sunday (Jan 15th). The conferences are: • MLDM: International Conference on Machine Learning and Data Mining -> July 15-19, 2023 in NYC • MLConf -> March 30, 2023 in NYC

Support & Release Update For Next Week The Dagster team will be going on an offsite next week, from 1/23 until the 1/27. During that time, we will still be responsive on Slack, but it might take us a little longer than usual to reply. Additionally, we'll have no scheduled Thursday release on 1/26. We'll resume weekly releases on Thursday the 2/2. Thank you for your patience!

👋🏻 <!here> just a reminder that our Airflow Migration event starts in 5 minutes. We think it will have valuable content for all community members, not just ones that are actively seeking to migrate, since I’m sure many of you field questions about why Dagster and not Airflow are a regular (if not constant) basis. We’ll be active in the chat as well. We’re also announcing 1.0 of our dagster-airflow library, which assists migrations. We can ingest unmodified Airflow DAGs into Dagster and execute them natively. It sounds like witchcraft––which it is––and also sounds impossible––which it is not. The demo alone is worth the price of admission. Also all of this content will be available offline! Thanks, — Nick

👋 Happy Thursday everyone! The Dagster team will be out of office Monday 2/20. Our Slack responsiveness will be limited on that day.

:daggy-celebrate: New channel highlight! We partnered with Weights &amp; Biases on an integration and released

dagster-wandb

in 1.1.18 (release highlights). You can now orchestrate your MLOps pipelines (e.g. launch jobs in W&B) and maintain ML assets (e.g. W&B Artifacts and Model Registry) with Dagster. Join the channel #dagster-wandb to receive updates about this integration as well as to discuss your use case with orchestrating your ML pipelines :wandb:

Following a number of requests by people either looking to offer freelance services or teams looking for freelancers, we have introduced a #gigs-freelance channel. Any full-time positions should still be posted in #jobs . Enjoy!

:daggy-celebrate: New channel highlight! This week’s release includes a new BigQuery I/O manager! If you’re using BigQuery, join #dagster-bigquery to get updates on new features, ask questions, and give feedback!

Our next open demo and live Q&A is tomorrow - Thursday 9th, Join us at 9:00 AM PST / 5:00 PM GMT for the demo or at 9:30 AM PST / 5:30 PM GMT if you just want to join the Q&A. Join us: https://dagster.io/dagster-demo-signup

👋👋👋 We'll be hosting our next Dagster community call on April 11st at 12PM ET, where we'll be walking through some of the things we've been working on lately. We see lots of developers doing really cool things with Dagster and we want to give you the opportunity to showcase some of those projects to the broader community. If this sounds exciting to you or someone on your team reply to this thread 🧵 to showcase your project in the next or subsequent community calls. Can't wait to see what you've been working on!

👋🏻👋🏻👋🏻 Coming in 1.3 (April 19th): Python Resources and Config Hey all I just wanted to highlight a blog post we just pushed out describing upcoming changes to the resources and config system. In February we discussed this in a github discussion. This has been going well and we are preparing to roll this out more broadly. Major changes: • Instead python dictionaries and a bespoke schema system, now the system has a Pydantic frontend. • Instead of untyped access on a context object, resources and config can be passed directly as function parameters to assets and ops • Resources passed to Definitions will automatically bind to jobs. • Sensors and schedules now support resources • Resources now have a place in the UI and their configuration is viewable and inspectable there. While this does not require any action on your part (existing code will continue to work) this is a big change to the system. We have strong conviction that this is a huge better API so we will be moving all docs, integrations, examples, and content to lead with this system instead of the old one starting on April 19th. This has been released for some time experimentally and you can use fit now. Documentation is here. Feel free to give it a spin and give us feedback.

We’re live here! Join us with @sandy talking about Partitions and @schrockn talking about Pythonic Config and Resources

:daggy-celebrate: New channel highlight! Many of our community members are using Dagster to orchestrate machine learning pipelines. We are launching a new machine learning channel as a place to discuss building ML pipelines and get feedback on new content and features in Dagster that can be used for ML. Join the channel #dagster-machine-learning to discuss your use cases orchestrating your ML pipelines and get updates on useful integrations and related content.

Hi everyone 👋👋! We're excited to announce the Dagster Community Advisory Board- @Jonathan Neo @Zach P @Stephen Bailey @Andrea @Vinnie @Daniel Gafni and are looking for new folks to join! We are so thankful to everyone in the Dagster community and want to recognize the champions that are out there helping others, showcasing their Dagster work, and giving us incredibly valuable product feedback. Some of the benefits of joining the board include community recognition, Dagster Cloud Serverless Credits, community meeting spotlights, beta-testing new features, access to our marketing and product team, and some other surprises we have up our sleeve. If you enjoy supporting other members of the community, spreading the word about Dagster, contributing to the project, getting together with other members of the community in a meetup, or leading discussions on software/data engineering best practices, you might be a great candidate to join the Board. Want in? Apply here Dagstar Application

Build Better Analytics Pipelines Join us today for a live showcase of building a data platform for data analytics using Dagster. The broadcast starts at 9 AM PST / 12 PM EST / 4 PM GMT / 6 PM CET https://youtube.com/live/kpco5u1zG9Y

Our next open demo and live Q&A is tomorrow - Thursday 18th, Join us at 9:00 AM PST / 5:00 PM GMT for the demo or at 9:30 AM PST / 5:30 PM GMT if you just want to join the Q&A. Join us: https://dagster.io/dagster-demo-signup

Today’s demo will start in just a few minutes. You can join directly with this link: https://us02web.zoom.us/j/81542945555?pwd=U3BTeGxFU2szalorNWdXb3hENEhXdz09

👋:daggy-celebrate:👋 We'll be hosting our next Dagster community call on June 13th at 12PM ET. We see lots of developers doing with Dagster across data engineering, machine learning, and analytics, and we want to give you the opportunity to showcase some of those projects to the broader community. If this sounds exciting to you or someone on your team reply to this thread 🧵 to showcase on the next call. Looking forward to seeing what you've been working on!

📈 📈 📈 Exciting news Elementl has raised a $33M Series B to accelerate the growth of Dagster and Dagster Cloud! See the blog post and tweetstorm for more details. 📈 📈 📈 We’re really starting to see Dagster as oriented around data engineering as a craft and discipline. This is a subtle yet important shift, as people often jump directly to the data engineer job title. But it’s not just about data engineers. Lots of ML engineers and data scientists use Dagster, and it makes sense since ML engineering is mostly data engineering. Analytics engineering is a data engineering subspecialty focused on sql-based analytics workflows. When they want to expand beyond that use case, or operationalize dbt in a broader data-engineering-driven platform, they bring in Dagster. Lots of you reading are also data platform engineers, whose job is to scale data engineering across an organization, and think of all the other job titles above as your stakeholders. Our vision is that Dagster is the best place to build python-based data pipelines and the system that ties the work of all of you together in a single, coherent layer and control plane. This capital will allow us to think bigger not just in terms of community and commercial growth, but also product expansion. In terms of realizing the potential of data engineering as a discipline, we are still in early days. Practitioners on the ground know there is so much more possible in terms of improving productivity and reducing chaos. We have an exciting roadmap ahead. Thanks to all our users, community members, and Cloud customers. Your usage and support is what makes this all possible. If you want to join us on this mission, we’re hiring across Sales, Marketing, Design, Operations, and Developer Relations. See our open roles here!

Our next open demo and live Q&A is tomorrow - Thursday, June 1st, Join us at 9:00 AM PST / 5:00 PM GMT for the demo or at 9:30 AM PST / 5:30 PM GMT if you want to join the Q&A. Sign up for reminders: https://dagster.io/dagster-demo-signup Or just jump on the call tomorrow: https://us02web.zoom.us/j/81542945555?pwd=U3BTeGxFU2szalorNWdXb3hENEhXdz09