Dagster External Execution Protocol (ext) Hey everyone. Just want to highlight a github discussion we just posted, detailing a feature coming out in our next release (currently scheduled on September 28th). We call it the Dagster External Execution Protocol or “ext.” ext has a few goals: • Make it easier to orchestrate existing code with Dagster with little to no modifications to that existing code • Make it easier for centralized data and data engineers to onboard stakeholder teams onto Dagster incrementally • Run code in external environments and easily get streaming logging and structured metadata back into Dagster’s UI and tools • Formal support for separation of orchestration and business logic environments • First-class multi-language support in Dagster Please read the github discussion for more details and to give feedback, but how are achieving these goals is shipping a toolkit for greatly improved integrations with external environments such as Kubernetes, Spark, and other external runtimes. It standardizes parameter/context passing, as well as streaming logs and metadata back to Dagster. It is also extensible and pluggable, so that users can build on it to orchestrate their own customized environments. We’ve created a channel #dagster-ext to follow updates and discuss in more detail. Thanks all! Looking forward to getting this out to everyone.

Welcome!

avneeshs1

joined #announcements.

Welcome!

avneesh saxena

joined #announcements.

Welcome!

edward852

joined #announcements.

Hi everyone, here at Dagster Labs, we are looking for our first Support Engineer! 🎉 If you know of anyone or want to be part of the team, let us know! The best candidates are experienced engineers who are excited about Dagster and helping our customers be successful in using it. Read more or apply here and ping me for questions. Thanks!

Welcome!

Adam May

joined #announcements. Also,

Yaroslav Volkov

joined.

📣 Community Call update 📣 Our next community call will be on November 14th, our last call for 2023. This is a change from the originally scheduled October 10th because we will be having a major event that week, with lots for the community to enjoy. • Add the invite for the November 14th community call to your calendar here. • Sign up for launch week here.

We will be hosting our Fall Launch Week from October 6th to the 13th. Each day, we will announce and showcase new features and capabilities on the Dagster platform. The theme of Launch Week is “*Escaping the Modern Data Trap*.” Why? The advent of the Modern Data Stack revolutionized the domain of data engineering. But over time, it drifted from its original premise. It became fragmented, overcomplicated, and expensive to run. Join members of the Dagster Labs team as we share our vision of how to escape this Modern Data Trap and get back to the initial intent: to deliver a developer experience that is unified, streamlined, collaborative, and where you are in control of costs. We hope you will join us for this discussion and new updates: https://dagster.io/events/dagster-launch-week-oct-2023

Welcome!

Alex Kwiatkowski

joined #announcements. Also,

Serge

,

Doran

and

phuongle16062000

joined.

ICYMI: Dagster 1.5 shipped this week - you can read about the major updates here: https://dagster.io/blog/dagster-1-5-how-will-i-know This sets the stage for next week’s Launch Week - be sure to join us, starting on Friday 6th, as we dive deeper into the new capabilities of Dagster: https://dagster.io/events/dagster-launch-week-oct-2023

Read today’s announcement on Dagster’s brand new Asset Checks - blogpost and video overview here: https://dagster.io/blog/dagster-asset-checks

Welcome!

sandman

joined #announcements.

Join us in a couple of hours for an overview of Embedded Insights. 👀 - a new capability that lets you pinpoint which steps in your data pipeline are inflating your cloud bill. Get operational observability on your data pipelines and bring cloud costs back under control! 💸 We start at 12:00 noon EST / 9:00 AM PST / 4 PM GMT https://youtube.com/live/o0IMRRg0eqM

Coming up next on Dagster’s Launch Week agenda: Stakeholder Education. Join us to learn about Dagster University and an overview of the Dagster Open Platform - a complex and at-scale Dagster project. Tune in today at 9 AM PST / 12 noon EST / 4 PM GMT. https://www.youtube.com/live/a2ygWoOlKiQ?feature=shared

Welcome!

Hector

joined #announcements. Also,

Kunal

joined.

Today is day 5 of Dagster’s Launch Week. Join us as @Pedram Navid shares details of the new Embedded ELT feature. Why deploy a cumbersome third-party SaaS tool for data movement tasks when an embedded and composable library will often do the job? https://www.youtube.com/live/Qgxu7dL29tg

Welcome!

theavilla

joined #announcements.

📚🏁 New channel alert 🏁📚 I just created the #dagster-university channel, for users going through the course to post questions, troubleshoot and/or provide feedback. We'd love to see your capstone projects posted here as well.

We are live now for the final day of Launch Week, featuring @schrockn

That’s a wrap for Launch Week. If you would like to access any of the content, you will find links to the videos and blog posts for each presentation here: https://dagster.io/events/dagster-launch-week-oct-2023 To summarize, the sessions included: • Escaping the Modern Data Trap • Introducing Asset Checks • Improving visibility into data operations with Dagster Insights • New resources for Mastering Dagster • Stop Reinventing Orchestration: Embedded ELT in the Orchestrator • Introducing External Assets • Introducing Dagster Pipes

Welcome!

Michal Ficek

joined #announcements.

Hello Dagster Community! I want to thank all of you for helping make this place an important part of our community. We’ve heard feedback from you all that Slack questions can get lost and aren’t discoverable, meaning we often see the same questions asked repeatedly. You may have also noticed that we’ve been seeing explosive growth in our support volumes that have sometimes exceeded our ability to provide timely help in response to your questions. As a result, we’re making a few changes to how we handle support questions to help improve the community experience by driving more activity to a searchable corpus while also helping create a more sustainable support experience. 🥇First, we ask that the community begin posting to GitHub Issues and Discussions. We have already started copying common questions and issues to GitHub and are asking for your help to make it easier to find answers to questions. Use GitHub Issues for technical issues with Dagster, while Discussions are best suited for feature requests, ideas, and other questions. During this transition period, we will continue to migrate questions out of #dagster-support and toward Github. Questions will be answered in Github rather than Slack. Finally, over the next few months, once the initial transition period is complete, we’ll rename #dagster-support to #ask-community. This will emphasize that our Slack community will still be open for you all to help each other, but for support from the Dagster team, questions will only be answered from Github. Thanks for being such a valuable member of the community!

Welcome!

Alexander Whillas

joined #announcements.

Welcome!

misha sauchuk

joined #announcements. Also,

Kingshuk

joined.

Hey Dagster Community, Dagster PM here. If you’ve got feedback you want to share about Dagster (and get paid for it) check out our latest user technology survey https://www.surveymonkey.com/r/CPCBT2M. This input influences our thinking about roadmap prioritization for the coming 3-6 months. We’ll send you a $25 dollar gift card of your choice for your time. Thanks for your ongoing support!

📣📣📣 Community call next week! 📣📣📣 Our last community call of 2023 is next week, November 14th, at 12:00 PM ET. We have an exciting agenda prepared with members of the community showcasing projects they have been working on. • @Daniel Gafni will be showcasing how sanas.ai is using Dagster. Daniel will be showing how they’re using a

RayClusterResource

to run neural network inference on millions (yes, millions) of audio files. We’ll also learn about using the

BranchingIOManager

for easy testing of Branch Deployments • @Andrew Maguire will be presenting Anomstack, a lightweight data app that detects anomalies in your data built with Dagster. Preview the project here! If you missed the call, check out the videos on Youtube: • Daniel's Presentation: • Andrew's Presentation:

Welcome!

jlrbi

joined #announcements. Also,

a meledin

and

Antoine Jouanjean

joined.

Community call starting shortly (at 12:00 PM EST): https://us02web.zoom.us/j/88939083535 See you there!

Welcome!

JoeYuZhou

joined #announcements. Also,

rajesh pal

and

avinash chakrawar

joined.

Hello folks! Just a heads up that the Dagster team will be out for the rest of the week as we celebrate Thanksgiving 🦃. During this period, we'll be unavailable for support. If you have any questions, we've built a large compilation of answers in GitHub Discussions, and you're still welcome to ask the community for help. Cheers!