Collecting Metadata from Ops During a Job Run I'd like allow ops to record metadata (their granular progress with comments and notes) when they run as part of a job. The idea is to collect the metadata from all the ops and run a reporting step once the job has completed or at the end of the job. This reporting step would probably take the form of an email or a slack message and would include details about each step (op) and most importantly, give some details about why a step might not have succeeded. This reporting is intended to go to business process owners and will need to be human readable. You could think of the "metadata" as a simple datastructure, e.g. list of dicts which each op would append to. I had thought about the ops receiving metadata as input (from previous ops) and add to it and the return it as an output (to the next op), but that confuses that flow of the "real" data between the ops I then thought to have the metadata as a resource available to the ops, which would allow each op to write to the metadata without cluttering inputs and outputs. And I could then use

failure_hook

and

success_hook

to access the metadata and create a report. But the issue with that is that the hooks apply at the op level and I only want to create the report when the job is complete. So, then I though I could use a run status sensor to detect when a run is complete and generate the report, but these sensors don't have access to the job's resources (they will already have been destroyed by that stage). Does anyone have any ideas that would allow me to collect detailed metadata from ops and build a report at the end of a job run?

New blog post I made (in Portuguese) about Dagster, trying to promote Dagster in Brazil as much as I can!! https://www.lucasgabrielb.com/adeus-dbt-como-orquestrar-seu-banco-de-dados-com-dagster-assets/

🆕 New Pandas 2.0 release. I found that version fascinating; that's why I wrote about it. With Apache Arrow as its backbone, Pandas is now faster and more powerful than ever before. I'll explore why and compare it to alternatives (Polars, Vaex, Koalas, or even DuckDB), consolidating the in-memory format to an open standard (Apache Arrow). I hope you like it. Also, I'm curious about what you think about recent development around the in-memory ecosystem.

Do we have Dagster Australian community? Would be good to meet-up with folks from down under who are using Dagster. 🙂

does anyone have any blogs for their companies data engineering infrastructure? In particular when doing data science / ML model training We have a setup which is appropriate for us, but we wanted to compare to how more established companies might be doing it. In particular, we have feature engineering pipelines which can be adjusted, so we are going to have many similar, but different, sets of feature values. We are trying to ensure we structure the data in the best way possible

cross posting a blog post that @Fraser Marlow and I wrote about using dagster, dbt, and bigquery to detect fake github stars: https://dagster.slack.com/archives/C01NXMEPJ7R/p1678992240303919 we also open sourced the code behind the story in this example dagster project: https://github.com/dagster-io/fake-star-detector

This may be off topic but wasn't sure where else to ask, does anyone know of a good parquet svg icon I can use for a website?

Little article on automating the boring part of my job! hope you enjoy https://www.nintoracaudio.dev/data-eng/2023/03/18/dbt-generator.html

Hi. This weekend I've been using GPT-4 to create Dagster code, and it went surprisingly well. The main issue is that the last version GPT-4 knows is 0.13.8 (Nov-2021), so most code is pretty obsolete (as a curious note, theoretically the knowledge cut for GPT-4 is Sep-2021, not Nov). Did anyone else try any other LLM to generate Dasgter code? Is the code generated by Copilot more modern?

just want to share out this great essay, since it was written by an unnamed member of this very community :blob_ninja: ! give it some love if you get a chance https://stkbailey.substack.com/p/data-in-a-downturn

RIP Gordon. https://www.moore.org/article-detail?newsUrlName=in-memoriam-gordon-moore-1929-2023

Dagster Slack emoji pack is 🔥:dogster:

Hey Nicolas! I do want to read your book, when is it coming out?

According to your experience how likely is that switching from dagster version 1.2.1 to 1.2.6 will break your code? Reformulated: what is your experience, how strictly dagster follows the semantic versioning?

It's been interesting seeing how things play out for Apache Arrow, python + rust. Polars is a new one (new to me at least) that I'm keeping an eye on, probably for use side by side with duckdb and offloading the heavy lifting from pandas. It uses ConnectorX for DB I/O with rust, then polars itself is rust too. I never quite got the hang of Flight services, a bit over my head, but adding arrow/parquet as supported content types in existing HTTP services was pretty straightforward. It reminds me of Spark in some ways, but vectorized - Sandy would like this one I think. Generally when it comes to decoupling business logic, unit tests, etc, I'm thinking of arrow as the interop format to bring things into and work from. https://www.pola.rs/ Better parquet reading vs pandas: https://pola-rs.github.io/polars-book/user-guide/howcani/io/parquet.html Podcast+video: https://talkpython.fm/episodes/show/402/polars-a-lightning-fast-dataframe-for-python

Just one of my loved patches for OSS Dagit:

sed -i 's/FIFTEEN_SECONDS = 15 \* 1000/FIFTEEN_SECONDS = 5 * 1000/' \
    js_modules/dagit/packages/core/src/app/QueryRefresh.tsx

Really wish ChatGPT has more current Dagster functionality 😅 It has been awesome to help learn more abstract things but unfortunately it's only updated with the 0.12.11 release.

Data lineage questions are like a never-ending story, but we’ve got your back! Check out our attempt to explain this concept with examples. https://www.decube.io/post/data-lineage-examples-concepts-and-techniques

Lets start building our own LLM. https://www.decube.io/post/building-your-own-large-language-model-llm-model-with-openai-using-microsoft-excel-file

Esoteric backfill pleasure 🥴

@Dagster Jarred Hiya Jarred!

👋 Does anyone have recommendations on a platform for online pipeline deployment with integration with ML models? What I mean by a platform for online pipeline deployment with ML: • Real-time, not batch workflows • Can do pure data processing using API (i.e., input, return output) • Can do ML prediction + Downstream Data Processing using API (i.e., input, prediction with ML model, perform data processing, return output) • Data/Model versioning is nice to have • Experiment Tracking is nice to have

Trying to share with my colleagues why this is cool functionality but no one gets why im excited 🥲

Has anyone tried out Scalene for profiling in Dagster? I was thinking of using it for op metrics like CPU and memory. It should work for digging into handoffs between duckdb, polars, arrow, etc - they mention C a lot but it looks like their approach works fine for rust libraries too. Maybe keeping it on for some always-on metrics if it's really lightweight enough, maybe emit some to prometheus... otherwise for profiling during dev. https://github.com/plasma-umass/scalene

Hello! Adopting Dagster has allowed PUDL to distribute more help data tables. This has encouraged us revisit our data modeling and naming conventions. We have some ideas on how to update our table naming conventions to accommodate our adoption of dagster and all of our new tables but we’d love some additional input. If you have experience with data warehouses and are interested in supporting an open source data engineering project I’d love to talk to you! Thanks!

Is there any plan to submit Dagster as a CNCF sandbox/incubating project? https://landscape.cncf.io/members #dagster-kubernetes is quite a popular channel, and I believe Dagster would gain a lot in terms of visibility

I wonder, if there are some more people here who are interested in spatial/geo data processing. Would love to learn how you use dagster and apply it to e.g. GeoPandas, GeoArrow, OSM or whatever datasets you are working with… If there is sufficient interest, a new channel e.g.

dagster-spatial

or

dagster-geo

might be an option!? @Gabe Schine @Philip Orlando @Fabio Picchi

Latest article of a 3 part series around data modeling. Dagster is mentioned here and there for obvious reasons 🙂 I hope you like it.

THANK YOU!!!!!!!!! 😛artyblob: