That was a great episode

and is actually how I got here.

hadn't used or heard and I listen to that podcast

I'm happy to hear it 🙂 As the host of the show it's always great to see that it actually has some impact.

Thanks everyone!

@Tobias Macey it's a really great show as I'm pretty new to the data space and it's a great listen on the train to work. Thanks for your hardwork

👍

Yeah thanks a lot for your show Tobias!

Realize it's short notice, but really curious if the Dagster team is going to do a compare/contrast to Netflix's Metaflow that was just open sourced yesterday.

No plans from us on the dagster team to do any sort of detailed compare/contrast with other systems at this point in time. Ideally something like that would come from a third party.

If someone wanted to weigh in on the comparison with Metaflow, this thread in Hacker News would be a good place to start: https://news.ycombinator.com/item?id=21701010

in honor of today's release:

Fishtown Analytics raised a round: https://blog.getdbt.com/fishtown-analytics-announces-12-9m-fundraise/

Hi all, We (Datafold) are hosting a virtual Data Quality Meetup on Thursday, June 4th at 10AM PST. If DQ is on your agenda or if you are just curious about what other Data teams are doing to monitor and improve the quality of their data, please come join us! Format: lightning talks, war stories, QA & a lightly moderated discussion. If you are excited to share how you've leveraged D  to improve data quality and pipeline robustness, that would be of great interest to the audience. Please shoot me a message if you are interested in presenting or participating in a panel. RSVP: here link for Zoom link + calendar invite, also an event page on LinkedIn where we post updates.

Hi all, I'm looking at implementing a data pipeline at work and Dagster seems really cool. One thing we would like to be able to do is to publish a data artifact from 1 workflow and then use that in another workflow. e.g. Write an import worker that reads from s3 and publishes a dataset Multiple pipelines read from this dataset (and others) and rerun when one changes, without needing to care about the import step. Ideally, these pipelines could be defined separately, rather than in one big file Is that the sort of model that dagster supports (I know there are materialized views) or will I be fighting it all the time?

you should be able to do it in dagster, but we dont yet have any formal support for understanding connections between pipelines so its up to you to make sure you run them in the right order and know how to get the data published from pipeline X in pipeline Y

Hi Everyone, we recently migrated our data warehouse from Redshift to BigQuery and wrote a post describing our process and experience - https://medium.com/omio-engineering/how-we-migrated-our-data-warehouse-from-redshift-to-bigquery-89f33988e1b5

I wrote a short blog post on our engineering values in the data-engineering team at Omio - https://medium.com/@rahulj51/guiding-principles-for-a-data-engineering-team-7baaa35b2954 . Would love to get some feedback/comments.

I also wrote a blog post with 12 Data Engineering approaches and Emerging Technologies. I mentioned dagster in three of them: • “Use notebooks to open up the data silos.” where dagster can productionize your notebooks and • “Use python (and SQL if possible).” where orchestrators like dagster are the prefered choice instead of writing all by yourself and • in "The evolution of Pipeline Orchestration". In case of interest read here at Towards Data Science.

TIL: If you're muted on a zoom call and need to quickly answer a question...just press the spacebar. While it is pressed, you are unmuted to talk.

Dagster seems great - the testability angle is 👌 Guessing I'm just missing this somewhere in the documentation so apologizes for the silly question. AirFlow has the concept of a sensor (to wait for a file to arrive for instance). Is there a comparable feature for that or is the idea that a solid is so versatile you can implement using that Sorry for the newbie question, guessing other interested parties might have the same question! - and thanks! nicolas cage party dagsir

hi guy it seems that neither

print

nor

logging.debug

works in @solid? This is from a 3rd module not a solid itself and I'm running using execute_pipeline

Where are docs like https://docs.dagster.io/_apidocs/libraries/dagster_dask sourced from? I’d like to write some content for the Dask resource.

just got started with dagster after digging around for a while. airflow, kubeflow, flyte etc all so complicated to even get started. dagster is looking great. thanks!

celery was a mistake

that's all

Anyone created awesome solids or resources? 😀 Let’s collect them somewhere centrally? Would be awesome! celebrate Check out my idea: https://github.com/dagster-io/dagster/discussions/3153

Is

dagster-dask

updated on conda-forge on a schedule? It's behind latest which led to a brief bit of confusion.