Hey im enjoying using dagster so far but a question has come up of comparing dagster to AWS step functions and the pros and cons between each. Anybody have any experience with both or know of some less vendor influenced documentation comparing the 2?

Hi, I am new to Dagster and read that

Dagster is good choice when building ML pipelines

can anyone explain in detail how it is good? Thank you

Has anybody used / run Dagster and DE tools on 2023 Macbook Pro M2 ?

Hi everyone, I want to share my blog article about Dagster and Dask in Dagster Blog Page(https://dagster.io/blog) to share information with community. How can I do that anyone knows? My blog link: https://ibrahimhkoyuncu.medium.com/

Hi Everyone, did any one used Dagster GraphQL with Postman. I am mainly looking for the feature Postman Provides - "Graph Introspection". I tried using the schema dump from the Dagster Instance, which does not provide the schema format looking for. If you one who worked on this can share the Schema File you used with Postman ?

Anyone running Apache Iceberg at a small scale? Have kafka connectors that make small files, dag runs that import daily/hourly exports from vendors. And really just looking for a super simple non-spark data compaction tool. Curious if iceberg is feasible with just like s3+dynamo for catalog

Also looking at handling pii gdpr/redactions in small files spread throughout aws that a tabular format like iceberg might help with

Hi all 🙂 Sharing an awesome podcast episode we just released with Dagster Creator @schrockn! https://podcasters.spotify.com/pod/show/ossstartuppodcast/episodes/E105-Bringing-Great-D[…]r-Experience-to-Data-Teams-with-Dagster-e29k6ru/a-aacqjjk

Is there anyone here familiar with the Python

data-diff

package, or potentially just

Delta Encoding

in general? In

data-diff

, it creates deltas for the differences between tables, and for Table A, it can either be exclusive or not, and the same can be said for Table B. What does this "exclusivity" mean in context to deltas? I'm trying to write a parser to parse these deltas that

data-diff

creates ands applies them to our production dataset

The link to Docs in the Changelog for 1.5.0 is broken. It points to https://dagster.dagster-docs.io/concepts/assets/asset-checks correct one is https://docs.dagster.io/concepts/assets/asset-checks

Are there any Data Engineers here? How did yall end up in DE? Are DE roles pretty rare? Over-rated? Under-rated? Paid well?

put our info in intro channel. joined this slack to see if anyone would be interested in having and using dagster vscode extension with pipelines visualization and insights ... https://marketplace.visualstudio.com/search?term=dagster&target=VSCode&category=All%20categories&sortBy=Relevance

In the dagster UI, is there a way to override the logo to put our company logo there?

How would I classify uploading data to Github or somewhere else, would that be a IO Manager? Thinking about the end of the pipeline when I want to publish data if it all looks good.

I'm working with a financial admin group who live in OneDrive/Sharepoint. While I had thought that connecting to my own OneDrive with python would be simple, I've gone in circles with the microsoft documentation and have yet to find a working example that allows me to use my username and password (or even something like a service account) to authenticate. Does anyone have any relevant links to a clean example? The relationship to Dagster is that I'd like to create a simple storage resource or, better yet, use a UPath-based IO manager with OneDrive/Sharepoint. Thanks for any pointers.

What are people using as their data front end? Experimenting with Evidence.dev, but running it keeps blowing up due to NPM/Node issues, which is unfortunate for a static site generator. So now I’m looking at Streamlit, but curious if other folks have ideas.

Please, vote: https://x.com/jaceksan/status/1719370527448457336?s=20

Hello all. I love Typescript but I hate Python. Can I write a Pipeline/Job with Typescript?

I'm using dynamic multi dimensional partitions. I heard about 100k limit for partitions. With our usage we will hit that limit quite soon. Is there a way to clean up old partitions?

Could something like [Mandala](https://github.com/amakelov/mandala) work with Dagster? 🍿 This was a very interesting read: https://amakelov.github.io/blog/pl/

Hey Dagster community! We’re announcing today that the team behind Meltano is now Arch Data, Inc.! We wrote a few blog posts about the change and our new product. https://www.arch.dev/blog/ai-product-development-is-being-held-back-by-data-engineering/

Hey, does a downloadable (pdf?) version of the dagster documentation exist?

Hello Guys, My name is Alessio and I am a student at the University of Turin. I am currently conducting a research for my thesis based on the evaluation of Dagster. I understand that your time is valuable, but I would be extremly grateful if you could take a few minutes to respond to a brief survey that will help me make informed decisions. Your opinions are crucial for my research. You can find the survey here, it is a google form: https://forms.gle/sb7YMYDsKgkDkq5B6 Your contribution will be completely anonymous and will not take more than 5 minutes. Thank you in advance for your participation!

Hello, does anyone have a guide for dag/data pipeline design patterns and advice? On a layout front I feel my dag looks bad, so looking for good principles ahead of a refactor.

I'm presenting tonight at a data engineering meetup. My talk can be summarized in one meme:

New Databricks Users slack channel for anyone who's interested - https://join.slack.com/t/databricks-users/shared_invite/zt-277w1vbk2-Eo_TSv2FZpFLzuT_KaitJA

cross posting https://dagster.slack.com/archives/CCCR6P2UR/p1701282707190949

I started a new online book about Data Engineering Design Patterns. I plan to cut through the hype and focus on patterns that matter in the long term. I just started, but I look forward to your feedback and suggestions 🙏🏻.

This might be a weird idea but does anyone use dagster as a "simple" data catalog? Because you have all your ingestion assets and data transformation assets you can document them within dagster.

I'm running some PySpark code and it's taking 10 seconds per iteration - 50 seconds total for 5 files. (no special installation, just pip install pyspark on a Macbook M1 Max) - Hit a parquet file, use a rolling mean, then find the top result. I converted the code into a SQL query for DuckDB in-memory... and it runs in under a second. Either my PySpark code is unoptimized, and/or DuckDB is magic?! Have other people seen similar speedups?