https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
b

Binoy Shah

07/06/2022, 7:34 PM
Hi Sorry for cross posting this from #random, I encountered this comparison of Dagster vs XYZ and wanted to get some feedback on its accuracy/bias check Any thoughts appreciated
z

Zach

07/06/2022, 7:43 PM
the dagster features seem accurate to me, but it pretty clearly reads as a biased comparison of Dagster vs. the others in favor of Dagster. granted, I'm biased myself that Dagster is the best and that everyone should be using it, but if the goal is to read as an unbiased comparison it doesn't quite come off that way
b

Binoy Shah

07/06/2022, 7:44 PM
Thanks, which portions can I ask for getting re-visited ? Airflow accuracy ?
z

Zach

07/06/2022, 7:45 PM
I can't really comment on Airflow / Prefect accuracy of features - it's more of a nuanced (and probably personally opinionated / me being picky) reading of the descriptions for "CI/CD Friendly" and "Test Driven Development" - for CI/CD something like "CI/CD can be achieved with convenient CLI tools, Separation of Server..." might be a less opinionated-sounding comparison against Prefect. Similarly something like "direct local execution of DAGs for code. Fixtures / helper functions allow for native and local TDD..." but honestly as I type that out it feels really picky and maybe most people wouldn't read it quite as critically as I am.
b

Binoy Shah

07/06/2022, 7:50 PM
Thanks for your thoughts
z

Zach

07/06/2022, 7:55 PM
Also might want to mention "optionally enforced by framework" in the Typesafe Python Pipeline section as type checking only really happens if you use Dagster Types - at least for inputs / outputs
b

Binoy Shah

07/06/2022, 7:56 PM
Okay, that’s a good point
z

Zach

07/06/2022, 7:56 PM
honestly maybe just ignore my first set of comments
b

Binoy Shah

07/06/2022, 7:56 PM
🙂
s

Stephen Bailey

07/07/2022, 12:06 AM
hey @Binoy Shah , are you affiliated with elementl at all? i think it makes a big difference if you're coming from the stance of "i am invested in dagster's success" vs "i am annoyed with all other tools". i think it reads fine if the latter (and i actually appreciate it), but if you're in the former camp i think the language is a bit stilted
one nit as well is that i usually see these charts have the "all green" rows ordered first, then start showing where the features differ
b

Binoy Shah

07/07/2022, 9:24 PM
Hi Stephen, I am future adopter of a workflow engine, I am asking around for evaluations, I received this. I have spent most of the time at #dagster-* channels compared to other communities and so I am not very conversant with alternatives. The current recommendations I have are as described in the slide and I wanted a “fair” evaluation ratings for the contenders
Secondly, We do have full fledged Kubernetes based Infrastructure running on top of AWS/EKS. We have stable CI/CD pipelines to build/deploy Docker Images and Helm Charts and environments are separated at Namespace level Our observability is via NewRelic
DW is Snowflake and data sources are plethora of custom Services
ELT is carried out by Meltano + DBT or Python + Celery Everything running on Kubernetes
@Stephen Bailey ☝️
s

Stephen Bailey

07/07/2022, 10:45 PM
Ahhh, for some reason I thought you were writing this for a blog! Then yeah, I think all of these are fair. I'll say that in my evaluation, I thought all the tools were wildly hard to grok completely, but I felt Dagster was the only tool that was designed from the ground up for engineers, especially when it came to the commercial offerings. I love the investments Dagster cloud is making into the CI/CD workflows, really stands apart from what I know of the other tools