https://dagster.io/ logo
#random
Title
# random
d

dixel

12/09/2021, 9:05 AM
Saw this question yesterday on reddit (https://www.reddit.com/r/dataengineering/comments/r9cy2k/dagster_in_place_of_airflow/), thought it would be interesting if someone from the community could also give some comments there.
👀 1
n

Noah K

12/09/2021, 9:10 AM
From my team's bakeoff about a year ago: Airflow is the OG in the space, it's well supported and has integrations with everything, but it's also just kind of a pain to work with.
The Python DSL is extremely verbose in ways that feel pointless, it's default deployment recipes are all very old school, expecting that you still run normal servers.
So it just felt like we would end up fighting the tools more than we really wanted.
t

Tim

12/09/2021, 3:42 PM
I’d also be interested to see other people’s experiences/comparisons between Dagster and Prefect
s

schrockn

12/09/2021, 4:43 PM
This is a great idea. We’d love to have a few answers and then we could use this as a reference post for user-generated answers to this (as opposed from coming from Elementl.) @ramshackle-jamathon is going to submit a response.
Any other volunteers?
n

Noah K

12/09/2021, 6:38 PM
Prefect only supported Dask for remote computation and IMO Dask should literally never ever ever be used. It's an entirely broken model of remote execution based on serializing code objects and sending them over the network. If you make heavy use of complicated C-backed modules in your Python code, expect Dask to explode when you try to version things.
👀 4
r

ramshackle-jamathon

12/09/2021, 7:17 PM
I dropped a comment on that post that was mainly from the perspective of a data engineer writing a job, there's a whole other angle from the perspective of the infra/ops person managing the platform.
TBH it might make for a good group therapy session to air grievances about the airflow scheduler, multi tenancy, python module name collisions cross dags..... 😱
nod2 2
😅 3
2 Views