so prefect is the other system in the ecosystem with the most overlap

for sure

set the channel description: General discussions about data workflow tools

they have identified airflow’s lack of explicit data dependencies as a design flaw, as have we, as an example

i have not used prefect in anger, nor have i talked to someone who has used prefect in production

they would have the most context

but off the top of my head

1) we provide an entire suite of open source GUIs for local development in the form of dagit. there’s no such analog (currently) in prefect

2) we have a fundamentally new abstraction layer (currently called a pipeline, but we are working on naming it different. think of it as a logical dag in contrast to the physical data in an execution engine). It encodes a type system, data quality tests, etc.

3) It gets compiled into an execution plan, which can be executed on a wide variety of execution engines. Currently supported: single proc, multiproc, dask, and airflow. There will be lots of these as the system matures.

4) Dagster has the config system with the typeahead tools, along with the resources concept. We think this makes it possible to write testable pipelines in a deep way. Without good abstraction support its too hard.

5) We have a more credible incremental adoption path, especially for existing airflow installations. You can make a single solid and execute on an existing airflow cluster alongside your existing dags.

so those are my thoughts for now

I think prefect has a lot of merits and should be considered for greenfield projects.

Thanks, Nick

i would love for someone to do a deeper dive as a practitioner

FWIW I just published my interview with the creator of Prefect :) https://www.dataengineeringpodcast.com/prefect-workflow-engine-episode-86/

I'm also planning on doing a POC of both Dagster and Prefect for some existing tasks that I currently have which are managed via Cron. Hopefully in the next couple weeks.

here they have a small comparison between dbt and prefect. Dagster is planned to compare next 🙂 https://medium.com/@squarecog/five-interesting-data-engineering-projects-48ffb9c9c501 As far as I can tell (from a quick look at prefect only, and a bit more in dept into dagster), is that dagster has the better higher-level-descriptive-abstraction which I see very valuable. Where prefect is a better airflow, but not the total abstraction. But I want to dig deeper into prefect, also kedro I’d like to compare, before I make my choice. But thanks to the team! Dagster feels very right and and exactly what I was looking for, but couldn’t really explain how the solution should be.

Hey folks - I'm looking to create a small data warehouse, using dagster as the ETL tool. Any suggestions on what warehousing tech to use?

I added a quick answer on Quora to “What are common alternatives to Apache Airflow?” It’s not very in dept, but at least it should give some comparison between the most used once: https://qr.ae/pNrIPi happy for comments or other opinions 🙂

How would you compare dagster to Apache Beam. Here an example of wordcount. I seems they have similar concepts, but less structured (no solid approach, so I guess more code duplications?), a bit more direct. One other difference they support streams: https://beam.apache.org/get-started/wordcount-example/ What are your comments on that?

How many other folks here are using dagster + dbt?

I'm seeing an error when attempting to import

dagster-pagerduty

TypeError: 'DagsterDictApi' object is not callable

Could this be version related.?We're on dagster 0.9.3

started a new discussion on turning the dbt rpc

_and_wait

solids into composite solids: https://github.com/dagster-io/dagster/discussions/3371

tqdm with dagster / dagit Hello, does anyone have implemented dagster with tqdm ?

I know I've asked this before, though it was several months ago. How are folks taking advantage of Singer taps/targets with Dagster? Has anyone built an integration (however hacky) with Meltano?

Just started using DBT with dagster and love it. I have a v. conventional Data Warehouse in Postgres taking in a combination of Mongo and django API data. Does anyone have any recommendations for a tool to manage the warehouse schema? I'm familiar with flyway and alembic, but haven't used either extensively.

Is anyone using dagster to publish OpenData? And if, do you use CSVW or frictionless or alike to publish metadata for your publications?

Hi all, does anyone have recommendations on data catalog tools?