https://dagster.io/ logo
#ask-community
Title
# ask-community
r

Robin Lindner

07/30/2023, 12:20 PM
Hello all, I am relatively new to Dagster. I have some previous experience with Airflow, however Dagster is a bit different. I am not sure yet how to implement the attached scheme in Dagster. What would you represent here as Op, Asset or Graph? Can anyone help me out a bit here? Thank you guys in advance.
q

Qwame

07/31/2023, 1:29 PM
Dagster is so flexible you have options. You can • use a sensor and dynamic partitions. The sensor will detect rhe presence of files in the SFTP folder and trigger a run request for each partition to materialize each partition. @geoHeil has a nice example of working with SFTP folders using sensors here • Use dynamic outputs. A nice example here using ops and jobs and another here using assets
r

Robin Lindner

07/31/2023, 3:05 PM
Thank you. I'll try
t

Tim Castillo

07/31/2023, 7:28 PM
Hi! By default, you should use assets for this. To be prescriptive: Each vertical in your diagram (zips, CSV, table, aggregate) is an asset, with each horizontal path being a dynamic partition.
a

Antoine Adam

08/20/2023, 11:38 AM
Very nice thread, I have an almost similar usecase (like a lot of people it seems, reading dagster-support channel as well as github issues), mapping an "extract from zip" asset with a downstream asset which compute thing. Both with dynamic partitioning based on the files found in the zip. It seems to me that the solution of @Tim Castillo is ideal and using only asset is nice. Unfortunately dynamic mapping is not yet supported as per https://github.com/dagster-io/dagster/issues/9559. Anyway looking forward to it, I'll wait patiently for the implementation to land on master ! And thanks for this awesome slack channel, found a lot of answers for my questions