Hello all, I am relatively new to Dagster. I have some previous experience with Airflow, however Dagster is a bit different. I am not sure yet how to implement the attached scheme in Dagster. What would you represent here as Op, Asset or Graph? Can anyone help me out a bit here? Thank you guys in advance.

Dagster is so flexible you have options. You can • use a sensor and dynamic partitions. The sensor will detect rhe presence of files in the SFTP folder and trigger a run request for each partition to materialize each partition. @geoHeil has a nice example of working with SFTP folders using sensors here • Use dynamic outputs. A nice example here using ops and jobs and another here using assets

Thank you. I'll try

Hi! By default, you should use assets for this. To be prescriptive: Each vertical in your diagram (zips, CSV, table, aggregate) is an asset, with each horizontal path being a dynamic partition.

Very nice thread, I have an almost similar usecase (like a lot of people it seems, reading dagster-support channel as well as github issues), mapping an "extract from zip" asset with a downstream asset which compute thing. Both with dynamic partitioning based on the files found in the zip. It seems to me that the solution of @Tim Castillo is ideal and using only asset is nice. Unfortunately dynamic mapping is not yet supported as per https://github.com/dagster-io/dagster/issues/9559. Anyway looking forward to it, I'll wait patiently for the implementation to land on master ! And thanks for this awesome slack channel, found a lot of answers for my questions