Andrew Grigorev04/13/2023, 7:57 AM
it is not possible to get different compute configurations in a single job. It could be achived with
, but we use AWS ECS for which there is no Executor (there is only a EcsRunLauncher). And after all, concurrency limits also don't work on the step basis. In two separate jobs, we can specify compute configurations and concurrency limits, but there would be no single graph of operations, and some bicycle would be needed to run the second job after the first job completed for corresponding partition. How should I build such pipeline in Dagster?
claire04/14/2023, 6:55 PM
Andrew Grigorev04/15/2023, 7:16 AM
code to work not only for the latest partitions of assets, but also for all defined partitions. It seemed to behave well and produced proper jobs to carry out a full backfill of all assets. However, I feel that using the
for this task might be a bit overkill and having its patched version in the pipeline codebase might not be the most suitable solution for a production environment. I'm interested to hear the Dagster team's thoughts on this use case and the most appropriate solution to handle it. Thank you once again for your assistance.
claire04/19/2023, 10:48 PM