Another question about managing external jobs. Are there examples of/best practices for kicking off an external pipeline from an op within my graph? I'm thinking about managing an external analysis which consists of multiple pipelines itself via dagster. So I could have a job where op-1 runs pipeline-1, op-2 runs pipeline-2, and op-3 runs pipeline-3. Should the ops start the external pipeline and then manually poll the external system for completion? Or should the external pipeline make a request to dagster for reporting completion? Thank you for any hints in the matter.
02/17/2023, 6:55 PM
hi @moritz! the polling approach fits best with the dagster architecture. We use that pattern internally in some of our integrations with cloud tools (e.g. Fivetran)
02/18/2023, 10:54 AM
Thanks for the pointer to the fivertran integration. Looking at the code for the resources was very instructive.