I’m merging multiple assets (CSVs) into a master f...
# ask-community
I’m merging multiple assets (CSVs) into a master file and I want to implement a sensor that listens for the existence of such master file and recreates it upon deletion. This would be done by triggering 4 full pipelines (that each creates one of the required assets) if those files do not exist, or executing the merge solid only if they already exist. So is there a way to 1. Have a single sensor yield
for more than one pipeline and 2. Define a
with logic inside the sensor (all solids if file does not exist, merge only if it exists), rather than passing it as an argument to the sensor definition? or do I need to define an extra pipeline that does all this?
I have sensors for the other files as well. So I guess I could guarantee they always exist with something like the snipper below. This way I can only merge them if the master file gets deleted.
targeting multiple pipelines from one schedule/sensor is not a pattern we currently support but are looking to in the future
@alex is the executing different pipelines from the same schedule implemented in dagster 0.13.1/ latest version of dagster? Do I have to define a separate schedule for each pipeline, even if the schedules will execute at the same time?
It looks like
and targeting multiple but
does not