I’m merging multiple assets (CSVs) into a master f...
# ask-community
m
I’m merging multiple assets (CSVs) into a master file and I want to implement a sensor that listens for the existence of such master file and recreates it upon deletion. This would be done by triggering 4 full pipelines (that each creates one of the required assets) if those files do not exist, or executing the merge solid only if they already exist. So is there a way to 1. Have a single sensor yield
RunRequests
for more than one pipeline and 2. Define a
solid_selection
with logic inside the sensor (all solids if file does not exist, merge only if it exists), rather than passing it as an argument to the sensor definition? or do I need to define an extra pipeline that does all this?
I have sensors for the other files as well. So I guess I could guarantee they always exist with something like the snipper below. This way I can only merge them if the master file gets deleted.
a
targeting multiple pipelines from one schedule/sensor is not a pattern we currently support but are looking to in the future
j
@alex is the executing different pipelines from the same schedule implemented in dagster 0.13.1/ latest version of dagster? Do I have to define a separate schedule for each pipeline, even if the schedules will execute at the same time?
a
It looks like
sensor
supports
jobs
and targeting multiple but
schedule
does not