https://dagster.io/ logo
#dagster-support
Title
# dagster-support
a

Arun Kumar

06/03/2021, 8:26 PM
Hi team, is it possible to set different tag values for different runs of the same pipeline? If yes, is there a way to search and filter the runs in the Dagit by the tag values?
a

alex

06/03/2021, 8:28 PM
yes and yes
the runs page in dagit has a search bar and clicking a tag should do a matching tag search if you want to see an example
a

Arun Kumar

06/03/2021, 8:44 PM
Thanks Alex. Do we have any examples on how to set the tag values while creating a pipeline run from sensors and dagit?
a

alex

06/03/2021, 8:46 PM
in dagit there is a tags editor in the playground in sensors
RunRequest
has a tags arg
a

Arun Kumar

06/03/2021, 9:09 PM
Ah got it. Thanks a lot for your help. Would also love to get your additional inputs on our pipeline design. Context: For our usecase, we have a set of entities stored in DB that needs to be processed and I am currently creating a separate pipeline to process each entity by dynamically generating the pipeline with the config set at definition using
configured
(instead of creating a single pipeline and creating a new run for each entity). I thought this would give us separate visibility into each entity process and also make it easier to manage them. But with this design, we might have to update the user-code frequently (al least daily) to include the new entities in the DB. Now I am thinking if we should just create a single configurable pipeline and pass the entity details through the config and tags and create different runs for different entities. With this design, the sensors can directly read the entities from the DB and trigger the runs and the user code need not be changed to include new entities.
@alex Just wanted to check again if you have any thoughts on this
a

alex

06/04/2021, 7:16 PM
yea this direction you are heading sounds good, a more static pipeline that varies its behavior based on config should go smoother for the reasons you mentioned of reducing user code changes
without knowing the details, I think there are two approaches to consider: * requesting multiple pipeline runs, one for each entity * submitting one pipeline run that uses dynamic mapping, where each entity will be a dynamic subtree of the pipeline https://docs.dagster.io/_apidocs/dynamic#dynamic-mapping-collect-experimental
a

Arun Kumar

06/04/2021, 7:22 PM
Thanks Alex, this is helpful. I might have to request multiple pipeline runs, because each entity has some dependencies which must be satisfied before starting the process on it. I am going to have a single sensor which will check the dependencies for all of the entities by calling an API on each and triggering a pipeline only for those that are satisfied. I was initially concerned about the performance of having a single sensor looping through all the entities, but I think it should be fine.