Hello everyone! Thanks for the great work on Dagst...
# ask-community
m
Hello everyone! Thanks for the great work on Dagster! In my use case, I have ~40 different monthly files to integrate, for which I like to use a single monthly partitioned job with dynamically generated ops, for the convenience of having the beautiful partition dashboard in Dagit to monitor the integration status. These files arrive at random dates, so I have another job to check for new files and download them, running on a daily schedule. When new files are in, the downloading job start a run of the monthly partitioned job for integration, only for the few files which arrived on that day, through the graphQL API just as Dagit does, which is the only way I found. The graphQL solution seems a bit hacky and I would prefer to switch to an asset materialization sensor, which seems close to be fit for this case. However RunRequest object does not allow to specify a subset of ops to run. Would it be possible to add an op_subset argument to RunRequest? Or is graphQL solution just fine (I am worried about stability)? Or do you see other solutions? Thanks a lot!
p
Hi Martin. I think this seems like a reasonable request… I’ll file a GH issue to track this.
Actually, this was just requested in this issue: https://github.com/dagster-io/dagster/issues/8209. Feel free to upvote and follow along.
m
Thanks a lot! Sorry I missed this issue.