When using op selection syntax, is it possible to ...
# ask-community
l
When using op selection syntax, is it possible to exclude one or more ops (and include the rest)?
y
Hi Leo, we don’t have a syntax supporting exclude rules at the moment. but curious to hear your thoughts! do you have anything in mind that what syntax you’d be writing to exclude one or more ops?
l
dbt has an exclude flag that has the same section rules as its
select
flag. Maybe dagster can do something similar? For reference, I was thinking this could be useful for a case where I am creating a weekly schedule based on an hourly schedule. The weekly schedule does a full refresh of a subset of some of the hourly tasks. I don't want it to target all of the hourly tasks because some of the them would take too long to fully refresh. So being able to exclude some tasks from the weekly schedule would make it easier to define and maintain the schedule.
y
Got it. Thanks for the content. I’m currently working on improving op selection. Will spend some more time exploring the exclude options while doing that.
l
Sounds good, thanks! Let me know if you have any additional questions as you explore this.
❤️ 1
@yuhan, I have an additional question. If I have a Job that contains nested graphs, is it possible to select specific subgraphs in a schedule definition? If this doesn't exist, I think could be helpful to have a
graph_selection
argument (similar to the
solid_selection
argument)
y
We don't have that functionality yet. This is what I'm actively working on building. Will keep you posted with the progress!
l
Perfect 🙂
@yuhan I just saw that the latest dagster release includes op selection inside subgraphs. This is really great, so I would like to thank you and the team 🙂 I'm also wondering, have you had a chance to explore exclude options for op selection?
y
Thanks @Leo Kell - I haven't had time to explore the exclude but it's def on the plan! Here's the tracking issue: https://github.com/dagster-io/dagster/issues/6351
@Leo Kell one more follow up question: are you using dagit’s selection text input or op_selection api? i was prioritizing the dagit flow, under the assumption that the dagit flow of this has more usage, but i could be wrong.
l
@yuhan sorry for the late reply! I am actually using both the dagit interface and the
op_selection
API (particular in the Graph
to_job
method). I use the dagit selection for development and troubleshooting certain ops, and I the
op_selection
API to create specific pipelines run specific op subsets. In terms of the exclude feature, I would have more immediate usage for it within the op_selection API.
y
got it. thanks for the info!
👍 1