Is it possible to execute jobs against particular values from within an op in another job/asset materialization? I think i'm at a place where it's not possible to statically render this graph and these values apriori, hence i'm not sure I can avoid needing to dynamically invoke jobs but i'm not sure, here's what we're trying to do We're doing a hyper-parameter optimization, and the key thing is that we need to sample the search space incrementally so that subsequent parameter suggestions are informed by results from previous trials. For example, without loss of generality say we start by picking

6

points from the search space at random, we then run

6

dagster graphs representing our trials in parallel against each of these points, the results of these

6

trials are recorded, and now we start the next batch of sampling 6 points and running the associated 6 trials. Our sampling of the subsequent

6

points is actually informed by the results that have been recorded thus far, so it's important to stagger the trials into epochs like this. I can eagerly and statically inform dagster of the subgraph that represents running a single trial, so what I was thinking is I could have an

op

that is responsible for sampling the search-space, then running the trial job, recording the results, then generate subsequent samples from the space, etc. Even if possible the thing I don't totally like is that logically it's like the trial subgraphs are child ops of this meta

op

that is sampling the search-space, but dagster's data model will not be aware of this, this logical connection only exists by way of the meta-op dynamically invoking the execution of those trial graphs/jobs If any of this is ill-advised, or not possible, or there is some better way to accomplish this please let me know.

Hey @Clement Emmanuel - have you seen https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions#dynamically-partitioned-assets? Each trial could be its own dynamic partition This won't address "Even if possible the thing I don't totally like is that logically it's like the trial subgraphs are child ops of this meta

op

that is sampling the search-space, but dagster's data model will not be aware of this, this logical connection only exists by way of the meta-op dynamically invoking the execution of those trial graphs/jobs", but it might still be useful for you?