Hi! Is there a way to re-execute selected ops with...
# ask-community
m
Hi! Is there a way to re-execute selected ops with a different config? Consider the scenario then I want to do hyperparameter optimization and have a job like
extract_data -> prepare_data -> train_model
. The job in typically launched as a whole with a pre-defined config via the "Launchpad" or a sensor. But in some cases I want to tweak config of the
train_model
op specifically while reusing the output of
prepare_data
from the previous run. Of course I can split this job in two, explicitly passing prepared data between jobs, but it feels like repeating
LocalArtifactStorage
logic...
s
Hi @Mikhail Rozhkov - this isn't currently possible, but I filed an issue to track it: https://github.com/dagster-io/dagster/issues/7474. Software-defined assets might be helpful for you here once they support config, which should happen in a few weeks