People who use Dagster for ML experimentation. If ...
# ask-community
People who use Dagster for ML experimentation. If you''re setting off a series of experiments which get triggered based on the results of ones before, do you: A) Use sensors to trigger multiple jobs B) Dynamically generate the job by creating ops (dynamic graphs or similar) I think my most common use case would be, continue running more extreme versions of the same experiment until no further model performance gain is achieved (bit hyperparameter sweep esk). Better question might be has anyone used dagster closely with a hyperparamter optimization framework like optuna to spin off a job for each trial? Each experiment is a multi-hour process so separating them out into their own jobs (with some distinguishing tag) feels most fitting.
🤖 1
I haven't implemented this myself, but the sensor approach sounds reasonable to me
Another option would be to, inside your Dagster job, invoke a framework like Ray, which I believe targets this kind of pattern