Is there any chance there's future support for visualizing an executed pipeline in dagit that was called with

execute_pipeline

from a solid in another pipeline? I've tried lots of different ways to get Dagster to run a schedule for a pipeline that needs to be re-generated fresh when the scheduler starts, but haven't been able to make any progress (lots of config, timing, and invariant errors -- tried sensors with no luck, assets, etc). For the time being, I'm simply going to have a helper pipeline that calls the code required to generate the dynamic pipeline then calls

execute_pipeline

and it would be great if this dynamic pipeline appeared in the runs tab. Let me know! or let me know if you're someone who's managed to get Docker + Dagster working with dynamic pipeline generation (note: I don't mean the dynamic output but defining the entire pipeline up front before it's executed).

we execute pipelines from jupyter with execute_pipeline and can see the runs in dagit after. try something like this

import yaml
from dagster import execute_pipeline, reconstructable, DagsterInstance
from my_repo import my_pipeline

res = execute_pipeline(
    reconstructable(my_pipeline),
    mode='my_mode',
    run_config=yaml.safe_load(open('my_config.yml')),
    instance=DagsterInstance.get(),
)

I will try this -- thanks!

the main thing here is to bind to the existing dagster instance with

DagsterInstance.get()

I see! okay, will try right now

Thomas did you look at the API docs for

execute_pipeline

? Im sure we could make it clearer there if we use the right phrasing

I had looked at the docs but wasn’t clear to me — also it highlighted using it for testing so I sort of tried to come up with another solution for a while to avoid using stuff designed for testing in my prod code. But I’m only on week 2 here so probably missed it — apologies about that. My default assumed all executed pipelines would appear, so when it didn’t I figured it was just something to work around.