You don't have to call execute_pipeline

so how do i define the pipeline

if it's starting up from no definitions yet

like let's say my first pipeline before any pipeline code has been written

I think you misunderstand something deep

First you write some code

When when you start dagit, you point it at that code

(directly or via a repo)

It loads and evaluates the definitions as part of startup

i see

yes had no idea that this was the case

Unrelated question to the above: is there a reason SyncInMemoryLauncher uses a weakref of the instance passed in from initialze instead of the instance passed to launch_run? https://github.com/dagster-io/dagster/blob/1aa474e04ccd91b089cffe4a597799dab4de5c3a/python_modules/dagster/dagster/core/launcher/sync_in_memory_run_launcher.py#L46

this definitely explains it

I would have expected those to be the same underlying instance object

Ahh because instance in launch_run is new (https://github.com/dagster-io/dagster/commit/f711a5bce92f08c18f56fa20999366690c0b2788#diff-d959f2aebbc66e2c809f152cde30aaa341fa3da77c8bed38f2b31611bf5a84bb) and SyncInMemoryLauncher was just never fixed 😄

It's alive! (my bootleg celery run launcher)

Frankensteined some code from the CLI and dagster-k8s and hit it with a hammer until it did the thing

What's the right usage of marker_start and marker_end on EngineEvents?

Seems like it's for folding markers in dagit but I don't see that implemented?

Is there any way to split the pipeline definitions out so they are versioned independently? With the gRPC origin it seems like stuff is still cached in dagit/celery until they also restart

(via the _CacheingDefinitionIndex which has no mechanism for cache invalidation)

So to upload the definitions I would need to both restart the gRPC daemon and dagit+launchers

It seems like the workaround would be to subclass RepositoryData and hack it to have some kind of either background update or cache invalidation TTL or something like that

It seems like a straight TTL would be the simplest and probably best? But that risks dagit and launchers getting out of sync, so maybe the real answer is leave the cache and use immutable pipeline times of some kind? That would trick to work with though.

I'm a little confused about the part about dagit here - I don't think dagit has access to RepositoryData? That should only be used on the process that does the actual execution

It has to render the pipelines list and visual display

Does it not get that from the repo object?

Not directly - dagit is isolated from the actual definitions, I wouldn't expect the dagit process to ever load a RepositoryDefinition

there should be a button in the UI for reloading the repository data that dagit does use though

Gotcha, so my idea to subclass RepoData and add TTL-based caching seems pretty workable then?