It seems like for the celery executor case, the la...
# announcements
n
It seems like for the celery executor case, the launcher would just be a no-op since all it's going to do is push a message into rabbit
a
Ya the run launcher just lets you control where the run coordinating process happens. If using celery - it’s pretty light weight and the default of “sub process on dagit box” is likely fine
n
Does the run coordinator stay up the whole time or just fork, fire celery task, exit?
And if the latter, can that just be removed? I could write one that is a literal no-op I think.
a
It stays up - it has to monitor the results of each step to determine if its dependencies should be submitted
n
Hmm, that seems not good
I am trying to avoid anything which launches a long-running thing immeidately for each DAG since I might dump in thousands all at once
Hence celery executor
Can I just not use the launcher system? I can't tell what it actually gets me if I don't need any resources allocated
(I would have assumed the thing watching the results of each step was the executor, not the coordinator)
a
Ya I could be clearer with terms - so the “run coordinator process” is the process that is running the Executor code. So the run launcher controls where the executor runs, and the executor controls where each step / solid / node in the dag runs
n
I thought the whole point of the celery executor was the executor runs in a celery task?
a
No it’s to have the steps run in celery tasks
n
Oh, hmm
a
Which it does sound like what we have might not be a good fit for your constraints
n
Is there any recommended option that can handle very large bursts of traffic? It seems like all the existing stuff would likely melt down if handed thousands of executions
I could probably look at the existing launcher examples and build what I thought the celery executor does 🙂
Basically just wrap the existing one into a celery task
a
Yeah it is a pluggable component so you should be able to write one that does that
The default one is relatively complicated - and your custom one can be much simpler if you can make assumptions about things such as how to load the pipeline
This is a reasonable request to be added to the dagster-celery library so feel free to file an issue as well
n
Unrelated but while I've got you, are there any docs for the network API? there must be one for the CLI to use but the docs don't seem to mention it.
It looks like it's just a few grpc calls?
a
What network apis are you lookin for / what’s the context? The grpc stuff is very fresh only used if you opt in
n
Having another web service kick off pipelines
a
Ah best to just use graphql - should be able to hit /graphql where ever you are hosting dagit to explore the schema use the playground
n
This whole thing is an image processing system, user uploads some images to our backend, it launches a suite of ML stuff, eventually calls a webhook to tell the backend it's done
Ahh okay, I usually think of GraphQL as for read operations 🙂
a
The dagster-graphql cli has some predefined queries& mutations and has a —remote flag for sending them to another box
Could def use better docs
n
Found 'em,
LAUNCH_PIPELINE_EXECUTION_MUTATION
Thanky 🙂
a
Good luck!
n
I'm sure I'll have more questions when I start trying to write the launcher 🙂 The API is very simple but those objects look big and full of many complexities
a
Sounds good