It seems like for the celery executor case the launcher woul

Question

It seems like for the celery executor case the launcher would just be a no op since all it s going to do is push a message into rabbit

alex · Answer

Ya the run launcher just lets you control where the run coordinating process happens If using celery it s pretty light weight and the default of sub process on dagit box is likely fine

Noah K · Answer

Does the run coordinator stay up the whole time or just fork fire celery task exit

Noah K · Answer

And if the latter can that just be removed I could write one that is a literal no op I think

alex · Answer

It stays up it has to monitor the results of each step to determine if its dependencies should be submitted

Noah K · Answer

Hmm that seems not good

Noah K · Answer

I am trying to avoid anything which launches a long running thing immeidately for each DAG since I might dump in thousands all at once

Noah K · Answer

Hence celery executor

Noah K · Answer

Can I just not use the launcher system I can t tell what it actually gets me if I don t need any resources allocated

Noah K · Answer

I would have assumed the thing watching the results of each step was the executor not the coordinator

alex · Answer

Ya I could be clearer with terms so the run coordinator process is the process that is running the Executor code So the run launcher controls where the executor runs and the executor controls where each step solid node in the dag runs

Noah K · Answer

I thought the whole point of the celery executor was the executor runs in a celery task

alex · Answer

No it s to have the steps run in celery tasks

Noah K · Answer

Oh hmm

alex · Answer

Which it does sound like what we have might not be a good fit for your constraints

Noah K · Answer

Is there any recommended option that can handle very large bursts of traffic It seems like all the existing stuff would likely melt down if handed thousands of executions

Noah K · Answer

I could probably look at the existing launcher examples and build what I thought the celery executor does slightly smiling face

Noah K · Answer

Basically just wrap the existing one into a celery task

alex · Answer

Yeah it is a pluggable component so you should be able to write one that does that

alex · Answer

The default one is relatively complicated and your custom one can be much simpler if you can make assumptions about things such as how to load the pipeline

alex · Answer

This is a reasonable request to be added to the dagster celery library so feel free to file an issue as well

Noah K · Answer

Unrelated but while I ve got you are there any docs for the network API there must be one for the CLI to use but the docs don t seem to mention it

Noah K · Answer

It looks like it s just a few grpc calls

alex · Answer

What network apis are you lookin for what s the context The grpc stuff is very fresh only used if you opt in

Noah K · Answer

Having another web service kick off pipelines

alex · Answer

Ah best to just use graphql should be able to hit graphql where ever you are hosting dagit to explore the schema use the playground

Noah K · Answer

This whole thing is an image processing system user uploads some images to our backend it launches a suite of ML stuff eventually calls a webhook to tell the backend it s done

Noah K · Answer

Ahh okay I usually think of GraphQL as for read operations slightly smiling face

alex · Answer

The dagster graphql cli has some predefined queries amp mutations and has a remote flag for sending them to another box

alex · Answer

Could def use better docs

Noah K · Answer

Found em `LAUNCH PIPELINE EXECUTION MUTATION`

Noah K · Answer

Thanky slightly smiling face

alex · Answer

Good luck

Noah K · Answer

I m sure I ll have more questions when I start trying to write the launcher slightly smiling face The API is very simple but those objects look big and full of many complexities

alex · Answer

Sounds good