Hi Dagster folks! My team likes dagster but I've ...
# announcements
Hi Dagster folks! My team likes dagster but I've got some questions from them! """ The problem is that our use case sits somewhere between the workflow execution problem, and the streaming data problem (we have a stream of data to execute a workflow on), and so nothing solves our problem fully. If dagster doesn't work we think we're left with writing our own orchestration using celery, or moving to a streaming approach, but we really don't like how hard to maintain/test either of those options will be. For dagster we want to use the dependency/output management, but need to run far more pipelines/tasks than it is usually designed for (and batching things together means we lose a lot of the useful features). Three questions that I think I need to answer: What's the per task overhead? I think I we can probably deal with anything up to ~1s (airflow was giving us 5-10s overhead per tasks) Can we programmatically kick off pipelines on some external trigger? (Ideally within the framework, but having to write a lambda to do it wouldn't be terrible). It looks like we can with the GraphQL API? How many pipelines/tasks can run at once? Our pipeline should take ~1 minute to run. It would be nice to be able to process 100-1000 images/minute, but that would mean 1000 pipeline runs per minute, and 10-40 task runs in that time. Airflow couldn't handle that, but could dagster? If you have insight into them that would be appreciated! """
interesting!! as to 2) you can certainly programmatically execute pipelines, either with GraphQL or the Python APIs.
you will have to write your own trigger
though one thing we've considered/sketched is a deployment & execution story that would really level lambda
for 2) and 3), i think the bottleneck will be the transaction processing in the event log storage -- and postgres can certainly be tuned to handle 40k writes per minute
this would certainly stress dagster along some axes that haven't yet been tested
you would need to figure out a parallel deployment story to handle 1k pipeline runs per minute, probably using the dagster-celery workers (or again, possibly on lambda/FaaS)
and i'm sure there's frontend stuff where this would imply making some performance improvements and UI changes
Thanks @max! ๐Ÿ™‚ I'll send this back to my team! Though off the bat i'm sure to be interested in k8s deployments with the celery workers!
We have a pretty flexible system, so we can make choices for how the pipeline run is processed and then how each step in the pipeline is processed. For instance given the constraints you mentioned we may choose to send each pipeline to a celery worker, then process all of the steps in that pipeline in process (to avoid process spin up overhead). These
components are pluggable so we can create new ones that attempt to meet your constraints.
I still struggle a bit to understand the difference between these 2 and when I would write my own version of one or the other
For example, for production one thing that I'm considering is using the Argo cluster that we have been using so far, it requires only a yaml file, so I guess I would need to convert the ExecutionPlan into this yaml equivalent, then each step in Argo would run in a container running an instance of dagster-graphql (I guess), so it can emit events back to the run master. Would something like this be done with RunLauncher or Engine?
Here let me draw a diagram quick to try to help. The answer in your case is potentially both. An
which would take the
distribute the steps to your Argo cluster. The
would come in to play for handling the process where the
would execute, if that should be on a different box than where
is hosted.
๐Ÿ™ 1
So given what you said @Fran Sanchez you might ned up with this
@alex you don't know how useful this diagram actually is!!
great glad its helpful
So, if I remember correctly, there is an equivalent to the RunLauncher at step level, right? The
i echo Fran's comment, thank you @alex this will be helpful to take back to my team! ๐Ÿ™‚
ya thats very new - but its basically a way to special case some steps. So the
determines the default behavior and then you can use the
machinery to special case some steps
The only one we have right now is taking
steps and submitting them to the appropriate cluster instead of handling those steps like the others
So, does it mean that it will execute another instance of the
or do you need to provide a special
-kind of class?
since its just a single step it should get executed directly and not go through any
(ideally, there may be some quirks currently, it is very new as i said)
I see, I'll have a look at the Spark example
So, in my case my
could be fairly simple, pretty much conversion to yaml and submission + monitoring of the remote workflow I guess.
๐Ÿ‘ 1
heres a diagram framed around the python code in dagster core


Everything else would be done in the Argo pods running
(I think this is what I need to run in these pods, right?)
fairly simple
Ya I think the hard part will be figuring out how to do the translation
Agree, I would need to borrow some of the conventions from the dask engine or celery to define resource limits, images, etc.
I think this is what I need to run in these pods, right?
I donโ€™t know enough about Argo to say anything with confidence.
No, I mean frm dagster point of view. I have seen how the k8s RunLauncher command is
so I guess that this is what I need to run in every step to stick to dagster expectations
we are in the middle of changing that a bit - but there will exist some roughly equivalent dagster cli command to invoke that specifies a run id and the step in that run to execute
@johann can keep you up to date as he makes progress
Ok, yes I was referring to this
Copy code
                        'runId': run.run_id,
                        'repositoryName': external_pipeline.handle.repository_name,
                        'repositoryLocationName': external_pipeline.handle.location_name,
One last thing, as of today, is it possible to configure the k8s job RunLauncher to execute every step in a separate k8s job?
We currently have the
which will end up submitting each step as a separate K8s job - but does so via celery queues to provide a means for global resource constraints
we have not yet made a
which just directly submits the jobs, but plan to in the near future
Ok, so it would be implemented as different
handling all the steps
I expect we should have it in the next few weeks* since most of the building blocks are all available *assuming some other refactors that take precedence go well
Awesome, I think I will start with a simple RunLauncher running everything in a single k8s job with multiprocessing, then switch to this once is ready and in parallel evaluate the feasibility/benefits of using Argo instead
๐Ÿ‘ 1
@alex you were extremely helpful!! Thanks
And please, add something like your diagram to the internal docs, so much easier to understand