I have a few assets that are loaded from a high-latency source. I can load them asynchronously. Is there a way to yield a process back to the dagster job executor while the data is being loaded so that something else can be run in the meanwhile, and have that asset added back to the op queue at a later time once the data is loaded? I haven't been able to find anything about this in the docs. Thank you.

I think you could add asset dependency and Dagster will figure out the DAG to run on itself.

Thanks, yes dagster has been able to figure out the dag. I may be misunderstanding your point, but what I'm wondering is whether there's support for asynchronous calls in assets.

I’m not an official Dagster support but I guess you could you use

tag_concurrency_limits

. So you could set the max concurrent ops higher, and tag all ops into

normal

and

slow

, and we could set max concurrent run for

normal

op to 4 and

slow

to a higher number.

Thanks, yes that does seem like an option. Not quite what I'm looking for, because I could still get unlucky and have all the slow ops hit computational intensive work at the same moment, etc.

hey @Noam Finkelstein doing async calls isn’t something i’ve experimented with much, and i don’t know of anyone who has, so i’m not going to be able to give a ton of helpful advice. we have one community contribution that does make some async calls, but it’s a pretty dense integration to parse so it may not be super useful as an example. I’m happy to pass it along if you’d like though! My best advice would be to try to set up a small test case and see what the behavior is

Thanks @jamie! I'd love to see the code you mentioned, if you wouldn't mind sharing a link.

sure thing - here’s the repo https://github.com/noteable-io/papermill-origami/tree/main/papermill_origami/noteable_dagstermill - the async call is here https://github.com/noteable-io/papermill-origami/blob/main/papermill_origami/noteable_dagstermill/engine.py#L6

Thanks!