https://dagster.io/ logo
#ask-community
Title
# ask-community
n

Noam Finkelstein

01/30/2023, 9:02 PM
I have a few assets that are loaded from a high-latency source. I can load them asynchronously. Is there a way to yield a process back to the dagster job executor while the data is being loaded so that something else can be run in the meanwhile, and have that asset added back to the op queue at a later time once the data is loaded? I haven't been able to find anything about this in the docs. Thank you.
s

Sean Han

01/30/2023, 9:39 PM
I think you could add asset dependency and Dagster will figure out the DAG to run on itself.
n

Noam Finkelstein

01/30/2023, 9:41 PM
Thanks, yes dagster has been able to figure out the dag. I may be misunderstanding your point, but what I'm wondering is whether there's support for asynchronous calls in assets.
For example, if I set the executor to run a maximum of 4 concurrent ops, and one of the 4 assets has made a slow database call, I'd like to be able to tell the executor it can use the corresponding process for a different op until the database call returns. Is there a way to do that?
s

Sean Han

01/30/2023, 9:45 PM
I’m not an official Dagster support but I guess you could you use
tag_concurrency_limits
. So you could set the max concurrent ops higher, and tag all ops into
normal
and
slow
, and we could set max concurrent run for
normal
op to 4 and
slow
to a higher number.
n

Noam Finkelstein

01/30/2023, 9:46 PM
Thanks, yes that does seem like an option. Not quite what I'm looking for, because I could still get unlucky and have all the slow ops hit computational intensive work at the same moment, etc.
j

jamie

01/31/2023, 3:46 PM
hey @Noam Finkelstein doing async calls isn’t something i’ve experimented with much, and i don’t know of anyone who has, so i’m not going to be able to give a ton of helpful advice. we have one community contribution that does make some async calls, but it’s a pretty dense integration to parse so it may not be super useful as an example. I’m happy to pass it along if you’d like though! My best advice would be to try to set up a small test case and see what the behavior is
n

Noam Finkelstein

01/31/2023, 5:17 PM
Thanks @jamie! I'd love to see the code you mentioned, if you wouldn't mind sharing a link.
n

Noam Finkelstein

01/31/2023, 5:55 PM
Thanks!