Hello dagster beginner here 👋 I am wondering how dagster integrates with asyncio. I see you can define an async

op

, but you can't define an async graph? at least I didn't see how I could use

DynamicOutput

together with an async

op

. We also use async clients, For instance using

aiohttp.ClientSession

. This is generally used by an async contextmanager. I want to define this as a resource (I saw it is done this way for non-async clients in the hackernews example). How should I do this in dagster?

dagster does not yet drive parallel execution of ops within a single process. Currently you have to use multiprocess execution to achieve parallel

@op

execution https://github.com/dagster-io/dagster/issues/4041

I am not an expert on how this then works with the multiprocess executor 🙂 In case you use a multiprocess executor does this mean that all asyncio calls run in their own event loop (i.e. each process has a different event loop)? or are these different threads and can there be issues because of the non-threadsafe behavior of asyncio?

in both in-process serial execution or multiprocess parallel, each

@op

will be run with its own event loop