Hello dagster beginner here wave I am wondering how dagster

Question

Hello dagster beginner here wave I am wondering how dagster integrates with asyncio I see you can define an async `op` but you can t define an async graph at least I didn t see how I could use `DynamicOutput` together with an async `op` We also use async clients For instance using `aiohttp ClientSession` This is generally used by an async contextmanager I want to define this as a resource I saw it is done this way for non async clients in the hackernews example How should I do this in dagster

Irven Aelbrecht · Answer

If you run for instance with multiple threads Should you make a resource from your event loop and use that to call the threadsafe run when running something with asyncio <https docs python org 3 library asyncio dev html concurrency and multithreading> What is the pattern generally used

alex · Answer

dagster does not yet drive parallel execution of ops within a single process Currently you have to use multiprocess execution to achieve parallel ` op` execution <https github com dagster io dagster issues 4041>

Irven Aelbrecht · Answer

I am not an expert on how this then works with the multiprocess executor slightly smiling face In case you use a multiprocess executor does this mean that all asyncio calls run in their own event loop i e each process has a different event loop or are these different threads and can there be issues because of the non threadsafe behavior of asyncio

alex · Answer

in both in process serial execution or multiprocess parallel each ` op` will be run with its own event loop