I'd like to understand dagster ECSrunlauncher and ...
# ask-community
m
I'd like to understand dagster ECSrunlauncher and what it means by "task" - if I materialize an asset that depends on upstream assets A->B->C - if I have a singleton created in A - would C be able to access the state generated in A? Or is each asset materialization run on a different ECS task - and C can't know what A created in memory. How do we securely keep auth tokens generated in A's task for C? We can't auth too many times (the api we are working with will not permit it). Even if we made a resource rather than a singleton - the state wouldn't persist across ecs tasks. How are people solving this issue?
🙏 1
j
hey Matthew!
I'd like to understand dagster ECSrunlauncher and what it means by "task" - if I materialize an asset that depends on upstream assets A->B->C - if I have a singleton created in A - would C be able to access the state generated in A? Or is each asset materialization run on a different ECS task - and C can't know what A created in memory.
"task" is referring to ECS Task (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html) in dagster a RunLauncher means that whatever is launched will be scoped for the entire JobRun (a StepLauncher would be scoped to every op). So in your example if you're creating side effects in op or asset A you should be able to access those from the materialization of asset C, however i wouldn't really recommend doing this and would recommend relying on a declarative pattern for defining that dependency
How do we securely keep auth tokens generated in A's task for C? We can't auth too many times (the api we are working with will not permit it). Even if we made a resource rather than a singleton - the state wouldn't persist across ecs tasks. How are people solving this issue?
im not aware of examples of people doing this but you could encrypt decrypt the tokens and keep them as inputs/outputs of ops, you could also store them in some external datastore and pass references between the assets/ops