wave Is it possible to have one solid return a value that is

Question

wave Is it possible to have one solid return a value that is to be used as a resource config value for another solid downstream

alex · Answer

solid outputs can only be consumed as inputs so if i am interpreting your question correctly no is the answer

alex · Answer

what the problem you are trying to solve

benyuel · Answer

long story but I was going to see if I could have solid a gt launch emr cluster and return cluster id solid b gt run pyspark code using emr step launcher resource with cluster id from solid a solid c gt terminate emr cluster

benyuel · Answer

The `emr step launcher` resource assumes you have a cluster id already so solid a would orchestrate creating that cluster

alex · Answer

hmm i see ya we ve talked about this sort of thing

alex · Answer

what you can do is write a new ` resource` that returns the `EmrPySparkStepLauncher` that does what you describe

alex · Answer

you can have context manager style ` resource` functions that can `yield` their resource and then will execute the code after the yield during pipeline shutdown allowing you to teardown your cluster

benyuel · Answer

I think I ve tried that but only one solid then executes on each cluster if I have more than one pyspark solid i e each pyspark solid then gets a unique emr cluster If I m understanding you right

alex · Answer

ah shoot ya this is only really valid with the `in process` executor

benyuel · Answer

I suppose I could maybe create the cluster prior to the pipeline definition and set the cluster id as a global var that the pipeline definition would then use

alex · Answer

ya doing it as solids is a bit risky since failures will cause down stream solids to skip

benyuel · Answer

ok so you d recommend any pipeline with more than one pyspark solid to have the cluster created outside of the pipeline then

alex · Answer

probably for now this is a problem we know we want to solve but i think we dont have the tools needed in the framework yet

alex · Answer

which executor are you using

benyuel · Answer

gt which executor are you using not sure I follow still new to dagster

alex · Answer

the `Executor` decides how to federate out each step it controlled by the `execution` key in your run config yaml dict

benyuel · Answer

Where would I find that I m not explicitely setting it anywhere in the logs

alex · Answer

ya you may see some `EngineEvent` s report the name

alex · Answer

if you are not setting it defaults to in process which would work with the solution i described above if you are doing multiprocess there may be a way to do the context manager style resource in conjunction with using the file system env var for coordination to have all the solids use the same cluster id

benyuel · Answer

ok yeah I think I am using in process

benyuel · Answer

I will take another look though when I retry to confirm

benyuel · Answer

thanks for the help

benyuel · Answer

1 follow up question based on the above is the behavior for the `in process` executor that a resource is only instantiated once even if multiple solids require the same resource

alex · Answer

Yea for in process that should be true