When using databricks pyspark step launcher I always see the

Question

When using databricks pyspark step launcher I always see the following log gt Using Spark s default log4j profile org apache spark log4j defaults properties Does this mean that the job is being set up locally Shouldn t it be in databricks instead

Bernardo Cortez · Answer

< owen> I am afraid this a step launcher related issue again I guess the problem is that the resources are being redundantly initialized both locally and in databricks isn t it

owen · Answer

< Bernardo Cortez> hm yes the resources will be initialized in both places unfortunately the local resources other than the step launcher will not be used for anything during execution does this issue cause a failure on your end or is it more of a performance kind of problem

Bernardo Caldas · Answer

It s a performance issue it takes ~500Mb RAM just for launching a step

owen · Answer

yikes yeah that s pretty bad is this happening with a custom resource or an imported one

Bernardo Caldas · Answer

Just importing the pyspark resource from dagster pyspark Yep it s pretty resource intensive Any chance of not intializing locally

owen · Answer

created an issue for it although my intuition is that this might be a bit tricky to solve as a bandaid hack on your end you might be able to create a `hacked pyspark resource` that is basically a copy paste of but it reads an environment variable to determine if it will return None or an actual resource

Bernardo Caldas · Answer

Ah thats a good trick Let me try that