When using databricks pyspark step launcher, I always see the following log:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Does this mean that the job is being set up locally? Shouldn't it be in databricks instead?

@Bernardo Cortez hm yes the resources will be initialized in both places unfortunately. the local resources (other than the step launcher) will not be used for anything during execution. does this issue cause a failure on your end, or is it more of a performance kind of problem?

It's a performance issue, it takes ~500Mb RAM just for launching a step...

yikes yeah that's pretty bad -- is this happening with a custom resource or an imported one?

Just importing the pyspark resource from dagster-pyspark. Yep, it's pretty resource intensive :) Any chance of not intializing locally?

https://github.com/dagster-io/dagster/issues/7253 created an issue for it, although my intuition is that this might be a bit tricky to solve. as a bandaid/hack on your end, you might be able to create a

hacked_pyspark_resource

, that is basically a copy/paste of the real one, but it reads an environment variable to determine if it will return None or an actual resource.

Ah, thats a good trick. Let me try that.