Hi Folks! I am trying to use Spark with Dagster as recommended in the tutorial but running into the error below.

dagster.core.errors.DagsterInvariantViolationError: spark not found at module scope in file /......./src/dagster_spark.py.
  File "/opt/anaconda3/lib/python3.7/site-packages/dagster/grpc/server.py", line 227, in __init__
    self._container_image,
  File "/opt/anaconda3/lib/python3.7/site-packages/dagster/grpc/server.py", line 103, in __init__
    loadable_target_origin.attribute,
  File "/opt/anaconda3/lib/python3.7/site-packages/dagster/grpc/utils.py", line 32, in get_loadable_targets
    if attribute
  File "/opt/anaconda3/lib/python3.7/site-packages/dagster/core/definitions/reconstruct.py", line 601, in load_def_in_python_file
    return def_from_pointer(CodePointer.from_python_file(python_file, attribute, working_directory))
  File "/opt/anaconda3/lib/python3.7/site-packages/dagster/core/definitions/reconstruct.py", line 607, in def_from_pointer
    target = pointer.load_target()
  File "/opt/anaconda3/lib/python3.7/site-packages/dagster/core/code_pointer.py", line 178, in load_target
    module, self.fn_name, f"at module scope in file {self.python_file}."
  File "/opt/anaconda3/lib/python3.7/site-packages/dagster/core/code_pointer.py", line 200, in _load_target_from_module
    raise DagsterInvariantViolationError(f"{fn_name} not found {error_suffix}")

Hi Sanidhya. Would you mind sharing which version of dagster you're running and also paste a link to the tutorial you mentioned? Not sure which resource you're referring to

Hi @claire I’m on 0.15.0 and this is the tutorial I tried https://docs.dagster.io/integrations/spark

Hi Sanidhya. Are you running this example? https://docs.dagster.io/integrations/spark#running-pyspark-code-in-ops The only modules you should need to install are dagster, dagit, and pyspark:

pip install dagster dagit pyspark

Hi Claire, That’s what I tried but it doesn’t work for me!