Can someone help me this? Getting the error on doc...
# ask-community
k
Can someone help me this? Getting the error on docker compose up
Copy code
dagster_user_code      | Traceback (most recent call last):
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 86, in load_python_file
dagster_user_code      |     return import_module_from_path(module_name, python_file)
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/seven/__init__.py", line 50, in import_module_from_path
dagster_user_code      |     spec.loader.exec_module(module)
dagster_user_code      |   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
dagster_user_code      |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
dagster_user_code      |   File "repository.py", line 3, in <module>
dagster_user_code      |     from demo.jobs.tutorials.say_hello import say_hello_job
dagster_user_code      | ModuleNotFoundError: No module named 'demo'
dagster_user_code      | 
dagster_user_code      | The above exception was the direct cause of the following exception:
dagster_user_code      | 
dagster_user_code      | Traceback (most recent call last):
dagster_user_code      |   File "/usr/local/bin/dagster", line 8, in <module>
dagster_user_code      |     sys.exit(main())
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/cli/__init__.py", line 50, in main
dagster_user_code      |     cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
dagster_user_code      |     return self.main(*args, **kwargs)
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
dagster_user_code      |     rv = self.invoke(ctx)
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
dagster_user_code      |     return _process_result(sub_ctx.command.invoke(sub_ctx))
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
dagster_user_code      |     return _process_result(sub_ctx.command.invoke(sub_ctx))
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
dagster_user_code      |     return ctx.invoke(self.callback, **ctx.params)
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
dagster_user_code      |     return __callback(*args, **kwargs)
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/cli/api.py", line 575, in grpc_command
dagster_user_code      |     server = DagsterGrpcServer(
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/grpc/server.py", line 844, in __init__
dagster_user_code      |     self._api_servicer = DagsterApiServer(
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/grpc/server.py", line 224, in __init__
dagster_user_code      |     self._loaded_repositories = LoadedRepositories(
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/grpc/server.py", line 98, in __init__
dagster_user_code      |     loadable_targets = get_loadable_targets(
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/grpc/utils.py", line 33, in get_loadable_targets
dagster_user_code      |     else loadable_targets_from_python_file(python_file, working_directory)
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/core/workspace/autodiscovery.py", line 26, in loadable_targets_from_python_file
dagster_user_code      |     loaded_module = load_python_file(python_file, working_directory)
dagster_user_code      |   File "/usr/local/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 100, in load_python_file
dagster_user_code      |     raise DagsterImportError(
dagster_user_code      | dagster.core.errors.DagsterImportError: Encountered ImportError: `No module named 'demo'` while importing module repository from file /opt/dagster/app/repository.py. Local modules were resolved using the working directory `/opt/dagster/app`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for `python_file`-based workspace targets.
The repo structure
The user code dockerfile
Copy code
FROM weather_combiner

WORKDIR /opt/dagster/app
ADD demo /opt/dagster/app

# Install the Dagster DAG package 
ADD setup.py .
RUN pip install .
COPY /demo/repository.py .

# Run dagster gRPC server on port 4000
EXPOSE 4000

# CMD allows this to be overridden from run launchers or executors that want
# to run other commands against your repository
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repository.py"]
j
Copy code
No module named 'demo'` while importing module repository from file
It looks like your workspace.yaml is pointing at a module which doesn’t exist in you imag
k
Fixed the issue there was some problem building the image
👍 1
Docker networks and other containers are spawned and UI, DAGs and other related data is also visible But when a flow is triggered it fails as it is not able find the environment variables I have passed the environment variables via docker compose file to the user code container & to the docker executor by dagster.yaml file
Copy code
KeyError: 'CUSTOM_WEATHER_API_CLIENT'

  File "/usr/local/lib/python3.9/site-packages/dagster/grpc/impl.py", line 82, in core_execute_run
    recon_pipeline.get_definition()
  File "/usr/local/lib/python3.9/site-packages/dagster/core/definitions/reconstruct.py", line 180, in get_definition
    defn = self.repository.get_definition().get_pipeline(self.pipeline_name)
  File "/usr/local/lib/python3.9/site-packages/dagster/core/definitions/reconstruct.py", line 83, in get_definition
    return repository_def_from_pointer(self.pointer)
  File "/usr/local/lib/python3.9/site-packages/dagster/core/definitions/reconstruct.py", line 686, in repository_def_from_pointer
    target = def_from_pointer(pointer)
  File "/usr/local/lib/python3.9/site-packages/dagster/core/definitions/reconstruct.py", line 607, in def_from_pointer
    target = pointer.load_target()
  File "/usr/local/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 176, in load_target
    module = load_python_file(self.python_file, self.working_directory)
  File "/usr/local/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 86, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/usr/local/lib/python3.9/site-packages/dagster/seven/__init__.py", line 50, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "repository.py", line 3, in <module>
    from demo.jobs.tutorials.say_hello import say_hello_job
  File "/opt/dagster/demo/__init__.py", line 1, in <module>
    from .repository import demo, yahoo_finance, weather_data_pipelines
  File "/opt/dagster/demo/repository.py", line 4, in <module>
    from demo.jobs.weather.builder import spire_current_weather_builtin
  File "/opt/dagster/demo/jobs/weather/builder.py", line 3, in <module>
    from demo.ops.weather.builder import get_class_object__, run_pipeline
  File "/opt/dagster/demo/ops/weather/builder.py", line 3, in <module>
    from weather_combiner.data_pipelines import CurrentWeather
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/data_pipelines/__init__.py", line 1, in <module>
    from .data_pipeline import WeatherDataPipeline
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/data_pipelines/data_pipeline.py", line 6, in <module>
    from weather_combiner.api_integration import API
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/api_integration/__init__.py", line 1, in <module>
    from .cc_api_integration import API
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/api_integration/cc_api_integration.py", line 4, in <module>
    from weather_combiner.logging import get_logger
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/logging/__init__.py", line 1, in <module>
    from .loggers import get_simple_logger, get_logger
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/logging/loggers.py", line 4, in <module>
    from weather_combiner.logging.configuration import setup_logging
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/logging/configuration.py", line 5, in <module>
    from weather_combiner.utils import read_config
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/utils/__init__.py", line 5, in <module>
    from weather_combiner.utils.BaseConfig import ClimateConnectAPIConfig
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/utils/BaseConfig.py", line 19, in <module>
    class CustomWeatherAPIConfig:
  File "/usr/local/lib/python3.9/site-packages/weather_combiner/utils/BaseConfig.py", line 23, in CustomWeatherAPIConfig
    client_name = os.environ["CUSTOM_WEATHER_API_CLIENT"]
  File "/usr/local/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
[DockerRunLauncher] Launching run in a new container 15aa5944071b209c3fc838854eaa3d32227de9b8073bc3eabd0013b7ab5f5ed6 with image dagster-demo
https://docs.dagster.io/_apidocs/libraries/dagster-docker What local environment in env_vars under docker run launcher mean here
Since I have already declared these env variables in docker-compose.yaml file and as per the docs I simply used them dagster.yaml file Now here it is able to get the database related env varibales but not the others
j
You’ll want to configure the Docker run launcher to pass the env
1
k
Yeah got the docker deployment working But here a major issue with this is that your environment variables or secrets gets exposed in the dagster.yaml file
j
If you just pass the name of the variable instead of the
var=value
syntax, it will use the value present in the daemon container
1
k
Oh I had embedded these under the user code's container which is why it wasn't to get those env variables