Can someone help me this Getting the error on docker compose

Question

Can someone help me this Getting the error on docker compose up ```dagster user code | Traceback most recent call last dagster user code | File usr local lib python3 9 site packages dagster core code pointer py line 86 in load python file dagster user code | return import module from path module name python file dagster user code | File usr local lib python3 9 site packages dagster seven init py line 50 in import module from path dagster user code | spec loader exec module module dagster user code | File lt frozen importlib bootstrap external gt line 850 in exec module dagster user code | File lt frozen importlib bootstrap gt line 228 in call with frames removed dagster user code | File repository py line 3 in lt module gt dagster user code | from demo jobs tutorials say hello import say hello job dagster user code | ModuleNotFoundError No module named demo dagster user code | dagster user code | The above exception was the direct cause of the following exception dagster user code | dagster user code | Traceback most recent call last dagster user code | File usr local bin dagster line 8 in lt module gt dagster user code | sys exit main dagster user code | File usr local lib python3 9 site packages dagster cli init py line 50 in main dagster user code | cli auto envvar prefix=ENV PREFIX pylint disable=E1123 dagster user code | File usr local lib python3 9 site packages click core py line 1130 in call dagster user code | return self main args kwargs dagster user code | File usr local lib python3 9 site packages click core py line 1055 in main dagster user code | rv = self invoke ctx dagster user code | File usr local lib python3 9 site packages click core py line 1657 in invoke dagster user code | return process result sub ctx command invoke sub ctx dagster user code | File usr local lib python3 9 site packages click core py line 1657 in invoke dagster user code | return process result sub ctx command invoke sub ctx dagster user code | File usr local lib python3 9 site packages click core py line 1404 in invoke dagster user code | return ctx invoke self callback ctx params dagster user code | File usr local lib python3 9 site packages click core py line 760 in invoke dagster user code | return callback args kwargs dagster user code | File usr local lib python3 9 site packages dagster cli api py line 575 in grpc command dagster user code | server = DagsterGrpcServer dagster user code | File usr local lib python3 9 site packages dagster grpc server py line 844 in init dagster user code | self api servicer = DagsterApiServer dagster user code | File usr local lib python3 9 site packages dagster grpc server py line 224 in init dagster user code | self loaded repositories = LoadedRepositories dagster user code | File usr local lib python3 9 site packages dagster grpc server py line 98 in init dagster user code | loadable targets = get loadable targets dagster user code | File usr local lib python3 9 site packages dagster grpc utils py line 33 in get loadable targets dagster user code | else loadable targets from python file python file working directory dagster user code | File usr local lib python3 9 site packages dagster core workspace autodiscovery py line 26 in loadable targets from python file dagster user code | loaded module = load python file python file working directory dagster user code | File usr local lib python3 9 site packages dagster core code pointer py line 100 in load python file dagster user code | raise DagsterImportError dagster user code | dagster core errors DagsterImportError Encountered ImportError `No module named demo ` while importing module repository from file opt dagster app repository py Local modules were resolved using the working directory ` opt dagster app` If another working directory should be used please explicitly specify the appropriate path using the ` d` or ` working directory` for CLI based targets or the `working directory` configuration option for `python file` based workspace targets ```

Kayvan Shah · Answer

The repo structure

Kayvan Shah · Answer

The user code dockerfile ```FROM weather combiner WORKDIR opt dagster app ADD demo opt dagster app Install the Dagster DAG package ADD setup py RUN pip install COPY demo repository py Run dagster gRPC server on port 4000 EXPOSE 4000 CMD allows this to be overridden from run launchers or executors that want to run other commands against your repository CMD dagster api grpc h 0 0 0 0 p 4000 f repository py ```

johann · Answer

gt ```No module named demo ` while importing module repository from file ``` It looks like your workspace yaml is pointing at a module which doesn t exist in you imag

Kayvan Shah · Answer

Fixed the issue there was some problem building the image

Kayvan Shah · Answer

Docker networks and other containers are spawned and UI DAGs and other related data is also visible But when a flow is triggered it fails as it is not able find the environment variables I have passed the environment variables via docker compose file to the user code container amp to the docker executor by dagster yaml file

Kayvan Shah · Answer

```KeyError CUSTOM WEATHER API CLIENT File usr local lib python3 9 site packages dagster grpc impl py line 82 in core execute run recon pipeline get definition File usr local lib python3 9 site packages dagster core definitions reconstruct py line 180 in get definition defn = self repository get definition get pipeline self pipeline name File usr local lib python3 9 site packages dagster core definitions reconstruct py line 83 in get definition return repository def from pointer self pointer File usr local lib python3 9 site packages dagster core definitions reconstruct py line 686 in repository def from pointer target = def from pointer pointer File usr local lib python3 9 site packages dagster core definitions reconstruct py line 607 in def from pointer target = pointer load target File usr local lib python3 9 site packages dagster core code pointer py line 176 in load target module = load python file self python file self working directory File usr local lib python3 9 site packages dagster core code pointer py line 86 in load python file return import module from path module name python file File usr local lib python3 9 site packages dagster seven init py line 50 in import module from path spec loader exec module module File lt frozen importlib bootstrap external gt line 850 in exec module File lt frozen importlib bootstrap gt line 228 in call with frames removed File repository py line 3 in lt module gt from demo jobs tutorials say hello import say hello job File opt dagster demo init py line 1 in lt module gt from repository import demo yahoo finance weather data pipelines File opt dagster demo repository py line 4 in lt module gt from demo jobs weather builder import spire current weather builtin File opt dagster demo jobs weather builder py line 3 in lt module gt from demo ops weather builder import get class object run pipeline File opt dagster demo ops weather builder py line 3 in lt module gt from weather combiner data pipelines import CurrentWeather File usr local lib python3 9 site packages weather combiner data pipelines init py line 1 in lt module gt from data pipeline import WeatherDataPipeline File usr local lib python3 9 site packages weather combiner data pipelines data pipeline py line 6 in lt module gt from weather combiner api integration import API File usr local lib python3 9 site packages weather combiner api integration init py line 1 in lt module gt from cc api integration import API File usr local lib python3 9 site packages weather combiner api integration cc api integration py line 4 in lt module gt from weather combiner logging import get logger File usr local lib python3 9 site packages weather combiner logging init py line 1 in lt module gt from loggers import get simple logger get logger File usr local lib python3 9 site packages weather combiner logging loggers py line 4 in lt module gt from weather combiner logging configuration import setup logging File usr local lib python3 9 site packages weather combiner logging configuration py line 5 in lt module gt from weather combiner utils import read config File usr local lib python3 9 site packages weather combiner utils init py line 5 in lt module gt from weather combiner utils BaseConfig import ClimateConnectAPIConfig File usr local lib python3 9 site packages weather combiner utils BaseConfig py line 19 in lt module gt class CustomWeatherAPIConfig File usr local lib python3 9 site packages weather combiner utils BaseConfig py line 23 in CustomWeatherAPIConfig client name = os environ CUSTOM WEATHER API CLIENT File usr local lib python3 9 os py line 679 in getitem raise KeyError key from None```

Kayvan Shah · Answer

DockerRunLauncher Launching run in a new container 15aa5944071b209c3fc838854eaa3d32227de9b8073bc3eabd0013b7ab5f5ed6 with image dagster demo

Kayvan Shah · Answer

<https docs dagster io apidocs libraries dagster docker> What local environment in env vars under docker run launcher mean here

Kayvan Shah · Answer

Since I have already declared these env variables in docker compose yaml file and as per the docs I simply used them dagster yaml file Now here it is able to get the database related env varibales but not the others

johann · Answer

You ll want to configure the Docker run launcher to pass the env

johann · Answer

See `env vars` on <https docs dagster io apidocs libraries dagster docker dagster docker DockerRunLauncher>

Kayvan Shah · Answer

Yeah got the docker deployment working But here a major issue with this is that your environment variables or secrets gets exposed in the dagster yaml file

johann · Answer

If you just pass the name of the variable instead of the `var=value` syntax it will use the value present in the daemon container

Kayvan Shah · Answer

Oh I had embedded these under the user code s container which is why it wasn t to get those env variables