https://dagster.io/ logo
Join Slack
Powered by
Hey team, having an issue implementing Docker with...
# ask-community
a
Hey team, having an issue implementing Docker with my pipeline. I'm currently seeing this error:
Copy code
Error loading repository location repo.py:FileNotFoundError: [Errno 2] No such file or directory: 'dbt': 'dbt'
Thought I define the directory in my docker file:
Copy code
FROM python:3.7-slim

RUN mkdir -p /opt/dagster/dagster_home /opt/dagster/app
RUN mkdir /opt/dagster/app/nhl_elt
RUN mkdir /opt/dagster/app/dbt

RUN pip install dagster dagit dagster-postgres dagster-dbt dagster-aws dagster-snowflake pandas requests

COPY dbt /opt/dagster/app/dbt/ 
COPY nhl_elt /opt/dagster/app/nhl_elt/
COPY workspace.yaml /opt/dagster/app/

ENV DAGSTER_HOME=/opt/dagster/dagster_home/

COPY dagster.yaml /opt/dagster/dagster_home/

WORKDIR /opt/dagster/app

EXPOSE 3000

ENTRYPOINT ["dagit", "-h", "0.0.0.0", "-p", "3000"]
and if I exec into the running container, the project looks as expected: ``````
This is more or less the same file structure that my local project takes on:
I've followed this example: https://docs.dagster.io/deployment/guides/docker#example for creating the Dockerfile
workspace.yaml
Copy code
load_from:
  - python_file: nhl_elt/repo.py
d
Hi Alec - if there was a stack trace along with that error, would it be possible to post it too?
a
Sure thing
Copy code
Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/server.py", line 217, in __init__
    self._container_image,
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/server.py", line 99, in __init__
    loadable_target_origin.attribute,
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/utils.py", line 33, in get_loadable_targets
    else loadable_targets_from_python_file(python_file, working_directory)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/autodiscovery.py", line 26, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file, working_directory)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/code_pointer.py", line 86, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/usr/local/lib/python3.7/site-packages/dagster/seven/__init__.py", line 47, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/dagster/app/nhl_elt/repo.py", line 4, in <module>
    from nhl_elt.jobs.dbt_transforms import dbt_transforms
  File "/opt/dagster/app/nhl_elt/jobs/dbt_transforms.py", line 8, in <module>
    dbt_assets = load_assets_from_dbt_project(project_dir = DBT_DIR, profiles_dir = DBT_DIR)
  File "/usr/local/lib/python3.7/site-packages/dagster_dbt/asset_defs.py", line 204, in load_assets_from_dbt_project
    project_dir, profiles_dir, target_dir, select or "*"
  File "/usr/local/lib/python3.7/site-packages/dagster_dbt/asset_defs.py", line 43, in _load_manifest_for_project
    target_path=target_dir,
  File "/usr/local/lib/python3.7/site-packages/dagster_dbt/cli/utils.py", line 67, in execute_cli
    process = subprocess.Popen(command_list, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)

  location_name=location_name, error_string=error.to_string()
DBT_DIR = 'dbt'
d
does it work any better if you change DBT_DIR to an absolute path? "/opt/dagster/app/dbt"? Might not be the root cause of the problem but could help figure out whats going on
a
Doesn't fix it but does shorten the stack trace:
Copy code
aryan-laptop:nhl-analytics-pipleine-dagster alecryan$ docker run nhl_dag:1.5  
E0503 13:25:12.266496765      10 <http://fork_posix.cc:70]|fork_posix.cc:70]>           Fork support is only compatible with the epoll1 and poll polling strategies
/usr/local/lib/python3.7/site-packages/dagster/core/workspace/context.py:563: UserWarning: Error loading repository location repo.py:FileNotFoundError: [Errno 2] No such file or directory: 'dbt': 'dbt'

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/server.py", line 217, in __init__
    self._container_image,
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/server.py", line 99, in __init__
    loadable_target_origin.attribute,
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/utils.py", line 33, in get_loadable_targets
    else loadable_targets_from_python_file(python_file, working_directory)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/autodiscovery.py", line 26, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file, working_directory)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/code_pointer.py", line 86, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/usr/local/lib/python3.7/site-packages/dagster/seven/__init__.py", line 47, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/dagster/app/nhl_elt/repo.py", line 4, in <module>
    from nhl_elt.jobs.dbt_transforms import dbt_transforms
  File "/opt/dagster/app/nhl_elt/jobs/dbt_transforms.py", line 8, in <module>
    dbt_assets = load_assets_from_dbt_project(project_dir = DBT_DIR, profiles_dir = DBT_DIR)
  File "/usr/local/lib/python3.7/site-packages/dagster_dbt/asset_defs.py", line 204, in load_assets_from_dbt_project
    project_dir, profiles_dir, target_dir, select or "*"
  File "/usr/local/lib/python3.7/site-packages/dagster_dbt/asset_defs.py", line 43, in _load_manifest_for_project
    target_path=target_dir,
  File "/usr/local/lib/python3.7/site-packages/dagster_dbt/cli/utils.py", line 67, in execute_cli
    process = subprocess.Popen(command_list, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)

  location_name=location_name, error_string=error.to_string()
It's still looking for a file/directory named 'dbt'
Full job code for dbt_transforms.py:
Copy code
from dagster import AssetGroup
from dagster_dbt import dbt_cli_resource
from dagster_dbt.asset_defs import load_assets_from_dbt_project

DBT_DIR = '/opt/dagster/app/dbt'

dbt_assets = load_assets_from_dbt_project(project_dir = DBT_DIR, profiles_dir = DBT_DIR)

dbt_asset_group = AssetGroup(dbt_assets,
resource_defs={
        'dbt': dbt_cli_resource
    }
)

dbt_transforms = dbt_asset_group.build_job(name="dbt_transforms")
d
What if you run 
docker run -it <your image here> dbt
? It seems like its having trouble finding the "dbt" executable
a
most likely that is the cas
*case
repo.py is located in nhl_elt/
weird that I don't have the issue outside of docker though
d
Did you have any luck running that command? When I run it in an image built with dagster-dbt it gives me output that indicates it's able to find the executable
I hope the problem isn't stemming from the fact that there's also a folder called 'dbt', but we could rule that out by renaming the folder
i've seen issues like that before with python modules, but not with finding an executable
a
that doesn't work for me:
I did rebuild the image renaming the directories to nhl_dbt
same issue
d
That looks like its running dagit (your entrypoint). Try adding a --entrypoint ""
Copy code
docker run -it --entrypoint "" <your image> dbt
a
Copy code
aryan-laptop:nhl-analytics-pipleine-dagster alecryan$ docker run -it --entrypoint "" nhl_dag:1.6 dbt
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "dbt": executable file not found in $PATH: unknown.
ERRO[0000] error waiting for container: context canceled 
aryan-laptop:nhl-analytics-pipleine-dagster alecryan$
d
Got it - OK, so we've reduced the problem to changing the Dockerfile / docker environment so that that command will succeed
I can actually repro with this simple Dockerfile:
Copy code
FROM python:3.7-slim

RUN pip install dagster dagit dagster-postgres dagster-dbt dagster-aws dagster-snowflake pandas requests

WORKDIR /opt/dagster/app

EXPOSE 3000

ENTRYPOINT ["dagit", "-h", "0.0.0.0", "-p", "3000"]
Ah, ok, I think this is simple enough. You'll want to include dbt-core in the list of requirements, it's not automatically included when you install dagster-dbt (probably so that it will work with multiple versions of dbt that may be installed differently? cc @owen who will likely know the true asnwer)
o
ah yeah that's right -- the library (at least some parts of it) work with dbt < 1.0 (as well as dbt 1.0+), so we don't want to be opinionated on if 
dbt
or 
dbt-core
is installed
a
I see, that seems to work
I'm guessing I'll need to install dbt-snowflake as well then, right?
o
yep exactly
a
Awesome, that clears that up. Thanks @johann and @owen for the assists
now that my container is running do I need to do anything special to access dagit?
d
It should appear on localhost:3000 in your browser, assuming your docker is set up to allow inbound connections like that
a
If I go to http://0.0.0.0:3000/ dagit doesn't load as expeceted
I'll need to check the connection
d
try localhost
a
(I'm new to docker if you couldn't tell) lol
same issue
d
try adding a -p 3000:3000 to your docker run
command
a
ah yes, that does it
thanks!
condagster 1
325 Views
Open in Slack
PreviousNext
Powered by