https://dagster.io/ logo
Join Slack
Powered by
Hello Everyone. I've been battling with getting my...
# ask-community
b
Hello Everyone. I've been battling with getting my dagster workspace to pick up my repositories. Does any one know what could be the issue.
s
Hi BK, what does your workspace.yaml look like?
b
Hello @sean i think the workspace is been managed using helm charts. This is what i can see.
Copy code
dbt_pipelines:
  secrets:
    dagsterRedshiftPw: ""
    DBT_GITHUB_TOKEN: ""
    SLACK_TOKEN: ""
    PAGERDUTY_TOKEN: ""

hudi_pipelines:
  secrets:
    EMR_SSH_KEY: ""
    MSK_PASSWORD: ""
    AWS_ACCESS_KEY_ID: ""
    AWS_SECRET_ACCESS_KEY: ""

dagster-user-deployments:
  deployments:
    - name: dbt-pipelines
      image:
        repository: localhost:5000/dbt_pipelines
        tag: "dev"
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--package-name"
        - "dbt_pipelines"
      port: 3031
      env:
        dagsterRedshiftHost: ""
        dagsterRedshiftUser: ""
        dagsterRedshiftDb: ""
        DBT_GITHUB_USER: ""
        DAGIT_BASE_URL: "<http://localhost:8080>"

      envSecrets:
        - name: dagster-redshift-pw
        - name: dagster-github-token
        - name: dagster-slack-token
        - name: dagster-pagerduty-token

    - name: hudi-pipelines
      image:
        repository: localhost:5000/hudi_pipelines
        tag: dev
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--package-name"
        - "hudi_pipelines"
      port: 3031

      envSecrets:
        - name: dagster-emr-ssh-key
        - name: dagster-msk-password
s
I’m guessing that there is a bug in your repository code. The repositories get loaded as grpc servers and if the code has bugs (like a python syntax error or something) the server will crash, the host process won’t be able to communicate with it, and you’ll get an error like above. Can you confirm that you’re able to load your code locations with e.g. (from the appropriate working directory);
Copy code
$ dagster dev -m dbt_pipelines
b
Thanks @sean. I'm not sure if there's an error there as i this is the repo that is running in prod which i just cloned. Currently having some issues with building the docker image. This is the image i'm working with
Copy code
FROM python:3.7.8-slim

RUN apt-get update -yqq \
    && apt-get install -yqq cron \
    && rm -rf /var \
    && rm -rf /root/.cache  \
    && rm -rf /usr/lib/python2.7 \
    && rm -rf /usr/lib/x86_64-linux-gnu/guile

WORKDIR /dbt_pipelines
COPY ./repositories/dbt_pipelines/requirements.txt ./requirements.txt
RUN pip install -r requirements.txt


COPY ./repositories/dbt_pipelines/ .
RUN pip install .
s
Does this run without issue on the cloned repo?
Copy code
$ dagster dev -m dbt_pipelines
b
It returns an Error: 
No such command 'dev'.
Could be because 
dbt_pipelines
is a folder not a file.
s
what version of dagster are you running?
(do 
dagster --version
)
b
Copy code
dagster, version 0.14.19
s
that is a very old 
dagster
version and it doesn’t have the 
dev
command. I’m guessing this is where your problem is coming from. You should try matching whatever you’re using in production locally. You could start out by just trying the most recent version though, which is 
1.3.3
b
That is actually the version in prod. That is why i am working on learning how it works so i can update the one in prod because i don't want to break anything. Requirements file in prod
Copy code
dagster==0.14.19
dagster-dbt==0.14.19
dagster_k8s==0.14.19
dagster-postgres==0.14.19
dbt-redshift==1.2.0
pandas==0.25.3
pygit2==1.7.0
dagster_slack==0.14.19
markupsafe==2.0.1
pdpyras==4.5.0
protobuf<3.21.0
s
hmm ok. Instead of 
dagster dev -m dbt_pipelines
, do 
dagit -m dbt_pipelines
b
error returned
Copy code
from dagster._core.libraries import DagsterLibraryRegistry
ModuleNotFoundError: No module named 'dagster._core'
s
can you post a larger backtrace? Where is that import statement? 
dagster._core
does not exist 0.14.x
b
Allright.
s
It looks like your dagit is not the same version as dagster-- can you 
pip install dagit==0.14.19
and then try
b
Just tried new error.
s
It’s hard for me to debug this without knowing what else is in your python environment. I think you should proceed with a fresh environment-- can you create a new virtual env and 
pip install dagster==0.14.19 dagit==0.14.19
.
b
You've been very helpful Sean thanks so much. I'm also trying to make sense of of it as it quite different from the docs i've seen. Quick question: What does the docker image then play if we still have to install libraries locally.
s
I’m also trying to make sense of of it as it quite different from the docs i’ve seen.
One reason for this is that your version is over a year old. The old docs are available here: https://docs.dagster.io/0.14.19 If you just look at the main docs site without this you’ll probably get confused with 0.14.19 code, Dagster has changed a lot.
What does the docker image then play if we still have to install libraries locally.
The docker image is typically used in a deployment context. I wouldn’t recommend working with it during local development.
b
One reason for this is that your version is over a year old. The old docs are available here: https://docs.dagster.io/0.14.19
Oh i see. This page returns a 404 error.
s
that little dropdown that shows 0.14.19 can always be used on the docs site to view older docs:
b
Hello @sean. I'm not seeming to be making much progress even after going through the docs. Could i book sometime with you to help debug 🙏.
23 Views
Open in Slack
PreviousNext
Powered by