Hi everyone! Is anyone familiar with running dbt-dagster on Kubernetes? I launch my pod successfully, run the dagitUI within the container and try to execute a job… But am met with a slew of errors. The logs show Errors stating that dagster isn’t able to reach my repo.

/usr/local/lib/python3.7/site-packages/dagster/core/workspace/context.py:560: UserWarning: Error loading repository location moonfire-dagster-repo:dagster.core.errors.DagsterUserCodeUnreachableError: Could not reach user code server

As well as issues with reaching the postgres server.

Started execution of step "dbt_project".

Executing command: dbt --log-format json run --project-dir ./dbt --profiles-dir ./dbt/config --vars {"test_split": 0.1, "train_split": 0.8} --select +sector_cls_eval_set

Running with dbt=1.0.1-rc1

Unable to do partial parsing because config vars, config profile, or config target have changed


Found 21 models, 131 tests, 0 snapshots, 0 analyses, 167 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics

ERROR
Encountered an error:
Database Error
  connection to server at "35.195.146.254", port 5432 failed: timeout expired
  
17:06:45.710

This may be indicative of a larger problem accessing sercrets/credentials - but all of the other jobs work just fine. I’ve entered the pod and it clearly contains the api-keys secret (so it has the DB password available as an env variable) so it shouldn’t be failing like this. Steps to recreate the error: This error came from me: • Entering the pod manually with the following commands ◦ export DAGIT_POD_NAME=$(kubectl get pods --namespace default -l “app.kubernetes.io/name=dagster,app.kubernetes.io/instance=dagster,component=dagit” -o jsonpath=“{.items[0].metadata.name}“) ◦ kubectl --namespace default port-forward $DAGIT_POD_NAME 8080:80 • clicking the link to get to the DagitUI • selecting the

sector_cls_ALL...

job • executing it Any help at all debugging these types of issues would be much appreciated 🙌

hi @Harpal! are non-dbt processes able to connect to this remote server from your kube pod?

Hi @owen. I’ve just checked and yes, Dagster sensors and jobs are able to connect to this remote server from the same kube pod.

hmm -- that's definitely strange. Are there any other environment variables that your profiles.yaml file is reading in other than the password which might be missing (like a schema or user, something like that)? It doesn't seem too likely that this is a dagster-specific issue (rather than a dbt-specific thing), but you could verify that by copy pasting the command that dagster's running the kube pod. I'll try to think of other things that might be different between the environments as well

Thanks again for your support @owen. I’ve asked the dbt Slack channel but haven’t heard back from anyone yet. All the information in the profiles.yaml file appear to be up to date (i even tried hard-coding the password field). I did notice that the

target

in the profiles.yml file was

dev

instead of

prod

. However that shouldn’t make a difference as dev/prod are both using the same credentials at the moment.

hi @Harpal! were you able to make any progress on this?

Hiya @owen. YES I managed to get to the root of the problem 🙂 It turns out that it wasn’t a dagster or a dbt error. It was related to how we GKE was trying to access our cloud SQL instance. Our containers weren’t configured to allow access to our DB. I’m following these steps to properly configure our Cloud SQL Auth proxy and that should do the trick.

nice! 🙌 glad you figured it out