We have been trying to deploy dagster on helm for a few hours now and we are hitting this problem:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "<http://core-shared-tools.cktverwlkjvp.us-east-1.rds.amazonaws.com|core-shared-tools.cktverwlkjvp.us-east-1.rds.amazonaws.com>" (10.20.177.69), port 5432 failed: FATAL:  password authentication failed for user "dagster"

42
connection to server at "<http://core-shared-tools.cktverwlkjvp.us-east-1.rds.amazonaws.com|core-shared-tools.cktverwlkjvp.us-east-1.rds.amazonaws.com>" (10.20.177.69), port 5432 failed: FATAL:  password authentication failed for user "dagster"

I know what you are thinking, have you verified that the password is correct? We have tried so many combinations of setting this password. autogenerated, from vault, plain text in the values.yaml file. With an external postgres instance, without. And it always says it is incorrect. I was wondering if anyone has hit a similar issue? This is the values.yaml file we have now:

Hi Majori (good to see you again, I think i remember you from FB London 🙂 ) - sorry for the trouble here. One thing I want to double check first - you have postgresql.enabled set to true there, so I think the helm chart is still expecting that it's setting up its own postgres pod for you. If you want it to connect to an extrernal postgres instance, I believe you want enabled to be false there. If enabled is true, I think it will ignore whatever you have set for postgresqlHost. Then the next thing I would double check is the structure of your dagster-postgres-secrets secret. This could be documented better, but if you supply it yourself, dagster is expecting that it will follow the same format as the one that it would generate for you if generatePostgresqlPasswordSecret were true (i.e. it should have a postgresql-password key like this: https://github.com/dagster-io/dagster/blob/master/helm/dagster/templates/secret-postgres.yaml#L14) Did you ever try using the simplest setup where there's no external postgres DB at all and everything is managed within the helm chart? For that it would be very surprising to see a password auth error.

Hi @daniel! Good to see you indeed 🙂

Did you ever try using the simplest setup where there’s no external postgres DB at all and everything is managed within the helm chart? For that it would be very surprising to see a password auth error.

I tried the simplest config with no external postgresDB, there were no errors here and the dagster-daemon and dagster-dagit pods connected to the database in the same cluster. But, we want to connect to an external RDS database with secrets that we created on vault. So to debug this I have tried : 1. Connecting to the external database, but letting the chart create the secrets file with

generatePostgresqlPasswordSecret=true

2. Creating our own secrets template (

generatePostgresqlPasswordSecret=false

with postgresql-password as the key) that gets the password from vault. With both of these I had to luck, it keeps saying the password is incorrect. I doubled checked that the password in the secrets is the same as for the user in the database. Checked that env variables in the pods are reading from the correct secrets file, and that the value retrieved is the correct password. I am really puzzled what is happening here.

- name: DAGSTER_PG_PASSWORD
          valueFrom:
            secretKeyRef:
              key: postgresql-password
              name: dagster-postgresql-secret

hmm, one thing you could try is seeing whether you can access the RDS DB with psql from some pod in the kubernetes cluster? Here's some debugging steps we've used for checking psql issues before (swapping in your username/database name as needed):

```
kubectl run psql-interactive --rm -i --tty --image library/postgres:9.6.21 -- psql -h <<RDS host>> --username postgres -d dagster

Then immediately enter the RDS password (won't be prompted).``` That would let you verify that the password is what you expect and that RDS should be accessible with those credentials within your k8s cluster (to rule out issues within dagster)

That is a great hint! I could connect to RDS through the interactive shell but still can’t through the dagster-{dagit,daemon} pod

ok one other thing to double-check: you base64-encoded the password when you created the secret yourself? (That wouldn't apply when generatePostgresqlPasswordSecret was true and dagster was creating the password - that part is particularly strange)

If I'm understanding correctly- it works in a fresh pod (kubectl run) but not when exc’ed within the dagit pod (kubectl exec)? That sounds to me like the Dagit pod is running with different networking or IAM permissions. Are you setting an IAM role for Dagit in annotations or anything like that?