https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
n

Neil

09/01/2023, 8:59 PM
Hey gang, I have a (hopefully small) question: I have Dagster on EKS, and made an RDS instance for it to connect to.
dagster-webserver
and
dagster-daemon
pods are able to authenticate I believe, but the pods fail with an insufficient privileges error like so:
Copy code
(psycopg2.errors.InsufficientPrivilege) permission denied for schema public: 
CREATE TABLE secondary_indexes (
                     ^

[SQL:CREATE TABLE secondary_indexes (
	id BIGSERIAL NOT NULL,
	name VARCHAR(512),
	create_timestamp TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,
	migration_completed TIMESTAMP WITHOUT TIME ZONE,
	PRIMARY KEY (id),
	UNIQUE (name)
)
Now, I gave the dagster user on Postgres ownership on the
dagster
database it's connected to , and I can create that darn
dagster.public.secondary_indexes
table myself when I log on to RDS as the dagster user. Does anyone know what's going on? Where is it trying to make this table?
Is there any documentation about what tables dagster is trying to make on its postgres instance, or what permissions it needs on the instance?
d

daniel

09/01/2023, 9:08 PM
Hi Neil - hen Dagster starts up for the first time it will check to see if the tables that it needs exists on the RDS instance - if it doesn't, it will create them. Is it just that one table that's missing in your RDS instance, or is it having trouble trying to create any tables?
n

Neil

09/01/2023, 9:11 PM
Hey Daniel! It's a completely fresh RDS instance, so all I did from there is make a
dagster
database and user, and give the user ownership on the
dagster
database. I'm able to create the table with the same SQL myself if I connect manually to RDS, so I don't understand why dagster can't unless it's trying to make it in another database aside from the one I told it about
d

daniel

09/01/2023, 9:11 PM
Did it create any other tables?
Or is "secondary_indexes" just the first one it tried to create and it failed?
I'm assuming the latter, but just checking
n

Neil

09/01/2023, 9:13 PM
I think the latter, I don't see any other tables
d

daniel

09/01/2023, 9:16 PM
If you're able to kubectl exec into either of the pods you mention - I think this would be enough to trigger the schema creation if that helps troubleshoot:
Copy code
from dagster import DagsterInstance
DagsterInstance.get()
What dagster does is uses sqlalchemy to create the tables using the postgres user / credentials that you set in the Helm chart. Are you sure you're using the same postgres schema and db when running the commands locally that you have configured in the helm cahrt?
n

Neil

09/01/2023, 9:21 PM
100%. There's no other user on this instance aside from the root one, and it wouldn't have permissions errors if it was that.
I'll try to exec in and see
Oh right, the db permissions thing is caushing a CrashLoopBackoff so can't exec in to the pod :X
Do you know where the db connection stuff happens in the source code? Curious if it's actually using the
database
parameter I sent or it's defaulting to the
postgres
database or something
If you send over the postgresql part of your helm chart values.yaml we could glance over it (other than any passwords of course)
n

Neil

09/01/2023, 9:30 PM
Sure, here's what I got:
Copy code
global:
  postgresqlSecretName: "dagster-postgresql-rds-secret"
postgresql:
  enabled: false
  postgresqlHost: "<http://blahblahblah.rds.amazonaws.com|blahblahblah.rds.amazonaws.com>"
  postgresqlUsername: "dagster"
  postgresqlDatabase: "dagster"
generatePostgresqlPasswordSecret: false
d

daniel

09/01/2023, 9:31 PM
that does look pretty standard yeah..
this is not for our product, but just from searching around there are some suggestions for the specific grants and permissions the user has to have in order to be able to programatically create tables here: https://forum.mattermost.com/t/pq-permission-denied-for-schema-public/14273 - although that doesn't square with you being able to create them locally when you login, unless there's something fundamentally different about doing it over a sqlalchemy connection
n

Neil

09/01/2023, 9:36 PM
Oh my gosh, that last bit was it:
GRANT USAGE, CREATE *ON* SCHEMA *PUBLIC*
*TO* dagster;
d

daniel

09/01/2023, 9:36 PM
phew!
n

Neil

09/01/2023, 9:36 PM
Here I thought setting the database owner to dagster would have given it everything, but I forgot public already existed
Thanks for the help, Daniel!
condagster 1
Weird that I could create the table in
dagster.public
myself as the
dagster
user but I won't complain now that it's working
9 Views