https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
g

Ginger Balmat

01/03/2022, 3:47 PM
Good morning all! I have Dagster deployed to GKE and I'm working to set up external access to a postgres db hosted in Cloud SQL. My db has a private IP and I've successfully connected by adding this config to my values.yml
Copy code
postgresql:
  enabled: false
  postgresqlHost: <postgres_private_ip_address>
  ...
I would prefer not to have to put the IP address in values.yml though. It would also be my preference not to connect directly using the private IP if I don't have to. Has anyone had success in setting up db access using the Cloud SQL Auth proxy? The Google docs recommend using the sidecar container pattern, which I'm guessing would rely on the
dagster-daemon
pod. Does anyone know if this is currently possible?
j

johann

01/03/2022, 3:49 PM
Hi Ginger- the sidecar would act as an ambassador, rerouting the requests from localhost to the correct DB address?
g

Ginger Balmat

01/03/2022, 3:51 PM
Hi johann! Yes, that is my understanding
I believe the pod configuration for the cloud sql proxy sidecar would need to be added under
containers
in the
dagster-daemon
Deployment manifest. At first glance it doesn't look like this is possible, but I thought I would ask first.
j

johann

01/03/2022, 4:06 PM
The issue will actually be in the pods that Dagster launches for your runs- we don’t provide a way to configure a sidecar container there. It’d be great if you could file an issue, we have a few ideas for how to make this configuration better
You’re correct that you’d also need to add it to the daemon deployment, but you could accomplish that with https://kustomize.io/ as a post render filter
👍 1
g

Ginger Balmat

01/03/2022, 4:10 PM
Ah ok, that's good to know about using kustomize. So it looks like the issue is actually with adding the sidecar to the user-code-deployments. Ok that makes sense. I'll file an issue. Thanks so much!
j

johann

01/03/2022, 4:11 PM
In the meantime- is it an ok workaround to use the IP?
g

Ginger Balmat

01/03/2022, 4:12 PM
Sure thing.. it's not a deal breaker for me 🙂
j

johann

01/03/2022, 4:12 PM
Great. And just to clarify- it’s the ephemeral Kubernetes Jobs that we launch for runs that pose the problem
👍 1
g

Ginger Balmat

01/03/2022, 4:13 PM
That makes sense. Thanks again!
j

johann

01/03/2022, 4:13 PM
Since they’re created spontaneously you can’t target them with kustomize, we just need to offer more customization when we launch them with the python k8s client
👍 2
a

Andrea Giardini

01/03/2022, 8:34 PM
Another way of solving this would be to create a separate service/deployment with cloudsql and then route the psql requests to this service. I believe it should work!
j

Javier Llorente Mañas

03/09/2022, 10:58 AM
hey @Andrea Giardini, do you have any example on how could I use cloud SQL this way?
a

Andrea Giardini

03/09/2022, 10:59 AM
Create a separate service/deployment running cloudsql and then add the service name to the helm chart
postgresqlHost
j

Javier Llorente Mañas

03/09/2022, 11:00 AM
aaa so basically I need an extra deployment and an extra service in the chart and later root the postgres host variable to the deployment name?
a

Andrea Giardini

03/09/2022, 11:01 AM
exactly. To the service name, not deployment name
it's not the "recommended" approach by google, but it's actually not bad
j

Javier Llorente Mañas

03/09/2022, 11:01 AM
thanks!! Do you know if there is a post explaining this?
Yeah but the sidecar container it is a really intrusive approach
a

Andrea Giardini

03/09/2022, 11:02 AM
Do you know if there is a post explaining this?
Not sure... i believe i came up with this idea myself
j

Javier Llorente Mañas

03/09/2022, 11:03 AM
it will be a good idea for a post!
Also, I see what you mean, and I don’t think the sidecar container option Google proposed will work in this context and in many DE apps. Most of these apps got an init container in their scheduler to check that the Db is up and running before running a Db migration I do not see an easy way to fit a sidecar container with an init container, the app will never start.
a

Andrea Giardini

03/09/2022, 1:37 PM
There is acually a reason why I decided to go for a deployment+service rather than a sidecar... Every cloudsqlproxy creates a couple of management connection (like 1-2). So when you use is as a sidecar every pod will basically create a couple of connections for management. In the past it was not possible to edit the max_connections parameters on cloudsql so we worked around that using this deployment + service combination.
j

Javier Llorente Mañas

03/09/2022, 1:38 PM
also using a sidecar container means changing a couple of deployments inside the chart, might not be the best option for maintainability
a

Andrea Giardini

03/09/2022, 1:39 PM
Let me know how you like this solution. I'm open to feedback
j

Javier Llorente Mañas

03/10/2022, 7:52 AM
I deploy Dagster using this solution, and I truly like it.
6 Views