Good morning all! I have Dagster deployed to GKE and I'm working to set up external access to a postgres db hosted in Cloud SQL. My db has a private IP and I've successfully connected by adding this config to my values.yml

postgresql:
  enabled: false
  postgresqlHost: <postgres_private_ip_address>
  ...

I would prefer not to have to put the IP address in values.yml though. It would also be my preference not to connect directly using the private IP if I don't have to. Has anyone had success in setting up db access using the Cloud SQL Auth proxy? The Google docs recommend using the sidecar container pattern, which I'm guessing would rely on the

dagster-daemon

pod. Does anyone know if this is currently possible?

Hi Ginger- the sidecar would act as an ambassador, rerouting the requests from localhost to the correct DB address?

Hi johann! Yes, that is my understanding

The issue will actually be in the pods that Dagster launches for your runs- we don’t provide a way to configure a sidecar container there. It’d be great if you could file an issue, we have a few ideas for how to make this configuration better

Ah ok, that's good to know about using kustomize. So it looks like the issue is actually with adding the sidecar to the user-code-deployments. Ok that makes sense. I'll file an issue. Thanks so much!

In the meantime- is it an ok workaround to use the IP?

Sure thing.. it's not a deal breaker for me 🙂

Great. And just to clarify- it’s the ephemeral Kubernetes Jobs that we launch for runs that pose the problem

That makes sense. Thanks again!

Since they’re created spontaneously you can’t target them with kustomize, we just need to offer more customization when we launch them with the python k8s client

Another way of solving this would be to create a separate service/deployment with cloudsql and then route the psql requests to this service. I believe it should work!

hey @Andrea Giardini, do you have any example on how could I use cloud SQL this way?

Create a separate service/deployment running cloudsql and then add the service name to the helm chart

postgresqlHost

aaa so basically I need an extra deployment and an extra service in the chart and later root the postgres host variable to the deployment name?

exactly. To the service name, not deployment name

thanks!! Do you know if there is a post explaining this?

Do you know if there is a post explaining this?

Not sure... i believe i came up with this idea myself

it will be a good idea for a post!

There is acually a reason why I decided to go for a deployment+service rather than a sidecar... Every cloudsqlproxy creates a couple of management connection (like 1-2). So when you use is as a sidecar every pod will basically create a couple of connections for management. In the past it was not possible to edit the max_connections parameters on cloudsql so we worked around that using this deployment + service combination.

also using a sidecar container means changing a couple of deployments inside the chart, might not be the best option for maintainability

Let me know how you like this solution. I'm open to feedback

I deploy Dagster using this solution, and I truly like it.