https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
v

Vitalii Malii

07/04/2023, 6:21 PM
Hi everyone. I previously asked for help when I encountered an error while connecting Dagster to an Azure PostgreSQL database through pgbouncer on port 6432. I have finally completed the task and realized that the issue did not stem from network connectivity, port reachability, SQLAlchemy and Psycopg2 libraries, or the Azure PostgreSQL server. In fact, all of these services were functioning properly without any problems. Airflow, which also has a similar configuration for connecting to the server database, worked without any issues in the same AKS cluster. Additionally, my container for testing purposes had similar versions of SQLAlchemy and Psycopg2, and the connection string was set to the default:
connection_string = f'postgresql://{username}:{password}@{server}:{port}/{database}'
. Despite these similarities, Dagster still didn't work with the error message stating port 6432 failed: FATAL: unsupported startup parameter: options. After further investigation, I discovered that the problem lay in a couple of parameters:
ignore_startup_parameters: options,extra_float_digits
this is exactly what I have missed. It turns out that Dagster passes some parameters (which I still don't know exactly what they are) that cause problems when using pgbouncer or RDS Proxy in AWS. By the way I encountered a similar issue when trying to use RDS Proxy for AWS RDS, with the error message stating Feature not supported: RDS Proxy currently doesn’t support command-line options. Unfortunately, we unable to configure
ignore_startup_parameters
or a similar in AWS provided services. I decided to use pgbouncer as an application within the cluster and set the these
ignore_startup_paramethers
, and now everything works fine with Dagster. In conclusion, on the one hand, everything seemed very simple, but on the other hand, time was spent on a problem that may not have existed in the first place. I still don't understand what parameters/options are being passed when connecting to the database and why Airflow doesn't encounter such issues, even though it uses the same libraries for connection.
n

Navah Farahat

07/05/2023, 9:18 PM
i didn't see your post asking for help, but this exact same thing happened to me a few months ago. i got approval to update the
ignore_startup_parameters
in the pgbouncer config on the cluster i was using (not something i set up in the first place) and it was gucci.
your post is a great reminder for me that i'll have to do this again for two other clusters with their own pgbouncers in the next month or so.
thank you!
i wonder if there's something we could contribute to make dagster more pgbouncer-friendly? it's been a few months since i thought about this, but it would be cool for new folks to have a more seamless experience using their connection pooler from day 1 of their dagster deployment (unlike you and i).
🌈 1
13 Views