https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Daniel Mosesson

12/21/2022, 2:32 PM
When starting a job I am getting
dagster._core.errors.DagsterLaunchFailedError: Error during RPC setup for executing run: dagster_postgres.utils.DagsterPosgresException: too many retries for DB connection
but other jobs work just fine. Any ideas what could be happening?
This seems to only be happening in 1/3 code locations if that helps
This is still happening, persisting across dagit restarts (consistently the same location). There aren't that many active DB connections at any one time (~20 total). Anyone have any ideas?
d

daniel

12/22/2022, 5:58 PM
are there any indications from your postgres logs about what query might be failing?
d

Daniel Mosesson

12/22/2022, 6:30 PM
I don't see any errors at all in PG
p

prha

12/22/2022, 8:06 PM
Are you using the default run launcher? It’s odd because the main DB query that gets executed while launching the run is just fetching the run from run storage, and that shouldn’t be affected by user code at all
Can you try spinning up dagit with just the problematic code location?
d

Daniel Mosesson

12/22/2022, 10:19 PM
Yes to the first, and tentatively we have a resolution: • The error that dagit was actually getting was
SCRAM authentication requires libpq version 10 or above
which cause the authentication to fail • psycopg2-binary is installed (dependancy of dagster-postgres) • the new code location includes psycopg (version 3) in a venv which doesn't play nice with psycopg2, but I somehow it is interfering with the host dagit process???
I guess the really strange part to me is that something in a venv can break the launch runner somehow on the host process (this is multiple locations on the same machine, using python packages)
p

prha

12/22/2022, 10:57 PM
wow… thanks for the detailed report. I’ll try to replicate and figure out what’s going on. Do you mind filing a GH issue to track?
And can you confirm that dagit and dagster-daemon are using the same virtual environment? (Trying to make sure I’m replicating the same launch path)
d

Daniel Mosesson

12/23/2022, 1:06 PM
They are, and the other code location is using a venv. gituhb ticket: https://github.com/dagster-io/dagster/issues/11353
Updated the github issue, but short version is the expected behavior is that dagit requires all code locations to be able to import from dagster_postgres iff it is using postgres for run storage. Adding an extra dependency so that works isn't the end of the world and confirmed it solves the issue
d

daniel

12/25/2022, 4:04 PM
that's certainly the case - although I'd have hoped the error message would be much clearer and be something like "could not import dagster-postgres"
d

Daniel Mosesson

12/25/2022, 4:08 PM
The reason for the initial terrible error message has to do with a pyscopg2 bug (https://github.com/psycopg/psycopg2/issues/1360), together with swallowing the "real" error from the failure to connect by dagit/code location
Should I explicitly have my packages take a dependency on
dagster-postgres
?
d

daniel

12/25/2022, 4:14 PM
That's what we would recommend, yeah
any packages referenced in your dagster.yaml
d

Daniel Mosesson

12/25/2022, 4:15 PM
makes sense. Do I have to be concerned about the versions of those things? I run a
dagster migrate
when there is a reason to on the
dagit
processes, but updating all the packages usually happens sometime later
d

daniel

12/25/2022, 4:22 PM
You can run dagit and the code packages on different dagster versions - within a single python environment the dagster versions all need to match, but there are pins enforcing that
35 Views