When starting a job I am getting `dagster core errors Dagste

Question

When starting a job I am getting `dagster core errors DagsterLaunchFailedError Error during RPC setup for executing run dagster postgres utils DagsterPosgresException too many retries for DB connection` but other jobs work just fine Any ideas what could be happening

Daniel Mosesson · Answer

This seems to only be happening in 1 3 code locations if that helps

Daniel Mosesson · Answer

This is still happening persisting across dagit restarts consistently the same location There aren t that many active DB connections at any one time ~20 total Anyone have any ideas

daniel · Answer

are there any indications from your postgres logs about what query might be failing

Daniel Mosesson · Answer

I don t see any errors at all in PG

prha · Answer

Are you using the default run launcher It s odd because the main DB query that gets executed while launching the run is just fetching the run from run storage and that shouldn t be affected by user code at all

prha · Answer

Can you try spinning up dagit with just the problematic code location

Daniel Mosesson · Answer

Yes to the first and tentatively we have a resolution The error that dagit was actually getting was `SCRAM authentication requires libpq version 10 or above` which cause the authentication to fail psycopg2 binary is installed dependancy of dagster postgres the new code location includes psycopg version 3 in a venv which doesn t play nice with psycopg2 but I somehow it is interfering with the host dagit process

Daniel Mosesson · Answer

I guess the really strange part to me is that something in a venv can break the launch runner somehow on the host process this is multiple locations on the same machine using python packages

prha · Answer

wow thanks for the detailed report I ll try to replicate and figure out what s going on Do you mind filing a GH issue to track

prha · Answer

And can you confirm that dagit and dagster daemon are using the same virtual environment Trying to make sure I m replicating the same launch path

Daniel Mosesson · Answer

They are and the other code location is using a venv gituhb ticket <https github com dagster io dagster issues 11353>

Daniel Mosesson · Answer

Updated the github issue but short version is the expected behavior is that dagit requires all code locations to be able to import from dagster postgres iff it is using postgres for run storage Adding an extra dependency so that works isn t the end of the world and confirmed it solves the issue

daniel · Answer

that s certainly the case although I d have hoped the error message would be much clearer and be something like could not import dagster postgres

Daniel Mosesson · Answer

The reason for the initial terrible error message has to do with a pyscopg2 bug <https github com psycopg psycopg2 issues 1360> together with swallowing the real error from the failure to connect by dagit code location

Daniel Mosesson · Answer

Should I explicitly have my packages take a dependency on `dagster postgres`

daniel · Answer

That s what we would recommend yeah

daniel · Answer

any packages referenced in your dagster yaml

Daniel Mosesson · Answer

makes sense Do I have to be concerned about the versions of those things I run a `dagster migrate` when there is a reason to on the `dagit` processes but updating all the packages usually happens sometime later

daniel · Answer

You can run dagit and the code packages on different dagster versions within a single python environment the dagster versions all need to match but there are pins enforcing that