https://dagster.io/ logo
Powered by
#dagster-support
Title
# dagster-support
l

Liezl Puzon

04/20/2022, 10:00 PM
Can I get a sanity check on my postgres config for my external server? Replaced X.X.X.X with my public IP for cloud SQL. But I keep getting this error in the dagster-dagit Service logs
Copy code
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "dagster-postgresql" to address: Name or service not known
still having a problem with this config. It seems like it should be straightforward but the simple config values don’t work Is there a recommended way to use secrets to set this up? I’m not sure what this part means
Copy code
global:
  postgresqlSecretName: "dagster-postgresql-secret"
d

daniel

04/21/2022, 1:40 AM
Hi Liezl - after deploying the helm chart there should be a configmap called something like dagster-instance (something ending in -instance). Is it possible to paste the dagster.yaml value inside that configmap here? That will help show exactly what it's using to try to connect to postgres
that host name error is confusing and almost seems like it's somehow still using the host or config from back when you were creating the postgres DB in your cluster? (From a previous post)
l

Liezl Puzon

04/21/2022, 1:46 AM
gotcha, yea I just recently switched back to my original config then ran 
helm upgrade --install dagster dagster/dagster --version 0.14.3 -f values/production.yaml
and seemed like it’s fine when I go back to postgres DB in cluster
am I supposed to do something more than just run 
helm upgrade --install dagster dagster/dagster --version 0.14.3 -f values/production.yaml
after I update the 
postgresql:
field to point to external?
d

daniel

04/21/2022, 1:47 AM
One thing to check - when you re-deployed with enabled:False, did the dagit service restart?
the error is almost like it kept going, expecting the in-cluster DB to still be there
l

Liezl Puzon

04/21/2022, 1:48 AM
ooo let me see if I can figure out how to see that by clicking around
so interestingly the configmap seems like the update timestamp is old
d

daniel

04/21/2022, 1:50 AM
if you scroll down to run_storage in the dagster.yaml data it should say what the postgres hostname is. I'd expect that to be updated as soon as you upgraded the helm chart
and then in theory the dagit deployment is set up to restart whenever that configmap changes
l

Liezl Puzon

04/21/2022, 1:51 AM
yeah looks good
let me fix the ip address
but essentially itll show me the same thing but with my db address
thats what i expect
Copy code
scheduler:      
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      username: postgres
      password:
        env: DAGSTER_PG_PASSWORD
      hostname: "X.X.X.X"
      db_name:  postgres
      port: 5432
      params: 
        dbname: postgres
        password: postgres
        port: 5432
        user: postgres

run_launcher:      
  module: dagster_k8s
  class: K8sRunLauncher
  config:
    load_incluster_config: true
    job_namespace: dagster
    image_pull_policy: Always
    service_account_name: dagster
    dagster_home:
      env: DAGSTER_HOME
    instance_config_map:
      env: DAGSTER_K8S_INSTANCE_CONFIG_MAP
    postgres_password_secret:
      env: DAGSTER_K8S_PG_PASSWORD_SECRET
    env_config_maps:
      - env: DAGSTER_K8S_PIPELINE_RUN_ENV_CONFIGMAP

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username: postgres
      password:
        env: DAGSTER_PG_PASSWORD
      hostname: "X.X.X.X"
      db_name:  postgres
      port: 5432
      params: 
        dbname: postgres
        password: postgres
        port: 5432
        user: postgres

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      username: postgres
      password:
        env: DAGSTER_PG_PASSWORD
      hostname: "X.X.X.X"
      db_name:  postgres
      port: 5432
      params: 
        dbname: postgres
        password: postgres
        port: 5432
        user: postgres
compute_logs:      
  module: dagster.core.storage.noop_compute_log_manager
  class: NoOpComputeLogManager
d

daniel

04/21/2022, 1:53 AM
got it - but back when you had enabled:true, the hostname in there was "dagster-postgresql" right?
l

Liezl Puzon

04/21/2022, 1:53 AM
ok i see the ip changed
d

daniel

04/21/2022, 1:53 AM
so the mystery is why the dagster-dagit service is giving you errors referencing an old hostname
l

Liezl Puzon

04/21/2022, 1:53 AM
helm upgrading
and checking that now
yep that’s right
d

daniel

04/21/2022, 1:55 AM
ok - so the configmap is updating fine, did you check if the dagit pod is restarting? It should be, but if it wasn't, that could explain it still referencing a hostname that no longer exists
l

Liezl Puzon

04/21/2022, 1:55 AM
let me check 
dagster-dagit
and see if its updating
it does sound like its something like that
does this look like a problem?
d

daniel

04/21/2022, 1:57 AM
potentially, yeah - does it give any more info why it would be stuck in PodInitializing?
it could be that it got stuck in this interim 'updating' state and the error you're seeing is from the old pod that will spin down as soon as the new one is ready
l

Liezl Puzon

04/21/2022, 1:57 AM
not sure how to force it to restart
but gotcha, sounds like a cluster thing
GKE blocking the restart or something?
d

daniel

04/21/2022, 1:58 AM
one thing you could do just to rule out weird upgrade issues is do a full helm uninstall first (we should be able to handle upgrades, but would let us start from a place where its working at least)
and yeah, maybe if your cluster is right at its resrouce limit there are no resources available for the new pod to spin up or something?
l

Liezl Puzon

04/21/2022, 2:01 AM
ok so that screenshot was from when I re-deployed my original “postgres in GKE” config, let me see if it still gets stuck with the external SQL config
seems like there’s space
d

daniel

04/21/2022, 2:03 AM
yeah that was just a shot in the dark - another possibility is that there's some other issue with the new pod connecting to the external postgres ( there's a pg_isready call that it makes)
which then keeps the old pod from spinning down, which would explain the confusing error about the old hostname
l

Liezl Puzon

04/21/2022, 2:21 AM
gotcha, ok I went back and tried to actually issue a query and realized I didn’t whitelist any IPs to connect to cloud SQL. once I fixed that it worked
d

daniel

04/21/2022, 2:23 AM
Ah great!
Powered by