Hi When deploying dagster with helm in kuber we have an issu dagster #ask-community

Hi! When deploying dagster with helm in kuber, we ...

Tomakhina Olha

03/27/2023, 9:53 AM

Hi! When deploying dagster with helm in kuber, we have an issue which happened on Thursday and everything worked before

Copy code

dagster._core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'postgres_url': Field(<dagster._config.source.StringSourceType object at 0x7f9695861bd0>, default=@, is_required=False), 'postgres_db': Field(<dagster._config.field_utils.Shape object at 0x7f96713d6260>, default=@, is_required=False), 'should_autocreate_tables': Field(<dagster._config.config_type.Bool object at 0x7f96957e85b0>, default=True, is_required=False)}.
Error 1: Post processing at path root:postgres_db:password of original value {'env': 'DAGSTER_PG_PASSWORD'} failed:
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_PG_PASSWORD" which is not set. In order for this execution to succeed it must be set in this environment.

It happened after recreating pods with jenkins, nothing was changed in deploy settings. what it can be? maybe subbest the way what we can think about.. for now - no idea what it can be

🤖 1

johann

03/27/2023, 5:04 PM

Which pod did this occur in?

johann

03/27/2023, 5:08 PM

The

DAGSTER_PG_PASSWORD

env gets set on dagit, daemon, and user deployments using a postgres secret

Copy code

env:
            - name: DAGSTER_PG_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ include "dagster.postgresql.secretName" . | quote }}
                  key: postgresql-password

Which by default is created by our helm chart https://github.com/dagster-io/dagster/blob/master/helm/dagster/templates/secret-postgres.yaml#L14

johann

03/27/2023, 5:10 PM

Maybe • jenkins created the pod in a different namespace that doesn’t have the secret? I think that would error on pod creation about a missing secret though • The value of the secret is empty now?

Tomakhina Olha

03/27/2023, 7:54 PM

Thank you very much for your answers! Looking what is the problem. Will be back with the result later.

Tomakhina Olha

03/27/2023, 9:26 PM

It seems the problem is in dagster daemon. Because user code and de[ployment is OK, I can run a job from launchpad, the problem is only with • schedule - it is not run when schedule is and the error I’ve sent above is raised at that moment • in dagit I don’t see any logs for any op. But pipeline runs well and gives the result I expect. (((

Tomakhina Olha

03/28/2023, 12:57 PM

@johann thank you for your answer. really there was an update of dagster and after this behaviour was changed. and yes, adding the DAGSTER_PG_PASSWORD to secrets helped.

Tomakhina Olha

03/28/2023, 12:58 PM

Maybe you can suggest why after the update I don’t see logs in dagit at all?

johann

03/28/2023, 1:02 PM

There was an accidental ui change that is hiding logs by default, it will be fixed in the release tomorrow

johann

03/28/2023, 1:04 PM

You can still see them using the new levels drop-down

Tomakhina Olha

03/28/2023, 1:31 PM

Nice, thanks.

2 Views

Open in Slack

Previous Next