https://dagster.io/ logo
#ask-community
Title
# ask-community
t

Tomakhina Olha

03/27/2023, 9:53 AM
Hi! When deploying dagster with helm in kuber, we have an issue which happened on Thursday and everything worked before
Copy code
dagster._core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'postgres_url': Field(<dagster._config.source.StringSourceType object at 0x7f9695861bd0>, default=@, is_required=False), 'postgres_db': Field(<dagster._config.field_utils.Shape object at 0x7f96713d6260>, default=@, is_required=False), 'should_autocreate_tables': Field(<dagster._config.config_type.Bool object at 0x7f96957e85b0>, default=True, is_required=False)}.
Error 1: Post processing at path root:postgres_db:password of original value {'env': 'DAGSTER_PG_PASSWORD'} failed:
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_PG_PASSWORD" which is not set. In order for this execution to succeed it must be set in this environment.
It happened after recreating pods with jenkins, nothing was changed in deploy settings. what it can be? maybe subbest the way what we can think about.. for now - no idea what it can be
🤖 1
j

johann

03/27/2023, 5:04 PM
Which pod did this occur in?
The
DAGSTER_PG_PASSWORD
env gets set on dagit, daemon, and user deployments using a postgres secret
Copy code
env:
            - name: DAGSTER_PG_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ include "dagster.postgresql.secretName" . | quote }}
                  key: postgresql-password
Which by default is created by our helm chart https://github.com/dagster-io/dagster/blob/master/helm/dagster/templates/secret-postgres.yaml#L14
Maybe • jenkins created the pod in a different namespace that doesn’t have the secret? I think that would error on pod creation about a missing secret though • The value of the secret is empty now?
t

Tomakhina Olha

03/27/2023, 7:54 PM
Thank you very much for your answers! Looking what is the problem. Will be back with the result later.
It seems the problem is in dagster daemon. Because user code and de[ployment is OK, I can run a job from launchpad, the problem is only with • schedule - it is not run when schedule is and the error I’ve sent above is raised at that moment • in dagit I don’t see any logs for any op. But pipeline runs well and gives the result I expect. (((
@johann thank you for your answer. really there was an update of dagster and after this behaviour was changed. and yes, adding the DAGSTER_PG_PASSWORD to secrets helped.
Maybe you can suggest why after the update I don’t see logs in dagit at all?
j

johann

03/28/2023, 1:02 PM
There was an accidental ui change that is hiding logs by default, it will be fixed in the release tomorrow
You can still see them using the new levels drop-down
t

Tomakhina Olha

03/28/2023, 1:31 PM
Nice, thanks.