getting an error in our user pods when deploying to a k8 cluster (GKE). thought i’d post this as-is since its our prod cluster and we have not changed any dagster helm configurations (besides the user code images) recently:

dagster.core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'postgres_url': Field(<dagster.config.source.StringSourceType object at 0x7f6e037ff2b0>, default=@, is_required=False), 'postgres_db': Field(<dagster.config.field_utils.Shape object at 0x7f6dfeb5ce20>, default=@, is_required=False), 'should_autocreate_tables': Field(<dagster.config.config_type.Bool object at 0x7f6e043c9970>, default=True, is_required=False)}.
    Error 1: Received unexpected config entry "scheme" at path root:postgres_db. Expected: "{ db_name: (String | { env: String }) hostname: (String | { env: String }) params?: { } password: (String | { env: String }) port?: (Int | { env: String }) username: (String | { env: String }) }".

@Paul Swithers @David Choy

Hi Solaris - do you have a full stack trace for this?

i do - sorry:

Traceback (most recent call last):
  File "/root/miniconda3/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/root/miniconda3/lib/python3.9/site-packages/dagster/cli/__init__.py", line 50, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/root/miniconda3/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/root/miniconda3/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/root/miniconda3/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/miniconda3/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/miniconda3/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/miniconda3/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/root/miniconda3/lib/python3.9/site-packages/dagster/cli/api.py", line 59, in execute_run_command
    DagsterInstance.from_ref(args.instance_ref)
  File "/root/miniconda3/lib/python3.9/site-packages/dagster/core/instance/__init__.py", line 439, in from_ref
    run_storage=instance_ref.run_storage,
  File "/root/miniconda3/lib/python3.9/site-packages/dagster/core/instance/ref.py", line 250, in run_storage
    return self.run_storage_data.rehydrate()
  File "/root/miniconda3/lib/python3.9/site-packages/dagster/serdes/config_class.py", line 81, in rehydrate
    raise DagsterInvalidConfigError(
dagster.core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'postgres_url': Field(<dagster.config.source.StringSourceType object at 0x7fbd41dd62b0>, default=@, is_required=False), 'postgres_db': Field(<dagster.config.field_utils.Shape object at 0x7fbd3d132e20>, default=@, is_required=False), 'should_autocreate_tables': Field(<dagster.config.config_type.Bool object at 0x7fbd4299f970>, default=True, is_required=False)}.
    Error 1: Received unexpected config entry "scheme" at path root:postgres_db. Expected: "{ db_name: (String | { env: String }) hostname: (String | { env: String }) params?: { } password: (String | { env: String }) port?: (Int | { env: String }) username: (String | { env: String }) }".

understood - and you're sure the helm chart didn't upgrade to a new version? I could imagine this happening if the Helm chart upgraded to a new version that now included https://github.com/dagster-io/dagster/pull/8126/files (which indicates a backwards compatibility problem that we can get out a fix for)

i bet that’s it - take a look at the postgres section in the problem job pod:

__class__: ExecuteRunArgs
instance_ref:
  __class__: InstanceRef
  compute_logs_data:
    __class__: ConfigurableClassData
    class_name: NoOpComputeLogManager
    config_yaml: |
      {}
    module_name: dagster.core.storage.noop_compute_log_manager
  custom_instance_class_data: null
  event_storage_data:
    __class__: ConfigurableClassData
    class_name: PostgresEventLogStorage
    config_yaml: |
      postgres_db:
        db_name: test
        hostname: dagster-postgresql
        params: {}
        password:
          env: DAGSTER_PG_PASSWORD
        port: 5432
        scheme: postgresql
        username: test
    module_name: dagster_postgres.event_log
  local_artifact_storage_data:
    __class__: ConfigurableClassData
    class_name: LocalArtifactStorage
    config_yaml: |
      base_dir: /opt/dagster/dagster_home
    module_name: dagster.core.storage.root
  run_coordinator_data:
    __class__: ConfigurableClassData
    class_name: DefaultRunCoordinator
    config_yaml: |
      {}
    module_name: dagster.core.run_coordinator
  run_launcher_data:
    __class__: ConfigurableClassData
    class_name: K8sRunLauncher
    config_yaml: |
      dagster_home: /opt/dagster/dagster_home
      image_pull_policy: Always
      instance_config_map: dagster-instance
      job_namespace: default
      load_incluster_config: true
      postgres_password_secret: dagster-postgresql-secret
      service_account_name: dagster
      volume_mounts:
      - mountPath: /dmg-secrets/google
        name: keyjson-repo-dmg
        readOnly: true
      - mountPath: /dbt-secrets/google
        name: keyjson-repo-dbt
        readOnly: true
      volumes:
      - name: keyjson-repo-dmg
        secret:
          secretName: tf-svc-dagster-repo-dmg
      - name: keyjson-repo-dbt
        secret:
          secretName: tf-svc-dagster-repo-dbt
    module_name: dagster_k8s
  run_storage_data:
    __class__: ConfigurableClassData
    class_name: PostgresRunStorage
    config_yaml: |
      postgres_db:
        db_name: test
        hostname: dagster-postgresql
        params: {}
        password:
          env: DAGSTER_PG_PASSWORD
        port: 5432
        scheme: postgresql
        username: test
    module_name: dagster_postgres.run_storage
  schedule_storage_data:
    __class__: ConfigurableClassData
    class_name: PostgresScheduleStorage
    config_yaml: |
      postgres_db:
        db_name: test
        hostname: dagster-postgresql
        params: {}
        password:
          env: DAGSTER_PG_PASSWORD
        port: 5432
        scheme: postgresql
        username: test
    module_name: dagster_postgres.schedule_storage
  scheduler_data:
    __class__: ConfigurableClassData
    class_name: DagsterDaemonScheduler
    config_yaml: |
      {}
    module_name: dagster.core.scheduler
  settings:
    telemetry:
      enabled: false
pipeline_origin:
  __class__: PipelinePythonOrigin
  pipeline_name: sleep_job
  repository_origin:
    __class__: RepositoryPythonOrigin
    code_pointer:
      __class__: FileCodePointer
      fn_name: repo_dmg
      python_file: /mixer/dagster/workspace_cbh/repo_dmg/repo.py
      working_directory: /mixer/dagster/workspace_cbh/repo_dmg/
    container_context: {}
    container_image: '<http://us.gcr.io/cityblock-orchestration/cbhdagster/dmg:v21|us.gcr.io/cityblock-orchestration/cbhdagster/dmg:v21>'
    entry_point:
      - dagster
    executable_path: /root/miniconda3/bin/python
pipeline_run_id: 3e546948-3c2f-4be3-beb3-1ac7c5d50739
set_exit_code_on_failure: null

ok - we should be able to fix this. in the short term while we sort that out, i suspect upgrading your code image to the latest version as well should fix it

you mean our py packages for dagster and related to 0.14.19 (which is our helm chart versions - just checked as you suspected)?

yeah

thank you!! i was thinking our team should have better practices when deploying if there’s compatibility issues if you didn’t state that - since our deploy scripts auto-update helm charts but not dagster package versions

@daniel we upgraded to 0.14.19 but have now run into this bug: https://github.com/dagster-io/dagster/issues/8201

I believe the helm chart compatibility issue was introduced in 0.14.19 - do you know what version your code was on before you upgraded and ran into the issue you just linked?

0.14.13

Got it. And the original change that caused all these troubles was upgrading the helm chart to 0.14.19 right? Were you looking for a specific change there that caused you to upgrade, or would rolling everything back to 0.14.13 be an option while we sort these two issues out? Sorry about all the trouble here, we should be able to get both these issues sorted out in the release this week.

is there a way to indicate the version in the helm upgrade statement? or can I only upgrade to latest version?

Yeah, you can specify a version - will pull that up, one sec

do I need to specify a helm version or a dagster version?

just helm version (which is the same as the dagster version)

we already include this in our upgrade statement? is it the same thing?

--set helm.versions="0.14.13"

I haven't seen that before - that looks to me that its setting a variable within the helm chart, as opposed to determining the release version of the helm chart to use

Thanks for your help here. Including the --version seems to be working. Much better overall if we control the version ourselves rather than getting the latest automatically.

Hi Paul, one followup - do you possibly have the code handy is is triggering that issubclass() error? Either to post here or to DM? No worries if its sensitive

Hey @daniel I’m getting the same error when collecting results from a dynamic op. Here’s a repro:

@op(
    out=DynamicOut(str),
)
def generate_names():
    for num in range(100):
        yield DynamicOutput(str(num), f"name_{num}")


@op
def name_to_path(_: OpExecutionContext, name: str) -> Path:
    return Path("/tmp", name)


@op
def concat_paths(_: OpExecutionContext, paths: list[Path]) -> str:
    names = [str(path) for path in paths]
    return "".join(names)


@graph
def path_graph():
    names = generate_names()
    paths = names.map(name_to_path)
    return concat_paths(paths.collect())

jobs = path_graph.to_job(resource_defs=resources)

@daniel I'm afraid that the error happens when I load the repo into dagit not when the code is actually executed and the error message doesn't indicate which line in my code is problematic only the line in the dagster code base where the exception is raised. The repo provided by @fahad is representative of my usage of the dynamic output feature.

quick question @fahad - should that list[Path] rather than List[Path]? Or is that a different type of annotation that i'm not familiar with? I'm having trouble reproing with this example code based on yours:

from dagster import DynamicOut, op, graph, repository, OpExecutionContext, DynamicOutput
from pathlib import Path
from typing import List

@op(
    out=DynamicOut(str),
)
def generate_names():
    for num in range(5):
        yield DynamicOutput(str(num), f"name_{num}")

@op
def name_to_path(_: OpExecutionContext, name: str) -> Path:
    return Path("/tmp", name)

@op
def concat_paths(_: OpExecutionContext, paths: List[Path]) -> str:
    names = [str(path) for path in paths]
    return "".join(names)

@graph
def path_graph():
    names = generate_names()
    paths = names.map(name_to_path)
    return concat_paths(paths.collect())

jobs = path_graph.to_job()

@repository
def my_repo():
    return [jobs]

Sorry I missed this! But you got it, it’s the 3.9

list

type

Roger - we're on track to get a fix out for this tomorrow, it's a particularly pathological interaction between a couple of bugs in the python typing module

I’ve definitely heard that one before 😬

For your consideration:

>> inspect.isclass(list[str])

True

>> inspect.isclass(DagsterType)

True

>> issubclass(list[str], DagsterType)

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/dgibson/.pyenv/versions/3.9.11/lib/python3.9/abc.py", line 123, in _subclasscheck return _abc_subclasscheck(cls, subclass) TypeError: issubclass() arg 1 must be a class_

>>> inspect.isclass(list[str])
True

and

TypeError: issubclass() arg 1 must be a class_

oh python lol