Hey all... I had my k8s deployment working with some simple pipelines but now with some slightly more complicated ones things are breaking. This works locally with

dagit -f repo.py

But my pod is now giving me

CrashLoopBackOff

with

kubectle logs <pod>

giving... Any ideas where I should poke first?

Traceback (most recent call last):
  File "/usr/local/bin/dagster", line 5, in <module>
    from dagster.cli import main
  File "/usr/local/lib/python3.7/site-packages/dagster/__init__.py", line 116, in <module>
    from dagster.core.launcher import DefaultRunLauncher
  File "/usr/local/lib/python3.7/site-packages/dagster/core/launcher/__init__.py", line 2, in <module>
    from .default_run_launcher import DefaultRunLauncher
  File "/usr/local/lib/python3.7/site-packages/dagster/core/launcher/default_run_launcher.py", line 7, in <module>
    from dagster.core.host_representation import ExternalPipeline
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/__init__.py", line 56, in <module>
    from .repository_location import (
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 6, in <module>
    from dagster.api.snapshot_execution_plan import sync_get_external_execution_plan_grpc
  File "/usr/local/lib/python3.7/site-packages/dagster/api/snapshot_execution_plan.py", line 8, in <module>
    from dagster.grpc.types import ExecutionPlanSnapshotArgs
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/__init__.py", line 11, in <module>
    from .client import DagsterGrpcClient, ephemeral_grpc_api_client
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 15, in <module>
    from grpc_health.v1 import health_pb2
  File "/usr/local/lib/python3.7/site-packages/grpc_health/v1/health_pb2.py", line 21, in <module>
    create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

what version of dagster are you using?

v0.10.2

do you know whats causing the older version to be installed? do we conflict with another library in use?

Re-ran a few times and it seems to be

dbt==0.18.1

i dont see direct dep - might be transitive

So it seems like this is related to something I mentioned in the dbt slack channel regarding dagster being downgraded to 0.8.5 when I have a requirements file with both

dagster-dbt 0.10.4

and

dbt 0.18.1

With dagster 0.10.4 already installed, I run

pip install -r requirements.txt

[redacted]/.pyenv/versions/3.8.7/envs/dagster-test/lib/python3.8/site-packages (from matplotlib->dagster-pandas->dagster-dbt==0.10.4->-r requirements-test.txt (line 2)) (1.3.1)
Installing collected packages: protobuf, dagster
  Attempting uninstall: protobuf
    Found existing installation: protobuf 3.14.0
    Uninstalling protobuf-3.14.0:
      Successfully uninstalled protobuf-3.14.0
  Attempting uninstall: dagster
    Found existing installation: dagster 0.10.4
    Uninstalling dagster-0.10.4:
      Successfully uninstalled dagster-0.10.4
Successfully installed dagster-0.8.5 protobuf-3.11.3

whats in your

requirements.txt

? does it fail if you try to hard pin

dagster

and

protobuf

as well

my Dockerfile looks like this now - seems to be working. Using

dagster 0.10.4

Seems like protobuf 3.11.3 was due to

dbt-bigquery

which I don't need but comes along with

pip install dbt

so the quickest way for a test

RUN pip install \
        dagster-dbt==${DAGSTER_VERSION} \
        dagster-shell==${DAGSTER_VERSION} \
        slack-sdk==3.2.1 \
        dbt==0.18.1 \ 
    && pip uninstall -y \
        dbt-bigquery==0.18.1 \
    && pip install \
        dagster==${DAGSTER_VERSION}

For a cleaner installation you can also install

dbt-core

(which does not depend on any DB plugin) as well as the appropriate pluging you use

dbt-{postgres,redshift,snowflake}

i was experiencing this weird behavior too!

dagster

got downgraded to

0.8.5

when i pinned

dbt==0.18.1