https://dagster.io/ logo
Powered by
#deployment-kubernetes
Title
# deployment-kubernetes
j

Jason

02/08/2021, 5:41 PM
Hey all... I had my k8s deployment working with some simple pipelines but now with some slightly more complicated ones things are breaking. This works locally with 
dagit -f repo.py
But my pod is now giving me 
CrashLoopBackOff
with 
kubectle logs <pod>
giving... Any ideas where I should poke first?
Copy code
Traceback (most recent call last):
  File "/usr/local/bin/dagster", line 5, in <module>
    from dagster.cli import main
  File "/usr/local/lib/python3.7/site-packages/dagster/__init__.py", line 116, in <module>
    from dagster.core.launcher import DefaultRunLauncher
  File "/usr/local/lib/python3.7/site-packages/dagster/core/launcher/__init__.py", line 2, in <module>
    from .default_run_launcher import DefaultRunLauncher
  File "/usr/local/lib/python3.7/site-packages/dagster/core/launcher/default_run_launcher.py", line 7, in <module>
    from dagster.core.host_representation import ExternalPipeline
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/__init__.py", line 56, in <module>
    from .repository_location import (
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 6, in <module>
    from dagster.api.snapshot_execution_plan import sync_get_external_execution_plan_grpc
  File "/usr/local/lib/python3.7/site-packages/dagster/api/snapshot_execution_plan.py", line 8, in <module>
    from dagster.grpc.types import ExecutionPlanSnapshotArgs
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/__init__.py", line 11, in <module>
    from .client import DagsterGrpcClient, ephemeral_grpc_api_client
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 15, in <module>
    from grpc_health.v1 import health_pb2
  File "/usr/local/lib/python3.7/site-packages/grpc_health/v1/health_pb2.py", line 21, in <module>
    create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'
a

alex

02/08/2021, 5:46 PM
what version of dagster are you using?
I believe this error comes from the 
protobuf
python dependency being too old. Dagster declares its dep as 
"protobuf>=3.13.0"
so might be worth running 
pip list
in the image to see where things are at
j

Jason

02/08/2021, 5:50 PM
v0.10.2
oh wait... think i say something regarding that
Docker build... 
dagster 0.10.2 requires protobuf>=3.13.0, but you have protobuf 3.11.3 which is incompatible.
Copy code
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
flower 0.9.7 requires celery<5.0.0,>=4.3.0; python_version >= "3.7", but you have celery 5.0.5 which is incompatible.
flower 0.9.7 requires vine==1.3.0, but you have vine 5.0.0 which is incompatible.
dagster 0.10.2 requires protobuf>=3.13.0, but you have protobuf 3.11.3 which is incompatible.
a

alex

02/08/2021, 5:58 PM
do you know whats causing the older version to be installed? do we conflict with another library in use?
j

Jason

02/08/2021, 6:01 PM
Re-ran a few times and it seems to be 
dbt==0.18.1
without it, i'm getting protobuf 3.14.0
a

alex

02/08/2021, 6:12 PM
i dont see direct dep - might be transitive
you could add an explicit install to protobuf to force it to the higher version and see what happens
j

Jason

02/08/2021, 6:19 PM
So it seems like this is related to something I mentioned in the dbt slack channel regarding dagster being downgraded to 0.8.5 when I have a requirements file with both 
dagster-dbt 0.10.4
and 
dbt 0.18.1
With dagster 0.10.4 already installed, I run 
pip install -r requirements.txt
Copy code
[redacted]/.pyenv/versions/3.8.7/envs/dagster-test/lib/python3.8/site-packages (from matplotlib->dagster-pandas->dagster-dbt==0.10.4->-r requirements-test.txt (line 2)) (1.3.1)
Installing collected packages: protobuf, dagster
  Attempting uninstall: protobuf
    Found existing installation: protobuf 3.14.0
    Uninstalling protobuf-3.14.0:
      Successfully uninstalled protobuf-3.14.0
  Attempting uninstall: dagster
    Found existing installation: dagster 0.10.4
    Uninstalling dagster-0.10.4:
      Successfully uninstalled dagster-0.10.4
Successfully installed dagster-0.8.5 protobuf-3.11.3
✅ 1
a

alex

02/08/2021, 6:50 PM
whats in your 
requirements.txt
? does it fail if you try to hard pin 
dagster
and 
protobuf
as well
j

Jason

02/08/2021, 7:03 PM
my Dockerfile looks like this now - seems to be working. Using 
dagster 0.10.4
Seems like protobuf 3.11.3 was due to 
dbt-bigquery
which I don't need but comes along with 
pip install dbt
so the quickest way for a test
Copy code
RUN pip install \
        dagster-dbt==${DAGSTER_VERSION} \
        dagster-shell==${DAGSTER_VERSION} \
        slack-sdk==3.2.1 \
        dbt==0.18.1 \ 
    && pip uninstall -y \
        dbt-bigquery==0.18.1 \
    && pip install \
        dagster==${DAGSTER_VERSION}
v

victor

02/09/2021, 10:14 AM
For a cleaner installation you can also install 
dbt-core
(which does not depend on any DB plugin) as well as the appropriate pluging you use 
dbt-{postgres,redshift,snowflake}
👍🏾 1
✅ 1
b

Brian Abelson

02/10/2021, 3:05 PM
i was experiencing this weird behavior too! 
dagster
got downgraded to 
0.8.5
when i pinned 
dbt==0.18.1
i was trying to pin that dbt version because of this issue, https://github.com/dagster-io/dagster/issues/3616
i posted my workaround in that thread for using 
dbt_cli_run
with 
dbt==0.19.0
Powered by