https://dagster.io/ logo
Title
r

Rama

12/08/2022, 10:44 AM
Hi! I got an error
dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server
when updating dagster locally to 1.1.1 from 0.14.9. Here's detail error :
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "DNS resolution failed for docker_dbt:4000: C-ares status is not ARES_SUCCESS qtype=A name=docker_dbt is_balancer=0: Domain name not found" debug_error_string = "{"created":"@1670495594.598923552","description":"DNS resolution failed for docker_dbt:4000: C-ares status is not ARES_SUCCESS qtype=A name=docker_dbt is_balancer=0: Domain name not found","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}" >
  File "/opt/conda/lib/python3.9/site-packages/dagster/_grpc/client.py", line 139, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File "/opt/conda/lib/python3.9/site-packages/dagster/_grpc/client.py", line 129, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/opt/conda/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/conda/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
Did I miss something when updating the version?
d

daniel

12/09/2022, 1:47 PM
Hi Rama - this error indicates that your user code server isn’t running. Can you share more information about how/where you’re deploying dagster? (Docker/k8s/locally/etc)
r

Rama

12/11/2022, 5:26 PM
Hi Daniel! I’m deploying locally.
d

daniel

12/11/2022, 5:27 PM
Can you share more details about your code or project, is there a way that we can try to reproduce this problem ourselves?
r

Rama

12/11/2022, 5:35 PM
I cannot share more details about the code cause it’s not my personal project. Do you think I miss something when update the version? What I did was just update the dockerfile
d

daniel

12/11/2022, 5:56 PM
All I can tell you from that error message is that it’s having trouble finding the server where your code is running. I think we will need more information about your setup in order to be helpful here
r

Rama

12/11/2022, 6:30 PM
what information do you need?
Also, I got this error
Traceback (most recent call last):

  File "/opt/conda/lib/python3.9/runpy.py", line 188, in _run_module_as_main

    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)

  File "/opt/conda/lib/python3.9/runpy.py", line 147, in _get_module_details

    return _get_module_details(pkg_main_name, error)

  File "/opt/conda/lib/python3.9/runpy.py", line 158, in _get_module_details

    code = loader.get_code(mod_name)

AttributeError: 'AliasedModuleLoader' object has no attribute 'get_code'
d

daniel

12/11/2022, 6:33 PM
Ultimately what I need is a way to reproduce the problem myself - so if there’s a way to share a version of your Dockerfile or code with identifiable information removed, I could try running it and see if I hit the same problem
I haven’t seen the error you just posted before - it doesn’t look like it’s being raised within dagster and might point to a problem with your python setup
r

Rama

12/12/2022, 10:53 AM
I used same python setup for dagster 0.14.9 version and worked fine. Do you think dagster 1.1.1 only compatible with certain python version?
d

daniel

12/12/2022, 12:10 PM
1.1.1 should work on the same python versions as before. If there are any other logs you can share we could take a look
r

Rama

12/12/2022, 12:57 PM
another error logs
debug_error_string = "{"created":"@1670849761.118030908","description":"DNS resolution failed for docker:4004: C-ares status is not ARES_SUCCESS qtype=A name=docker is_balancer=0: Domain name not found","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}"
d

daniel

12/12/2022, 12:58 PM
I see docker in the error message there - are you deploying dagster on docker? details of that nature will be really helpful to understand what’s going on here and get this fixed
If you’re running on docker, one thing you could do is see if our docker example runs out of the box for you with no changes: https://github.com/dagster-io/dagster/tree/master/examples/deploy_docker
r

Rama

12/14/2022, 2:43 PM
Here's my dockerfile
FROM continuumio/miniconda3:latest

RUN pip install dagster==1.1.1 dagit==1.1.1 dagster-graphql==1.1.1 dagster-postgres==0.17.1 dagster-aws==0.17.1
I think the issue with the docker image
d

daniel

12/14/2022, 3:06 PM
Nothing jumps out to me as a problem from that Dockerfile
Did you have any luck running the docker example? If you can explain how your deployment differs from that example I may be able to be more helpful
Alternately if there’s a set of steps we can run ourselves to reproduce the problem that would also work
r

Rama

12/15/2022, 7:41 PM
yes, I can run the docker example. I don't see any different with mine
here's my dockerfile
FROM continuumio/miniconda3:latest


RUN pip install dagster dagit dagster-graphql dagster-postgres dagster-aws
RUN apt-get update && apt-get install make


# Create envs
RUN conda create -n workspace python=3.8

RUN /opt/conda/envs/workspace/bin/pip install dagster dagit dagster-graphql dagster-postgres dagster-aws boto3 GitPython dagster-docker
You can run this grpc command to get same error using docker_example deployment
python -m dagster.grpc -h 0.0.0.0 -p 4000 -f repo.py
d

daniel

12/15/2022, 9:18 PM
Got it - so just to make sure I understand the repro steps: • Run the docker example locally (by spinning up docker-compose) • Run that command in a separate terminal, it will error with a "grpc._channel._InactiveRpcError"?
r

Rama

12/15/2022, 9:22 PM
Yes, you'll get error like this
Traceback (most recent call last):
  File "/Users/miniconda3/lib/python3.8/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/miniconda3/lib/python3.8/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Users/miniconda3/lib/python3.8/runpy.py", line 153, in _get_module_details
    code = loader.get_code(mod_name)
AttributeError: 'AliasedModuleLoader' object has no attribute 'get_code'
d

daniel

12/15/2022, 9:25 PM
I haven't seen that error before - it doesn't look like its part of a Dagster stack trace. I can ask around with the rest of the team to see if anybody has seen it before, but my suspicion is that the problem lies with your Python environment rather than a problem specific to Dagster. my first recommendation would be to try making a new conda environment and see if the problem persists there.
Errrr wait, hang on - I'm actually reproducing that error with that command 🙂 one moment
Try running this command instead:
python -m dagster._grpc -h 0.0.0.0 -p 4000 -f repo.py
Or even better if this works:
dagster api grpc -h 0.0.0.0 -p 4000 -f repo.py
Did the "python -m dagster.grpc" command come from docs somewhere?
wow, what a cryptic error message!
r

Rama

12/15/2022, 9:34 PM
I read it in https://docs.dagster.io/concepts/code-locations/workspace-files, but maybe the docs already updated. Because I used version 0.14.9
d

daniel

12/15/2022, 9:34 PM
"python -m dagster api grpc" Is actually probably best - I bet that will work
r

Rama

12/15/2022, 9:47 PM
Thank you so much! really appreciate it!🙌
:condagster: 1