19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
k
Kevin Day
12/13/2022, 2:40 AMI also am stuck getting the above type of error, DagsterUserCodeUnreachableError, in Dagster Cloud but not for a sensor. It's for an asset materialization that worked fine until my latest deployment. The stack trace doesn't have any of my code in it though. How can I debug this?
dagster._core.errors.DagsterUserCodeUnreachableError: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server
Stack Trace:
File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 822, in _process_api_request
api_result = self._handle_api_request(
File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 601, in _handle_api_request
serialized_snapshot_or_error = client.execution_plan_snapshot(
File "/dagster/dagster/_grpc/client.py", line 201, in execution_plan_snapshot
res = self._query(
File "/dagster/dagster/_grpc/client.py", line 141, in _query
raise DagsterUserCodeUnreachableError("Could not reach user code server") from e
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1670898150.767518014","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81,"grpc_status":4}"
>
Stack Trace:
File "/dagster/dagster/_grpc/client.py", line 139, in _query
return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
File "/dagster/dagster/_grpc/client.py", line 129, in _get_response
return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
File "/dagster-cloud-backend/dagster_cloud_backend/daemon/event_log_consumer/updated_run_consumer.py", line 43, in consume_updated_runs
retry_run(workspace_process_context, run, retry_number)
File "/dagster-cloud-backend/dagster_cloud_backend/daemon/event_log_consumer/retry_utils.py", line 98, in retry_run
new_run = instance.create_reexecuted_run(
File "/dagster/dagster/_core/instance/__init__.py", line 1221, in create_reexecuted_run
external_execution_plan = repo_location.get_external_execution_plan(
File "/dagster-cloud-backend/dagster_cloud_backend/user_code/workspace.py", line 452, in get_external_execution_plan
execution_plan_snapshot_or_error = self.api_call(
File "/dagster-cloud-backend/dagster_cloud_backend/user_code/workspace.py", line 362, in api_call
return dagster_cloud_api_call(
File "/dagster-cloud-backend/dagster_cloud_backend/user_code/workspace.py", line 129, in dagster_cloud_api_call
for result in gen_dagster_cloud_api_call(
File "/dagster-cloud-backend/dagster_cloud_backend/user_code/workspace.py", line 261, in gen_dagster_cloud_api_call
raise DagsterUserCodeUnreachableError(error_infos[0].to_string())FYI, I'm not expecting a reply this evening. Just posting it here for the morning.
d
daniel
12/13/2022, 3:51 PMHi Kevin - happy to help get to the bottom of this, is this something you are seeing every time you launch a run?
In the short term I think what I would recommend is disabling non-isolated runs as described here: https://docs.dagster.io/dagster-cloud/deployment/serverless#non-isolated-runs - that will make the runs take a bit longer to start up but would likely resolve the specific issue that you're seeing here
k
Kevin Day
12/13/2022, 4:04 PMThanks! I don't think that's enabled for me now.
I am working on something that might be the root cause.
d
daniel
12/13/2022, 4:04 PMAre you sure? I believe it's on by default for now for runs that are launched from the launchpad
k
Kevin Day
12/13/2022, 4:04 PMOh, interesting. I didnt know.
Because they work fine when run on a schedule.
d
daniel
12/13/2022, 4:04 PMIf you set your deployment settings to
non_isolated_runs:
enabled: Falsek
Kevin Day
12/13/2022, 4:04 PMso maybe that's it?
I'll try that now.
I would also get this error when viewing dagit, I think because my asset graph had one layer with 15 similar assets that feed into others. I started getting this type of error once I added those:
Operation name: AssetGraphLiveQuery
Message: Internal Server Error (Trace ID: 1781565868964960348)
Path: ["assetNodes",6,"projectedLogicalVersion"]
Locations: [{"line":44,"column":3}]But I'm refactoring it now to combine those into one asset instead for now.
The isolated run seems to have fixed that DagsterUserCodeUnreachableError issue. My run is now running.
d
daniel
12/13/2022, 4:32 PMOK, glad its back up and running - we may turn this feature off by default to prevent things like this from happening without people realizing, apologies for the bad experience there
k
Kevin Day
12/13/2022, 4:32 PMNo problem!
also, I had turned off "fast deployments" a week or so ago after I had a snag and saw a thread about them causing an issue. Are fast deployments now working again in the Github Action deployment to Dagster Cloud?
d
daniel
12/13/2022, 4:34 PMA couple of snags have been fixed - do you have the error that you ran into handy?
k
Kevin Day
12/13/2022, 4:36 PM> [ 9/11] RUN if [ -f "requirements.txt" ]; then pip install -r requirements.txt; fi:
#15 37.31 The conflict is caused by:
#15 37.31 The user requested dagster==1.1.3
#15 37.31 dagit 1.1.3 depends on dagster==1.1.3
#15 37.31 dagster-cloud 1.0.17 depends on dagster==1.0.17
#15 37.31
#15 37.31 To fix this you could try to:
#15 37.31 1. loosen the range of package versions you've specified
#15 37.31 2. remove package versions to allow pip attempt to solve the dependency conflict
#15 37.31
#15 37.31 ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
------
Dockerfile:24
--------------------
23 | # Install the rest of dependencies in case there is a requirements.txt
24 | >>> RUN if [ -f "requirements.txt" ]; then \
25 | >>> pip install -r requirements.txt; \
26 | >>> fi
27 |
--------------------
ERROR: failed to solve: process "/bin/sh -c if [ -f \"requirements.txt\" ]; then pip install -r requirements.txt; fi" did not complete successfully: exit code: 1
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c if [ -f \"requirements.txt\" ]; then pip install -r requirements.txt; fi" did not complete successfully: exit code: 1d
daniel
12/13/2022, 4:36 PMIs it possible to share your requirements.txt?
k
Kevin Day
12/13/2022, 4:37 PMaeppl==0.0.35
aesara==2.8.2
aiobotocore==2.4.0
aiohttp==3.8.1
aioitertools==0.10.0
aiosignal==1.2.0
alembic==1.8.1
amqp==5.1.1
aniso8601==9.0.1
anyio==3.6.1
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arviz==0.12.1
asttokens==2.0.8
async-timeout==4.0.2
atomicwrites==1.4.1
attrs==22.1.0
Babel==2.10.3
backcall==0.2.0
backoff==1.11.1
beautifulsoup4==4.11.1
billiard==3.6.4.0
black==22.8.0
bleach==5.0.1
blinker==1.5
bokeh==2.4.3
boto3==1.24.59
botocore==1.27.59
cachetools==5.2.0
celery==5.2.7
certifi==2022.6.15
cffi==1.15.1
cftime==1.6.1
charset-normalizer==2.1.1
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
cloudpickle==2.1.0
coiled==0.2.27
colorama==0.4.5
coloredlogs==14.0
comet-ml==3.31.12
commonmark==0.9.1
configobj==5.0.6
cons==0.4.5
croniter==1.3.5
cycler==0.11.0
dagit==1.1.3
dagster==1.1.3
dagster-cloud==1.1.3
dagster-cloud-cli==1.1.3
dagster-graphql==1.1.3
dask==2022.8.1
debugpy==1.6.3
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.13
distributed==2022.8.1
docker==6.0.0
docstring-parser==0.15
dulwich==0.20.45
elastic-transport==8.4.0
elasticsearch==8.4.0
entrypoints==0.4
envkey==2.0.7
etuples==0.3.5
everett==3.0.0
executing==1.0.0
factory-boy==3.2.1
Faker==14.1.1
fastapi==0.81.0
fastjsonschema==2.16.1
fastprogress==1.0.3
filelock==3.8.0
Flask==2.2.2
Flask-Migrate==3.1.0
flask-redis==0.4.0
Flask-RESTful==0.3.9
Flask-SQLAlchemy==2.5.1
flower==1.2.0
fonttools==4.37.1
frozenlist==1.3.1
fsspec==2022.8.2
gql==3.4.0
graphene==3.1.1
graphql-core==3.2.3
graphql-relay==3.2.0
graphviz==0.20.1
greenlet==1.1.3
grpcio==1.47.2
grpcio-health-checking==1.43.0
gunicorn==20.1.0
h11==0.13.0
HeapDict==1.0.1
hiredis==2.0.0
honcho==1.1.0
htmlmin==0.1.12
httptools==0.5.0
humanfriendly==10.0
humanize==4.3.0
idna==3.3
ImageHash==4.2.1
importlib-metadata==4.12.0
importlib-resources==5.9.0
infinity==1.5
iniconfig==1.1.1
intervals==0.9.2
ipykernel==6.15.2
ipython==8.4.0
ipython-genutils==0.2.0
ipywidgets==7.7.2
itsdangerous==2.1.2
jedi==0.18.1
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.1.0
json5==0.9.10
jsondiff==2.0.0
jsonschema==4.15.0
jupyter-client==7.3.4
jupyter-core==4.11.1
jupyter-server==1.18.1
jupyterlab==3.4.5
jupyterlab-pygments==0.2.2
jupyterlab-widgets==1.1.1
jupyterlab_server==2.15.1
kiwisolver==1.4.4
kombu==5.2.4
lightgbm==3.3.2
llvmlite==0.39.0
locket==1.0.0
logical-unification==0.4.5
lxml==4.9.1
Mako==1.2.2
MarkupSafe==2.1.1
marshmallow==3.17.1
marshmallow-oneofschema==3.0.1
matplotlib==3.5.3
matplotlib-inline==0.1.6
miniKanren==1.0.3
missingno==0.5.1
mistune==2.0.4
mlxtend==0.20.0
msgpack==1.0.4
multidict==6.0.2
multimethod==1.8
multipledispatch==0.6.0
mypy-extensions==0.4.3
nbclassic==0.4.3
nbclient==0.6.7
nbconvert==7.0.0
nbformat==5.4.0
nest-asyncio==1.5.5
netCDF4==1.6.0
networkx==2.8.6
notebook==6.4.12
notebook-shim==0.1.0
numba==0.56.0
numpy==1.22.4
packaging==21.3
pandas==1.4.4
pandas-profiling==3.2.0
pandocfilters==1.5.0
parso==0.8.3
partd==1.3.0
pathspec==0.10.0
pendulum==2.1.2
pexpect==4.8.0
phik==0.12.2
pickleshare==0.7.5
Pillow==9.2.0
pkgutil_resolve_name==1.3.10
platformdirs==2.5.2
plotly==5.10.0
pluggy==1.0.0
prefect==1.3.0
prometheus-client==0.14.1
prompt-toolkit==3.0.30
protobuf==3.20.3
psutil==5.9.1
psycopg2-binary==2.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
py==1.11.0
pycparser==2.21
pydantic==1.10.1
Pygments==2.13.0
pymc==4.1.7
pyparsing==3.0.9
pyrsistent==0.18.1
pytest==7.1.2
python-box==6.0.2
python-dateutil==2.8.2
python-dotenv==0.20.0
python-multipart==0.0.5
python-slugify==6.1.2
pytz==2022.2.1
pytzdata==2020.1
PyWavelets==1.3.0
PyYAML==6.0
pyzmq==23.2.1
questionary==1.10.0
redis==4.3.4
requests==2.28.1
requests-toolbelt==0.9.1
rich==12.5.1
rollbar==0.16.3
Rx==1.6.1
s3fs==2022.8.2
s3transfer==0.6.0
scikit-learn==1.1.2
scipy==1.9.1
seaborn==0.11.2
semantic-version==2.10.0
semver==2.13.0
Send2Trash==1.8.0
sentry-sdk==1.9.6
shellingham==1.5.0
six==1.16.0
slugify==0.0.1
sniffio==1.2.0
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.40
SQLAlchemy-Utils==0.38.3
stack-data==0.5.0
starlette==0.19.1
tabulate==0.8.10
tangled-up-in-unicode==0.2.0
tblib==1.7.0
tenacity==8.0.1
terminado==0.15.0
text-unidecode==1.3
textual==0.1.18
threadpoolctl==3.1.0
tinycss2==1.1.1
toml==0.10.2
tomli==2.0.1
toolz==0.12.0
toposort==1.7
tornado==6.1
tqdm==4.64.0
traitlets==5.3.0
typer==0.6.1
typing-compat==0.1.0
typing_extensions==4.3.0
universal_pathlib==0.0.21
urllib3==1.26.12
uvicorn==0.18.3
uvloop==0.17.0
vine==5.0.0
visions==0.7.4
watchdog==2.1.9
watchfiles==0.18.1
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.3
websockets==10.4
Werkzeug==2.2.2
widgetsnbextension==3.6.1
wrapt==1.14.1
wurlitzer==3.0.2
xarray==2022.6.0
xarray-einstats==0.3.0
xgboost==1.6.2
yarl==1.8.1
zict==2.2.0
zipp==3.8.1d
daniel
12/13/2022, 4:37 PMAny idea where that dagster-cloud 1.0.17 would be coming from? I don't see that in the requirements.txt and I don't think we specify it anywhere
(from the error message)
k
Kevin Day
12/13/2022, 4:38 PMNo, not really. I can't remember what exactly I installed to get it running.
Should I delete it?
d
daniel
12/13/2022, 4:38 PMSorry, delete what exactly?
k
Kevin Day
12/13/2022, 4:39 PMSorry, should I uninstall dagster-cloud?
d
daniel
12/13/2022, 4:39 PMYou want to have dagster-cloud installed, but I'm having trouble seeing why the github action would be trying to install it at 1.0.17
#15 37.31 The conflict is caused by:
#15 37.31 The user requested dagster==1.1.3
#15 37.31 dagit 1.1.3 depends on dagster==1.1.3
#15 37.31 dagster-cloud 1.0.17 depends on dagster==1.0.17Maybe if you could share the full logs from the github action that failed?
k
Kevin Day
12/13/2022, 4:42 PMI don't think so
sure
d
daniel
12/13/2022, 4:47 PMVery strange - I see "Collecting dagster-cloud==1.0.17" in there which implies that there's a pin somewhere
k
Kevin Day
12/13/2022, 4:47 PMInteresting
d
daniel
12/13/2022, 4:47 PMAre you completely certain that your requirements.txt hasn't been updated since then?
k
Kevin Day
12/13/2022, 4:48 PMI don't think so, but I can do another try of it now.
I have one slow deploy running now. As soon as it finishes, I'll try the fast deploy
s
Shalabh Chaturvedi
12/13/2022, 5:02 PMHi Kevin - could you also please share your
setup.pyk
Kevin Day
12/13/2022, 5:03 PMI don't seem to have one for this project.
s
Shalabh Chaturvedi
12/13/2022, 5:04 PMok
I'm trying to reproduce the issue right now. Do you know if the requirements.txt was generated from another source since it has a long list of fully pinned dependencies? Do you have Pipfile in the same directory?
k
Kevin Day
12/13/2022, 5:26 PMI had been using Pipenv previously, but switched to just pip install after Pipenv got too slow resolving dependencies.
I just install them and then pip freeze into requirements.txt
s
Shalabh Chaturvedi
12/13/2022, 5:34 PMAh I see.
After analyzing the above requirements.txt with pipenv, I was not able to see where dagster-cloud version 1.0.17 is coming from. Trying a different way to repro the issue now.
I am unable to reproduce the issue after I copied the above
requirements.txt#13 114.3 Successfully installed Babel-2.10.3 Deprecated-1.2.13 Faker-14.1.1 Flask-2.2.2 Flask-Migrate-3.1.0 Flask-RESTful-0.3.9 Flask-SQLAlchemy-2.5.1 HeapDict-1.0.1 ImageHash-4.2.1 Jinja2-3.1.2 Mako-1.2.2 MarkupSafe-2.1.1 Pillow-9.2.0 PyWavelets-1.3.0 PyYAML-6.0 Pygments-2.13.0 Rx-1.6.1 SQLAlchemy-1.4.40 SQLAlchemy-Utils-0.38.3 Send2Trash-1.8.0 Werkzeug-2.2.2 aeppl-0.0.35 aesara-2.8.2 aiobotocore-2.4.0 aiohttp-3.8.1 aioitertools-0.10.0 aiosignal-1.2.0 alembic-1.8.1 amqp-5.1.1 aniso8601-9.0.1 anyio-3.6.1 appnope-0.1.3 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 arviz-0.12.1 asttokens-2.0.8 async-timeout-4.0.2 atomicwrites-1.4.1 attrs-22.1.0 backcall-0.2.0 backoff-1.11.1 beautifulsoup4-4.11.1 billiard-3.6.4.0 black-22.8.0 bleach-5.0.1 blinker-1.5 bokeh-2.4.3 boto3-1.24.59 botocore-1.27.59 cachetools-5.2.0 celery-5.2.7 certifi-2022.6.15 cffi-1.15.1 cftime-1.6.1 charset-normalizer-2.1.1 click-8.1.3 click-didyoumean-0.3.0 click-plugins-1.1.1 click-repl-0.2.0 cloudpickle-2.1.0 coiled-0.2.27 colorama-0.4.5 colo...
#13 114.3 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: <https://pip.pypa.io/warnings/venv>
#13 114.5 WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
#13 114.5 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#13 DONE 116.3s
#14 [8/9] RUN if [ -f "dagster_cloud_post_install.sh" ]; then ./dagster_cloud_post_install.sh; fi
#14 DONE 0.6s
#15 [9/9] RUN dagster-cloud --version
#15 1.151 Dagster Cloud version: 1.1.3
#15 DONE 1.3s
#16 exporting to image
#16 exporting layers
#16 exporting layers 109.1s done
#16 exporting manifest sha256:c30a40b0a642139a2d0f24ffeee1cae3b21904d9852d9440afb1a3bcb4c0e124 done
#16 exporting config sha256:ecae1130013760c1a2e2717d0b2f10708e7e463c3ff4405e52a00b656225bb57 done
#16 pushing layersdagster_cloud.yamlk
Kevin Day
12/13/2022, 5:41 PMYes, I'll do that now.
Also, to initiate fast deploys, do I just add this line to the action file?
ENABLE_FAST_DEPLOYS: 'true'
s
Shalabh Chaturvedi
12/13/2022, 5:43 PMyes, under the top level
env:k
Kevin Day
12/13/2022, 6:21 PMThanks
Sorry, I'm encountering a different issue at the moment. I updated my assets and I'm getting this error now: botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
I'm not using S3 for my assets. Does the serverless use S3 behind the scenes?
s
Shalabh Chaturvedi
12/13/2022, 6:24 PMgood it know it got built and deployed at least - was it using fast deploys?
yes serverless uses s3 behind the scenes. we can look into this at our end. did you switch from hybrid to serverless at some point?
k
Kevin Day
12/13/2022, 6:25 PMSorry, I'm testing the fast deploy now.
No, but I get odd behavior like that when I use assets that return None.
I've changed it to return just a bool of True, and I think that might fix it.
Fast deploy is running now.
s
Shalabh Chaturvedi
12/13/2022, 6:27 PMok let us know how that goes. I don't think returning
Nonek
Kevin Day
12/13/2022, 6:27 PMSo the fast deploy is still going. "Still waiting for agent to sync changes to baker. This can take a few minutes."
d
daniel
12/13/2022, 6:40 PMThe first one can take a bit but subsequent ones should be speedier. Our logs show that it completed but then had an error loading the code
k
Kevin Day
12/13/2022, 7:21 PMYes. Here it is:
Error loading baker: {'__typename': 'PythonError', 'message': "dagster._core.errors.DagsterImportError: Encountered ImportError: `No module named 'baker'` while importing module baker. Local modules were resolved using the working directory `/`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets. \n", 'stack': [' File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_grpc/server.py", line 241, in __init__\n self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(\n', ' File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_grpc/server.py", line 104, in __init__\n loadable_targets = get_loadable_targets(\n', ' File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_grpc/utils.py", line 53, in get_loadable_targets\n else loadable_targets_from_python_package(package_name, working_directory)\n', ' File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_core/workspace/autodiscovery.py", line 49, in loadable_targets_from_python_package\n module = load_python_module(\n', ' File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_core/code_pointer.py", line 143, in load_python_module\n raise DagsterImportError(\n']}
ERROR:root:Error updating code location 'baker'
Traceback (most recent call last):
File "/home/runner/.pex/unzipped_pexes/49372ab9408277f5d7bd2e77673bec0f851c7a57/builder/deploy.py", line 482, in run_code_location_update
code_location.wait_for_load(
File "/home/runner/.pex/unzipped_pexes/49372ab9408277f5d7bd2e77673bec0f851c7a57/builder/code_location.py", line 38, in wait_for_load
workspace.wait_for_load(
File "/home/runner/.pex/installed_wheels/3246dfc7dfca70bb82fbb6c13331f05c2566ab79f8354565eb769bd1e9634f0d/dagster_cloud_cli-1.0.16-py3-none-any.whl/dagster_cloud_cli/commands/workspace/__init__.py", line 185, in wait_for_load
raise ui.error(error_string)
click.exceptions.Exit
INFO:root:Not within a PR, not adding PR comment.
Error: Deploy failed. To view status of your code locations, visit ***/prod/instance/code-locations
Error: Process completed with exit code 1.And from the console:
dagster._core.errors.DagsterImportError: Encountered ImportError: `No module named 'baker'` while importing module baker. Local modules were resolved using the working directory `/`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets.
File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_grpc/server.py", line 241, in __init__
self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_grpc/server.py", line 104, in __init__
loadable_targets = get_loadable_targets(
File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_grpc/utils.py", line 53, in get_loadable_targets
else loadable_targets_from_python_package(package_name, working_directory)
File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_core/workspace/autodiscovery.py", line 49, in loadable_targets_from_python_package
module = load_python_module(
File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_core/code_pointer.py", line 143, in load_python_module
raise DagsterImportError(
The above exception was caused by the following exception:
ModuleNotFoundError: No module named 'baker'
File "/root/.pex/installed_wheels/7221994bcf7469a7afb502485ec1c37dabf9050df04ffbd39305a3c72b694b4a/dagster-1.1.3-py3-none-any.whl/dagster/_core/code_pointer.py", line 138, in load_python_module
return importlib.import_module(module_name)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlockedI'm doing a non-fast deploy now to get back to functional.
s
Shalabh Chaturvedi
12/13/2022, 7:58 PMdid the non-fast deploy work? also where does the 'baker' package come from?
k
Kevin Day
12/13/2022, 8:09 PMIt's in the root directory of the project
the non-fast deploy worked
s
Shalabh Chaturvedi
12/13/2022, 8:14 PMI see. I expected the fast deploy to also work in this situation and I will debug at my end.
:rainbow-daggy: 1
k
Kevin Day
12/13/2022, 8:15 PMWould it help if I created a setup.py?
s
Shalabh Chaturvedi
12/13/2022, 8:19 PMYes that would make it a proper package and it should work. Fast deploys uses
setup.pysetup.pyk
Kevin Day
12/13/2022, 9:15 PMFWIW, in the discussion above about the None return from the asset borking things, I was right. I changed it to return True, and then it worked. I also had a local issue with assets that returned empty, where dagster failed looking for an actual value. It's not really an issue for me to return True, just thought I would pass it along.
For the package, I will try that out tomorrow. Thanks!
d
daniel
12/13/2022, 9:16 PMBelieve we have a fix out for the borking: https://github.com/dagster-io/dagster/pull/11083 that should go out in the release later this week
k
Kevin Day
12/14/2022, 1:38 PMGreat, thanks!