https://dagster.io/ logo
#ask-community
Title
# ask-community
r

Rodrigo Parra

12/12/2022, 5:43 PM
Hello, I am trying to deploy a Github project to Dagster cloud. The deploy triggered via Github actions fails with this root error:
Copy code
ERROR:root:b"pid 1790 -> /home/runner/work/trx_etl/trx_etl/build/.pex/venvs/b9b8c4d1ace8e0b01c93d90648332402b22cf90d/0c9ffbfa1b4080a12413e6319eb7d3da7eff4f40/bin/python -sE /home/runner/work/trx_etl/trx_etl/build/.pex/venvs/b9b8c4d1ace8e0b01c93d90648332402b22cf90d/0c9ffbfa1b4080a12413e6319eb7d3da7eff4f40/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/runner/work/trx_etl/trx_etl/build/.pex/pip_cache download --dest /home/runner/work/trx_etl/trx_etl/build/.pex/downloads/resolver_download.fkkuxf9x/opt.hostedtoolcache.Python.3.8.15.x64.bin.python3.8 --requirement /home/runner/work/trx_etl/trx_etl/build/deps-requirements-18a180dd18384e5a8abcf4448807cd0618dfdca8.txt --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:\nERROR: Double requirement given: dagster==1.0.15 (from -r /home/runner/work/trx_etl/trx_etl/build/deps-requirements-18a180dd18384e5a8abcf4448807cd0618dfdca8.txt (line 28)) (already in dagster (from -r /home/runner/work/trx_etl/trx_etl/build/deps-requirements-18a180dd18384e5a8abcf4448807cd0618dfdca8.txt (line 24)), name='dagster')\n\n"
81
My
requirements.txt
currently lists:
dagster==1.0.15
dagster-graphql==1.0.15
The project runs well locally. Any hints to what might be going on?
🤖 1
Complete log attached.
s

Shalabh Chaturvedi

12/12/2022, 5:58 PM
Hi Rodrigo - can you please check if you have the same requirement also specified in
setup.py
? If so you can just add the version to
setup.py
and remove it from
requirements.txt
.
Looking closer at your logs the
dagster
dependency is includes two times:
Copy code
dagster   # here
dagster-aws
dagster-cloud
dagster-graphql==1.0.15
dagster==1.0.15   # here
I suspect one is coming from
setup.py
and one from
requirements.txt
. We'll make the de-duping better at our end but in the meantime see if removing one of these helps.
r

Rodrigo Parra

12/12/2022, 6:04 PM
thanks for the pointer, I’ll try that
Did what you suggested and now the error changed to:
ERROR:root:b'Failed to resolve compatible distributions:\n1: grpcio-status==1.51.1 requires protobuf>=4.21.6 but protobuf 3.20.3 was resolved\n2: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved\n'
pip install -e ".[dev]"
runs successfully on a clean venv on my local machine.
s

Shalabh Chaturvedi

12/12/2022, 6:46 PM
Looks like the fast deploy build is running into some dependency conflict while pip works fine. I'll keep looking into this - can you please send me your new build log file? In the meantime, you could try turning off fast deploys by commenting out the
ENABLE_FAST_DEPLOYS
line in your
.github/workflows/*.yml
files.
r

Rodrigo Parra

12/12/2022, 6:49 PM
thank you box 1
s

Shalabh Chaturvedi

12/12/2022, 6:50 PM
Also it would be super helpful if you can share your
setup.py
,
requirements.txt
, or just the list of dependencies as specified in those two files. That should let me reproduce the problem at my end.
r

Rodrigo Parra

12/12/2022, 6:51 PM
from setuptools import find_packages, setup
if __name__ == "__main__":
setup(
name="my_dagster_project",
packages=find_packages(exclude=["my_dagster_project_tests"]),
install_requires=[
"dagster",
"dagster-aws",
"dagster-cloud",
"firebase-admin",
"pandas",
"pyarrow",
"boto3"
],
extras_require={"dev": ["dagit", "pytest"]},
)
I got rid of
requirements.txt
s

Shalabh Chaturvedi

12/12/2022, 6:53 PM
OK. Here's the line I referred to above: https://github.com/dagster-io/quickstart-etl/blob/main/.github/workflows/deploy.yml#L14 Deleting or commenting out this line will use full Docker builds for deploys. That uses pip so hopefully it lets you continue, even though the deploy will be slower (~3 minutes)
r

Rodrigo Parra

12/12/2022, 6:53 PM
thanks, tried that and the workflow retriggered, lets see how it goes
the first job’s green now
s

Shalabh Chaturvedi

12/12/2022, 6:55 PM
The real work happens in the second job for these Docker deploys.
r

Rodrigo Parra

12/12/2022, 6:55 PM
gotcha, past runs failed on the first job already though, so we are making progress
the second job failed
it’s a dep error on my end
s

Shalabh Chaturvedi

12/12/2022, 6:59 PM
odd, is it the the same error regarding
grpcio
?
r

Rodrigo Parra

12/12/2022, 7:00 PM
nope, I’m using the Arrow lib to handle dates, the right dependency is
arrow
not
pyarrow
If I retry adding the code location here, would it rewrite the
deploy.yml
file?
the build in Github is otherwise green
d

daniel

12/12/2022, 7:09 PM
Hi rodrigo - I think it would rewrite the deploy.yml, yeah - is there a particular reason you want to do that? Once the github action completes, the code location will show up on that page
r

Rodrigo Parra

12/12/2022, 7:10 PM
I needed to disable fast deploys to get the build to run. Otherwise, deps did not install correctly. The actual error and the logs are above on the thread.
d

daniel

12/12/2022, 7:11 PM
That makes sense, but I don't totally follow why you're retrying adding the code location from the UI - is the goal to retry the build/deploy?
If so that can be done from the Github action UI
r

Rodrigo Parra

12/12/2022, 7:12 PM
nope, I just want the Github repo to appear on my list of code locations (it does not, currently, because the builds were failing when triggered by the UI)
message has been deleted
d

daniel

12/12/2022, 7:14 PM
Is that screenshot from a previous time when it was appearing?
sorry if i'm missing something obvious
r

Rodrigo Parra

12/12/2022, 7:17 PM
no, the screenshot is current. The build is successful on Github, but I need to add the project as a code location to Dagster Cloud.
d

daniel

12/12/2022, 7:18 PM
Do you possibly have two different code locations with the same "example_location" location name? You shouldn't need to add it manually in the UI once the github repo is created
You can set the name of the location in the dagster_cloud.yaml file: https://github.com/dagster-io/dagster-cloud-serverless-quickstart/blob/main/dagster_cloud.yaml#L2
If two different github repos have the same name, they'll overwrite each other
r

Rodrigo Parra

12/12/2022, 7:20 PM
oh, it’s actually me who missed something obvious, due to the location name, sorry. The repo’s already added. Can I rename the location somehow?
d

daniel

12/12/2022, 7:20 PM
Yeah, if you change it in the dagster_cloud.yaml, it will upload it with a new name. You may have to remove the old manually
r

Rodrigo Parra

12/12/2022, 7:20 PM
got it, thanks and sorry again for the confusion
d

daniel

12/12/2022, 7:21 PM
No worries - we should probably make it raise an error (or at least warn) when two different github repos are writing to the same code location
r

Rodrigo Parra

12/12/2022, 7:33 PM
my process is now crashing with this error:
Copy code
dagster._core.executor.child_process_executor.ChildProcessCrashException

Stack Trace:
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/executor/multiprocess.py", line 222, in execute
    event_or_none = next(step_iter)
,  File "/usr/local/lib/python3.8/site-packages/dagster/_core/executor/multiprocess.py", line 338, in execute_step_out_of_process
    for ret in execute_child_process_command(multiproc_ctx, command):
,  File "/usr/local/lib/python3.8/site-packages/dagster/_core/executor/child_process_executor.py", line 170, in execute_child_process_command
    raise ChildProcessCrashException(exit_code=process.exitcode)
any way to check the underlying error that caused the child process to crash?
moreover, after installing Dagster via pip using setup.py, which installed version 1.1.6, I’m getting this error when trying to start dagit locally:
python3.8(49407,0x1f4b06500) malloc: Corruption of free object 0x15164f700: msizes 8/7 disagree
python3.8(49407,0x1f4b06500) malloc: *** set a breakpoint in malloc_error_break to debug
[1]    49407 abort      DAGSTER_HOME=~/.dagster dagit
please ignore the last error, as I fixed it locally, the cloud error remains though
d

daniel

12/12/2022, 8:38 PM
Usually what that means is a memory issue. One way to get around that is to launch your run in isolated mode (or turn off non-isolated runs so that every run happens in its own container with more memory allocated): https://docs.dagster.io/dagster-cloud/deployment/serverless#run-isolation
r

Rodrigo Parra

12/12/2022, 8:53 PM
thanks, that did it
s

Shalabh Chaturvedi

12/14/2022, 12:32 AM
Hi Rodrigo we have released a new version of of our GitHub action that should fix the original dependency issue with fast deploys. https://dagster.slack.com/archives/C01U954MEER/p1670870082287539?thread_ts=1670866987.341699&amp;cid=C01U954MEER If you want to try again you can re-add the
ENABLE_FAST_DEPLOYS: 'true'
line to your
.github/workflows/deploy.yml
(reference)
🌈 1
m

Matt Dean

12/17/2022, 10:24 PM
@Rodrigo Parra -- I'm running into the same
msizes 8/7 disagree
issue and am having a hard time tracking it down. How did you end up fixing it?
r

Rodrigo Parra

12/18/2022, 1:13 AM
upgrading to Python 3.9
m

Matt Dean

12/18/2022, 1:14 AM
Interesting — I’m on 3.10 so perhaps I should downgrade.
r

Rodrigo Parra

12/18/2022, 1:16 AM
I was running Python 3.8. Currently on Python 3.9.15
👍 1
m

Matt Dean

12/18/2022, 1:24 AM
Odd -- same thing on 3.9.16, too
thanks for sharing your fix
(For anyone who finds this later -- I have no idea why this is happening, but this only happens within VS Code's integrated terminal. If I run it within Terminal, it's fine.)
daggy love 1
Opening a terminal within VS Code was setting the MallocNanoZone environment variable to
0
, which caused problems with grpc. I updated my environment to explicitly unset MallocNanoZone and now I don't have this problem. It appears this was introduced into VS Code via an Electron change (https://github.com/electron/electron/commit/192a7fad0d548d1883c58bdf95ab7a2ff1391881).
🙏 3
a

Alex Claydon

01/31/2023, 7:33 PM
This really helped me, thanks 🙏🏻. Was tearing my hair out.
b

Benedikt Buchert

02/10/2023, 10:59 AM
Copy code
"terminal.integrated.env.osx": {
        "MallocNanoZone": "1",
  }
❤️ 3
p

pdpark

03/20/2023, 11:05 PM
Thanks, @Matt Dean & @Benedikt Buchert you saved me a ton of time and frustration! 🙌 I didn't get this until I moved to my M1 MacBook, so I figured it must be related to that. Not getting this on my Intel laptop and I believe VS Code is up to date on that machine. Odd. 🤔
daggy love 2
64 Views