https://dagster.io/ logo
Join Slack
Powered by
Hello, I am trying to deploy a Github project to D...
# ask-community
r
Hello, I am trying to deploy a Github project to Dagster cloud. The deploy triggered via Github actions fails with this root error:
Copy code
ERROR:root:b"pid 1790 -> /home/runner/work/trx_etl/trx_etl/build/.pex/venvs/b9b8c4d1ace8e0b01c93d90648332402b22cf90d/0c9ffbfa1b4080a12413e6319eb7d3da7eff4f40/bin/python -sE /home/runner/work/trx_etl/trx_etl/build/.pex/venvs/b9b8c4d1ace8e0b01c93d90648332402b22cf90d/0c9ffbfa1b4080a12413e6319eb7d3da7eff4f40/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/runner/work/trx_etl/trx_etl/build/.pex/pip_cache download --dest /home/runner/work/trx_etl/trx_etl/build/.pex/downloads/resolver_download.fkkuxf9x/opt.hostedtoolcache.Python.3.8.15.x64.bin.python3.8 --requirement /home/runner/work/trx_etl/trx_etl/build/deps-requirements-18a180dd18384e5a8abcf4448807cd0618dfdca8.txt --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:\nERROR: Double requirement given: dagster==1.0.15 (from -r /home/runner/work/trx_etl/trx_etl/build/deps-requirements-18a180dd18384e5a8abcf4448807cd0618dfdca8.txt (line 28)) (already in dagster (from -r /home/runner/work/trx_etl/trx_etl/build/deps-requirements-18a180dd18384e5a8abcf4448807cd0618dfdca8.txt (line 24)), name='dagster')\n\n"
81
My 
requirements.txt
currently lists: 
dagster==1.0.15
dagster-graphql==1.0.15
The project runs well locally. Any hints to what might be going on?
🤖 1
Complete log attached.
logs_dagster.txt
s
Hi Rodrigo - can you please check if you have the same requirement also specified in 
setup.py
? If so you can just add the version to 
setup.py
and remove it from 
requirements.txt
.
Looking closer at your logs the 
dagster
dependency is includes two times:
Copy code
dagster   # here
dagster-aws
dagster-cloud
dagster-graphql==1.0.15
dagster==1.0.15   # here
I suspect one is coming from 
setup.py
and one from 
requirements.txt
. We'll make the de-duping better at our end but in the meantime see if removing one of these helps.
r
thanks for the pointer, I’ll try that
Did what you suggested and now the error changed to: 
ERROR:root:b'Failed to resolve compatible distributions:\n1: grpcio-status==1.51.1 requires protobuf>=4.21.6 but protobuf 3.20.3 was resolved\n2: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved\n'
pip install -e ".[dev]"
runs successfully on a clean venv on my local machine.
s
Looks like the fast deploy build is running into some dependency conflict while pip works fine. I'll keep looking into this - can you please send me your new build log file? In the meantime, you could try turning off fast deploys by commenting out the 
ENABLE_FAST_DEPLOYS
line in your 
.github/workflows/*.yml
files.
r
logs-dagster-2.txt
thank you box 1
s
Also it would be super helpful if you can share your 
setup.py
, 
requirements.txt
, or just the list of dependencies as specified in those two files. That should let me reproduce the problem at my end.
r
from setuptools import find_packages, setup
if __name__ == "__main__":
setup(
name="my_dagster_project",
packages=find_packages(exclude=["my_dagster_project_tests"]),
install_requires=[
"dagster",
"dagster-aws",
"dagster-cloud",
"firebase-admin",
"pandas",
"pyarrow",
"boto3"
],
extras_require={"dev": ["dagit", "pytest"]},
)
I got rid of 
requirements.txt
s
OK. Here's the line I referred to above: https://github.com/dagster-io/quickstart-etl/blob/main/.github/workflows/deploy.yml#L14 Deleting or commenting out this line will use full Docker builds for deploys. That uses pip so hopefully it lets you continue, even though the deploy will be slower (~3 minutes)
r
thanks, tried that and the workflow retriggered, lets see how it goes
the first job’s green now
s
The real work happens in the second job for these Docker deploys.
r
gotcha, past runs failed on the first job already though, so we are making progress
the second job failed
it’s a dep error on my end
s
odd, is it the the same error regarding 
grpcio
?
r
nope, I’m using the Arrow lib to handle dates, the right dependency is 
arrow
not 
pyarrow
If I retry adding the code location here, would it rewrite the 
deploy.yml
file?
the build in Github is otherwise green
d
Hi rodrigo - I think it would rewrite the deploy.yml, yeah - is there a particular reason you want to do that? Once the github action completes, the code location will show up on that page
r
I needed to disable fast deploys to get the build to run. Otherwise, deps did not install correctly. The actual error and the logs are above on the thread.
d
That makes sense, but I don't totally follow why you're retrying adding the code location from the UI - is the goal to retry the build/deploy?
If so that can be done from the Github action UI
r
nope, I just want the Github repo to appear on my list of code locations (it does not, currently, because the builds were failing when triggered by the UI)
message has been deleted
d
Is that screenshot from a previous time when it was appearing?
sorry if i'm missing something obvious
r
no, the screenshot is current. The build is successful on Github, but I need to add the project as a code location to Dagster Cloud.
d
Do you possibly have two different code locations with the same "example_location" location name? You shouldn't need to add it manually in the UI once the github repo is created
You can set the name of the location in the dagster_cloud.yaml file: https://github.com/dagster-io/dagster-cloud-serverless-quickstart/blob/main/dagster_cloud.yaml#L2
If two different github repos have the same name, they'll overwrite each other
r
oh, it’s actually me who missed something obvious, due to the location name, sorry. The repo’s already added. Can I rename the location somehow?
d
Yeah, if you change it in the dagster_cloud.yaml, it will upload it with a new name. You may have to remove the old manually
r
got it, thanks and sorry again for the confusion
d
No worries - we should probably make it raise an error (or at least warn) when two different github repos are writing to the same code location
r
my process is now crashing with this error:
Copy code
dagster._core.executor.child_process_executor.ChildProcessCrashException

Stack Trace:
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/executor/multiprocess.py", line 222, in execute
    event_or_none = next(step_iter)
,  File "/usr/local/lib/python3.8/site-packages/dagster/_core/executor/multiprocess.py", line 338, in execute_step_out_of_process
    for ret in execute_child_process_command(multiproc_ctx, command):
,  File "/usr/local/lib/python3.8/site-packages/dagster/_core/executor/child_process_executor.py", line 170, in execute_child_process_command
    raise ChildProcessCrashException(exit_code=process.exitcode)
any way to check the underlying error that caused the child process to crash?
moreover, after installing Dagster via pip using setup.py, which installed version 1.1.6, I’m getting this error when trying to start dagit locally: 
python3.8(49407,0x1f4b06500) malloc: Corruption of free object 0x15164f700: msizes 8/7 disagree
python3.8(49407,0x1f4b06500) malloc: *** set a breakpoint in malloc_error_break to debug
[1]    49407 abort      DAGSTER_HOME=~/.dagster dagit
please ignore the last error, as I fixed it locally, the cloud error remains though
d
Usually what that means is a memory issue. One way to get around that is to launch your run in isolated mode (or turn off non-isolated runs so that every run happens in its own container with more memory allocated): https://docs.dagster.io/dagster-cloud/deployment/serverless#run-isolation
r
thanks, that did it
s
Hi Rodrigo we have released a new version of of our GitHub action that should fix the original dependency issue with fast deploys. https://dagster.slack.com/archives/C01U954MEER/p1670870082287539?thread_ts=1670866987.341699&amp;cid=C01U954MEER If you want to try again you can re-add the 
ENABLE_FAST_DEPLOYS: 'true'
line to your 
.github/workflows/deploy.yml
(reference)
🌈 1
m
@Rodrigo Parra -- I'm running into the same 
msizes 8/7 disagree
issue and am having a hard time tracking it down. How did you end up fixing it?
r
upgrading to Python 3.9
m
Interesting — I’m on 3.10 so perhaps I should downgrade.
r
I was running Python 3.8. Currently on Python 3.9.15
👍 1
m
Odd -- same thing on 3.9.16, too
thanks for sharing your fix
(For anyone who finds this later -- I have no idea why this is happening, but this only happens within VS Code's integrated terminal. If I run it within Terminal, it's fine.)
daggy love 1
Opening a terminal within VS Code was setting the MallocNanoZone environment variable to 
0
, which caused problems with grpc. I updated my environment to explicitly unset MallocNanoZone and now I don't have this problem. It appears this was introduced into VS Code via an Electron change (https://github.com/electron/electron/commit/192a7fad0d548d1883c58bdf95ab7a2ff1391881).
🙏 3
a
This really helped me, thanks 🙏🏻. Was tearing my hair out.
b
Copy code
"terminal.integrated.env.osx": {
        "MallocNanoZone": "1",
  }
❤️ 3
p
Thanks, @Matt Dean & @Benedikt Buchert you saved me a ton of time and frustration! 🙌 I didn't get this until I moved to my M1 MacBook, so I figured it must be related to that. Not getting this on my Intel laptop and I believe VS Code is up to date on that machine. Odd. 🤔
daggy love 2
66 Views
Open in Slack
PreviousNext
Powered by