https://dagster.io/ logo
#dagster-plus
Title
# dagster-plus
s

Simrun Basuita

02/09/2024, 10:07 AM
Hey guys, getting an error deploying to dagster cloud– haven't seen this one before. Any ideas? This is from the github actions workflow. cc @Phillip Marks
Copy code
Error loading spectacle: {'__typename': 'PythonError', 'message': 'dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server\n', 'stack': ['  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1420, in _reconcile\n    self._wait_for_new_server_ready(\n', '  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 558, in _wait_for_new_server_ready\n    self._wait_for_dagster_server_process(\n', '  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1823, in _wait_for_dagster_server_process\n    self._wait_for_server_process(\n', '  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1872, in _wait_for_server_process\n    additional_check()\n', '  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 552, in <lambda>\n    additional_check = lambda: _assert_new_pex_server_did_not_crash()\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 522, in _assert_new_pex_server_did_not_crash\n    .get_crashed_pex_servers(\n     ^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/dagster-cloud/dagster_cloud/pex/grpc/client.py", line 62, in get_crashed_pex_servers\n    res = self._query(\n          ^^^^^^^^^^^^\n', '  File "/dagster-cloud/dagster_cloud/pex/grpc/client.py", line 109, in _query\n    raise DagsterUserCodeUnreachableError("Could not reach user code server") from e\n']}
j

James Robinson

02/09/2024, 10:17 AM
FYI, I have been getting a lot of these kind of errors lately as well
sometimes will work when re-running
s

Simrun Basuita

02/09/2024, 11:24 AM
yeah it worked running a second time
d

daniel

02/09/2024, 12:25 PM
Thanks for the report - we have a fix lined up for our next release that I believe will help with this problem. In the short term I think pinning your code to the 1.6.0 version would work as a workaround until that fix is live, as the problem being fixed was introduced in the 1.6.1 release.
dagster yay 1
s

Simrun Basuita

02/09/2024, 12:25 PM
@Phillip Marks looks like we might need to downgrade to 1.6.0
thanks daniel. when is your next release?
we've disabled fast deploys and it looks like it has successfully deployed
d

daniel

02/09/2024, 12:27 PM
Temporarily changing to ENABLE_FAST_DEPLOYS: false would also solve the problem. We may be able to get this fix out later today without you needing to change anything
👍 1
s

Simrun Basuita

02/09/2024, 12:29 PM
thanks for the information @daniel super helpful
is enable_fast_deploys recommended / stable? seems like every time there's a bug the workaround is to disable it
👍 1
Also, what is the recommended stable version of Dagster to be running? I upgraded us yesterday from 1.5 to 1.6 in the hope it would fix some other bugs.
Do you guys have a stable / LTS cycle?
d

daniel

02/09/2024, 12:32 PM
We don’t currently have an LTS cycle - I’ll pass that feedback on to the rest of the team.
👍 1
@Simrun Basuita @James Robinson can you share your Dagster Cloud organization names? I can verify the fix using that and make sure it goes out to those orgs quickly since you're currently experiencing problems
s

Simrun Basuita

02/09/2024, 1:23 PM
thank you so much, will pm you
@daniel we re-enabled fast deploys because we thought it might fix another issue. And the same error (at the top of this thread) is back
5 Views