https://dagster.io/ logo
Join Slack
Powered by
Hey all, I'm running into some issues with Dagster...
# ask-community
a
Hey all, I'm running into some issues with Dagster cloud, I have my agent running within AWS ECS, and am hitting some errors
Copy code
Task logs:
[32m2023-07-18 20:25:49 +0000[0m - dagster.code_server - [34mINFO[0m - Started Dagster code server for package integrations_core on port 4000 in process 1

For more information about the failure, check the ECS console for logs for task arn:aws:ecs:us-east-2:ACCOUNT:task/Dagster-Cloud-mammoth-prod-Cluster/TASKID in cluster Dagster-Cloud-mammoth-prod-Cluster.

Most recent connection error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE

Stack Trace:
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1745, in _wait_for_server_process
So it seems that the actual code location is loading up ok in ECS, but the Dagster Cloud UI is throwing the above error Any ideas?
🤖 1
Both my ECS services seem to be functional, with nothing racey in their logs Where is this user code server that it's trying to reach?
To the extent it's useful, I deployed the agent through the ECS cloudformation stack
Hmmmm the domain it seems to be trying to hit doesn't exist AFAIK
Copy code
details = "DNS resolution failed for integrationscore-245446153a6ee3b4ba2775dd9f56563f89dd-3183fc.dagster-agent-mammoth-prod-029c8e04ce93.local:4000: C-ares status is not ARES_SUCCESS qtype=A name=integrationscore-245446153a6ee3b4ba2775dd9f56563f89dd-3183fc.dagster-agent-mammoth-prod-029c8e04ce93.local is_balancer=0: Domain name not found"
What's determining this URL?
I'm also not totally sure what's mapping port 4000, looking at the ECS task_definition that gets made nothing's mapping it? But the generation of this happens behind the scenes through Dagster, so I'm not sure how to make sure that port's mapped
Copy code
{
    "taskDefinitionArn": "arn:aws:ecs:us-east-2:ACCOUNT:task-definition/server_mammoth_prod_integrations_core_ab141c3f:5",
    "containerDefinitions": [
        {
            "name": "dagster",
            "image": "<http://ghcr.io/mammothclimate/dagster_jobs:latest|ghcr.io/mammothclimate/dagster_jobs:latest>",
            "repositoryCredentials": {
                "credentialsParameter": "arn:aws:secretsmanager:us-east-2:ACCOUNT:secret:github_pat-7589400-nreQbj"
            },
            "cpu": 0,
            "portMappings": [],
            "essential": true,
            "command": [
                "dagster",
                "api",
                "grpc"
            ],
            "environment": [
                {
                    "name": "DAGSTER_CLI_API_GRPC_HOST",
                    "value": "0.0.0.0"
                },
                {
                    "name": "DAGSTER_CLI_API_GRPC_PORT",
                    "value": "4000"
                },
Ah I see, that name it's referencing is from the cloud map namespace service I'm not very familiar with AWS Cloud Map, is there anything necessary to ensure the port is reachable from the agent to the code server?
To answer my own question, the fix was here: https://discuss.dagster.io/t/2651263/hi-team-i-m-trying-to-add-a-code-location-but-i-m-running-in#b037a5b7-2dca-4dcc-93d6-8f716544080b To ensure that the VPC has 
dns_hostnames
enabled, flipping that on I was able to deploy successfully
👍 1
5 Views
Open in Slack
PreviousNext
Powered by