Hey all I m trying to get a Dagster hybrid environment up an dagster #ask-community

Hey all I’m trying to get a Dagster hybrid environ...

Nicholas Lilleyman

07/19/2023, 2:33 PM

Hey all I’m trying to get a Dagster hybrid environment up and running, however whenever I try to specify a ‘directory’ and ‘registry’ under ‘build’ in the dagster_cloud.yaml it through an error “Received unexpected config entry “build” at the root.” Its like it doesn’t even accept build as a possible entry in the yaml at all…? This is the documentation I am trying to follow, any ideas, thanks! https://docs.dagster.io/dagster-cloud/managing-deployments/dagster-cloud-yaml#build # dagster_cloud.yaml locations: - location_name: ph-bi-elt code_source: package_name: quickstart_elt build: directory: ./ registry: europe-west1-docker.pkg.dev/********

Sean Lopp

07/19/2023, 3:45 PM

Hey Nic, those entries all need to be under the location_name, so I think this is just an indent error

Copy code

locations:
  - location_name: ph-bi-elt
    code_source:
      package_name: quickstart_elt
    build:
     directory: ./
     registry: europe-west1-docker.pkg.dev/********

Nicholas Lilleyman

07/19/2023, 4:31 PM

@Sean Lopp even in dagster cloud directly I’m still getting the error, is this indent incorrect..? Screenshot of my code locally is below too, looks to be indented correct no?

Sean Lopp

07/19/2023, 4:40 PM

@Shalabh Chaturvedi if you have a chance could you take a look at this?

Shalabh Chaturvedi

07/19/2023, 5:25 PM

The config looks correct to me as well. Nic - can you please copy paste the

dagster_cloud.yaml

here in a code block? Are you getting this error from CI/CD or a specific command?

Nicholas Lilleyman

07/20/2023, 7:57 AM

Copy code

locations:
  - location_name: ph-bi-elt
    code_source:
      package_name: quickstart_elt
    build:
     directory: ./
     registry: europe-west1-docker.pkg.dev/********

Nicholas Lilleyman

07/20/2023, 7:59 AM

this is my yaml @Shalabh Chaturvedi, running the command below gives me the error

Copy code

% dagster-cloud workspace sync -w dagster_cloud.yaml

Shalabh Chaturvedi

07/20/2023, 2:23 PM

Hi @Nicholas Lilleyman, can you try using this command instead:

Copy code

dagster-cloud workspace add-location --location-file=dagster_cloud.yaml --location-name=ph-bi-elt

This is one used in our CICD deploy workflows. I'll look into why the

sync

command seems to be broken right now.

Nicholas Lilleyman

07/20/2023, 2:42 PM

Copy code

Error: Some locations failed to load after being synced by the agent:
Error loading ph-bi-elt: {'__typename': 'PythonError', 'message': "Exception: Your agent's configuration requires you to specify an image. Use the `--image` flag when specifying your location to tell the agent which image to use to load your code.\n", 'stack': ['  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1301, in _reconcile\n    self._check_for_image(code_deployment_metadata)\n', '  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1064, in _check_for_image\n    raise Exception(\n']}

Nicholas Lilleyman

07/20/2023, 2:43 PM

@Shalabh Chaturvedi getting the error above now, any ideas?

Shalabh Chaturvedi

07/20/2023, 3:20 PM

You'll need to specify an

--image

in your registry that you want to deploy. Have you built and published a dagster image to your registry?

Shalabh Chaturvedi

07/20/2023, 3:25 PM

Here is a sample Dockerfile you can use to build a basic dagster project image. You'll need to build and push this to your registry above. Whatever tag you publish this to can then be provided in the

--image

parameter for

add-location

Nicholas Lilleyman

07/21/2023, 8:28 AM

Thanks for the help @Shalabh Chaturvedi. I ran the below and it worked

Copy code

dagster-cloud workspace add-location --location-file=dagster_cloud.yaml --location-name=ph-bi-elt --image=europe-west1-docker.pkg.dev/prj-ops-artifact-registry-707c/container/bi/dagster:5e142d4234fa1e2010eb0d86e1cdef08c434c9bf-5599647162-1-quickstart_etl

That said, I’m overall pretty confused what the Dagster development workflow looks like… Struggling to conceptualise it and explain it to my team, do we: • Make changes to code locally • commit to branch • push to remote • create PR to master • PR triggers GitHub action that dumps an image of my branch into the registry • Run ‘dagster-cloud workspace add-location’ command locally to load this specific image into Dagster Cloud?? • Execute jobs in Dagster Cloud, verify their success • Merge to Master • ...???

Shalabh Chaturvedi

07/21/2023, 8:58 AM

Hi Nic, You don't need to run

dagster-cloud workspace add-location

locally unless you want to update something one-off outside CICD. The github workflow will do everything necessary to update the code location: create the branch deployment for the PR if needed, build and upload the docker image, update the code location to point to the new image. Other than that the workflow above sounds reasonable. If you want to set up GitHub the best template to follow is here: https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/

Nicholas Lilleyman

07/21/2023, 10:20 AM

Hey @Shalabh Chaturvedi I’m using that cloud hybrid quickstart currently. I’ve just created a PR in github and: • image was built and put into the registry • i can see the branch deployment in dagster cloud, but the code location is blank, do I need to manually add the location of the image in the dagster Cloud UI??? Seems a bit odd

Shalabh Chaturvedi

07/21/2023, 8:39 PM

Hi Nic - can you share the logs from the action in GitHub?

Nicholas Lilleyman

07/24/2023, 7:39 AM

@Shalabh Chaturvedi logs attached 🙂

logs_14.zip

Shalabh Chaturvedi

07/24/2023, 5:07 PM

Hi Nic, I'm seeing all the logs except the final deploy to dagster cloud. I've attached a screenshot of how it should look in GitHub. I see logs for

Update build session with image tag

but no logs for

Deploy to dagster cloud

. Can you please verify you have this section in your workflow yaml: https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/blob/46bfd6c31966cda[…]0fba852f5cb2777343a4/.github/workflows/dagster-cloud-deploy.yml

Nicholas Lilleyman

07/24/2023, 5:35 PM

@Shalabh Chaturvedi ahh my bad, I forgot that section was commented out….

logs_16.zip

Shalabh Chaturvedi

07/24/2023, 5:37 PM

Looks like the branch deployment is updating now?

Nicholas Lilleyman

07/24/2023, 5:37 PM

It fails due to some agent issue, I have a running agent in prod but not in the branch deployment

Shalabh Chaturvedi

07/24/2023, 7:19 PM

I see

Error: No Dagster Cloud agent is actively heartbeating. Make sure that you have a Dagster Cloud agent running

. Have you enabled branch deployments on this agent? For eg there's a flag in ECS: https://docs.dagster.io/dagster-cloud/deployment/agents/amazon-ecs/creating-ecs-agent-new-vpc#step-3-configure-the-agent

Shalabh Chaturvedi

07/24/2023, 8:59 PM

For kubernetes you can see the instructions to set the helm values here: https://docs.dagster.io/dagster-cloud/managing-deployments/branch-deployments/using-branch-deployments-with-github#step-2-create-a[…]figure-an-agent

Nicholas Lilleyman

07/25/2023, 8:03 AM

😄 😄 Its working @Shalabh Chaturvedi, thanks so much for you support mate!

🎉 1

3 Views

Open in Slack

Previous Next