https://dagster.io/ logo
#dagster-plus
Title
# dagster-plus
i

Ivan Trajkovic

04/25/2023, 7:20 PM
hello all, I have a quick question about an issue we're having with Hybrid deployments, it would be a huge help if anyone can point me at the right docs or some pointers. we're deploying the Hybrid agent container on an AWS EC2 instance. We can successfully pull a code location AWS ECR image on that instance. however when we try to materialize assets defined in that code location, that depend on AWS auth ENV vars being defined in their environment, Im getting errors about undefined AWS ENV vars. The AWS auth ENV vars are being defined in the Hybrid agent container. So Im curious about: • how I would go about defining those AWS auth ENV vars in the code location container? • is there some Hybrid agent config that I can add for ENV vars defined in it to get passed to the code location container? (so that the AWS auth ENV vars defined in the agent container can be passed as is to the code location container) • in general, is there some recommended approach for both the agent and code location containers to acquire AWS credentials from the local EC2 instance metadata service? Im not having these issues when running the hybrid agent and code location locally, and just defining the AWS auth creds ENV vars on the Hybrid agent container; the assets execute correctly and py code using boto3 authenticates itself with AWS correctly. Thank you very much for your assistance. Any hints or direction is greatly appreciated! 🙂
I see on the code location container spawned by the agent that there is a
DAGSTER_INJECT_ENV_VARS_FROM_INSTANCE=1
ENV var defined, which Im assuming configs the code server to inject ENV vars from the host into the continer. so for starters I could resolve the AWS auth creds from the EC2 instance metadata service into explicint ENV vars on the instance. The issue there is that those creds would expire after a period of time, since they're tied to a IAM role. Ideally there is some recommended approach where the code location container could resolve those creds with the instance metadata service.
i

Ivan Trajkovic

04/27/2023, 10:07 PM
that solved it, thank you @alex! 🙂 appreciate it
dagsir 1
3 Views