The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Has anyone seen problems with `dagster-daemon` having an inability to assume IAM roles on k8s? More details in :thread:

I have independently deployed `dagster-daemon` and `dagster-user-code-deployment` pods. Both use the service account `dagster`

Both deployments have `serviceAccount.annotations` set to the same IAM role:
```    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::account_id:role/terraform/dagster```

Both deployments are running with the same user code package installed -- actually wait no, the daemon is not. The daemon must be using gRPC to trigger sensor runs on the user code deployment.

The pipelines deployed by the user code deployment are able to access S3 resources via the specified IAM role. But the dagster-daemon is continually logging an STS role assumption error.

_oh wait_ the user for the code deployment was different from `dagster`.

so.....even though the `dagster-daemon` was using the correct user.....it was invoking the sensor in the user code deployment remotely. and since the user code deployment had the wrong user. it couldn't assume the role appropriately.

OK. Makes sense. Ignore me :slightly_smiling_face:

Glad you figured it out! Is there a spot in the docs where you would have expected to see info for this

I think it would have to be <https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#separately-deploying-dagster-infrastructure-and-user-code|here>, if anywhere. Maybe as a final `note` subsection at the end of that section.

Yeah tricky, it’s a bit of overall system content mixed with k8s content mixed with aws content. Thanks for the suggestion!