AWS Secrets tagged with `dagster` are automatically showing dagster #deployment-ecs

AWS Secrets tagged with `dagster` are automaticall...

Yepher

08/09/2023, 6:31 PM

AWS Secrets tagged with

dagster

are automatically showing up in my

run

instance because of

EcsRunLauncher

and that has been great. I've now been working towards "Structuring your Dagster project" and as part of that I've moved some code into

__init__.py

Copy code

secret = json.loads(os.getenv('dagsterSecret'))

defs = Definitions(
    assets=all_assets,
    resources={
        "bigquery": BigQueryResource(gcp_credentials=base64.b64encode(secret['bq_credentials'].encode("utf-8"))),
        "redshift": RedshiftClientResource(
            host=secret['redshift_host'],
            port=5439,
            user=secret['redshift_user'],
            password=secret['redshift_password'],
            database=secret['redshift_database'],
        )
    },

)

but when I put that in init.py I get an error on the user-code server. And the server fails to startup

Copy code

secret = json.loads(os.getenv('dagsterSecret'))
raise TypeError(f'the JSON object must be str, bytes or bytearray,

That makes since because EcsRunLauncher only sets it on the run container and not the daemon, web server, or user-code containers. My question is, how do folks tend to handle this? I can certainly handle an env variable not being set, but I would like to fail start-up if that env variable is not set. This example code seems like it would suffer the same problem if anything depended on a "secret" tagged

dagster

Yepher

08/09/2023, 6:35 PM

My work around is something like this:

Copy code

ecret_value = os.getenv('dagsterSecret', '')
secret = json.loads(secret_value) if secret_value else {}

bq_credentials = secret.get('bq_credentials', '')
redshift_host = secret.get('redshift_host', '')
redshift_user = secret.get('redshift_user', '')
redshift_password = secret.get('redshift_password', '')
redshift_database = secret.get('redshift_database', '')

defs = Definitions(
    assets=all_assets,
    resources={
        "slack": build_client(),
        "bigquery": BigQueryResource(gcp_credentials=base64.b64encode(bq_credentials.encode("utf-8"))),
        "redshift": RedshiftClientResource(
            host=redshift_host,
            port=5439,
            user=redshift_user,
            password=redshift_password,
            database=redshift_database,
        )
    },

But this feels like it is not the correct solution to the problem and is likely to mask a real issue later, like actually not setting a proper env var in the env

johann

08/09/2023, 7:53 PM

Tricky. We have an