https://dagster.io/ logo
#deployment-ecs
Title
# deployment-ecs
y

Yepher

08/09/2023, 6:31 PM
AWS Secrets tagged with
dagster
are automatically showing up in my
run
instance because of
EcsRunLauncher
and that has been great. I've now been working towards "Structuring your Dagster project" and as part of that I've moved some code into
__init__.py
Copy code
secret = json.loads(os.getenv('dagsterSecret'))

defs = Definitions(
    assets=all_assets,
    resources={
        "bigquery": BigQueryResource(gcp_credentials=base64.b64encode(secret['bq_credentials'].encode("utf-8"))),
        "redshift": RedshiftClientResource(
            host=secret['redshift_host'],
            port=5439,
            user=secret['redshift_user'],
            password=secret['redshift_password'],
            database=secret['redshift_database'],
        )
    },

)
but when I put that in init.py I get an error on the user-code server. And the server fails to startup
Copy code
secret = json.loads(os.getenv('dagsterSecret'))
raise TypeError(f'the JSON object must be str, bytes or bytearray,
That makes since because EcsRunLauncher only sets it on the run container and not the daemon, web server, or user-code containers. My question is, how do folks tend to handle this? I can certainly handle an env variable not being set, but I would like to fail start-up if that env variable is not set. This example code seems like it would suffer the same problem if anything depended on a "secret" tagged
dagster
My work around is something like this:
Copy code
ecret_value = os.getenv('dagsterSecret', '')
secret = json.loads(secret_value) if secret_value else {}

bq_credentials = secret.get('bq_credentials', '')
redshift_host = secret.get('redshift_host', '')
redshift_user = secret.get('redshift_user', '')
redshift_password = secret.get('redshift_password', '')
redshift_database = secret.get('redshift_database', '')

defs = Definitions(
    assets=all_assets,
    resources={
        "slack": build_client(),
        "bigquery": BigQueryResource(gcp_credentials=base64.b64encode(bq_credentials.encode("utf-8"))),
        "redshift": RedshiftClientResource(
            host=redshift_host,
            port=5439,
            user=redshift_user,
            password=redshift_password,
            database=redshift_database,
        )
    },
But this feels like it is not the correct solution to the problem and is likely to mask a real issue later, like actually not setting a proper env var in the env
j

johann

08/09/2023, 7:53 PM
Tricky. We have an
EnvVar
class that only gets resolved at runtime, but you can’t call encode etc. on it. Would you mind filing an issue for this?
👍 1
y

Yepher

08/09/2023, 8:15 PM
@johann So to clarify. If I were to use
EnvVar
that would defer loading "stuff" until it is on the "run" instance and not on the "usercode" instance?
j

johann

08/09/2023, 8:22 PM
Correct
4 Views