Hey y'all. Sorry I'm searching for this and not se...
# ask-community
j
Hey y'all. Sorry I'm searching for this and not seeming to find a straightforward answer from the previous discussions on here or the docs. I'm using helm to deploy dagster in kubernetes. Everything works great, but I want the ops in my jobs to use environment variables to get some other configuration at runtime. There is an example in the docs that shows how to read environment variables when setting up a job in a repository, but the syntax is confusing. Where is "database_client" coming from (undefined). Where is get_env() defined? I don't actually need to return different configuration (local vs production) in my repository's job configuration, I just need a few values that will always be read from environment variables. https://docs.dagster.io/concepts/configuration/configured#specifying-per-environment-configuration
j
In this example,
database_client
is a resource that is being provided for the
database
resource key.
get_env
is a function that gets the environment name (
local
or
production
from the environment - under the hood it's just an
os.getenv
call) If you want to read values from the environment as config for you ops you'll want to use the
StringSource
or
IntSource
config types. These allow you to specify the name of an environment variable you want read to provide the actual value for the configuration. So you can do something like this
Copy code
@op(
    config_schema={"foo": StringSource}
)
def my_op_reads_from_env(context):
    <http://context.log.info|context.log.info>(context.op_config["foo"])

op_configured = configured(my_op_reads_from_env, name="op_configured")({"foo": {"env": "FOO"}})

@job(
)
def my_job():
    op_configured()
j
Thanks. Since I have a bunch of ops that all use the same config, is there a way to pass that config you have in the op_configured definition from within the job, or the repository instead?
I figured it out. Ended up doing
default_config = {
"resources": {
"values": {
"config": {
"postgresql_host": {"env": "POSTGRESQL_HOST"},
"postgresql_app_database": {"env": "POSTGRESQL_APP_DATABASE"},
"postgresql_app_username": {"env": "POSTGRESQL_APP_USERNAME"},
"postgresql_app_password": {"env": "POSTGRESQL_APP_PASSWORD"},
"asset_files_dir": {"env": "ASSET_FILES_DIR"}
}
}
}
}
@job(
config=default_config,
resource_defs={
"values": make_values_resource(
postgresql_host=StringSource,
postgresql_app_database=StringSource,
postgresql_app_username=StringSource,
postgresql_app_password=StringSource,
asset_files_dir=StringSource),
"dbt": my_dbt_resource
}
)
def load_source_data_job():
Then just defined my ops as
@op(required_resource_keys={"values"})
Thanks for your help!
j
that looks great! glad it worked
blob cheer 1