Jayme Edwards
09/01/2022, 2:28 PMjamie
09/01/2022, 3:25 PMdatabase_client
is a resource that is being provided for the database
resource key. get_env
is a function that gets the environment name (local
or production
from the environment - under the hood it's just an os.getenv
call)
If you want to read values from the environment as config for you ops you'll want to use the StringSource
or IntSource
config types. These allow you to specify the name of an environment variable you want read to provide the actual value for the configuration. So you can do something like this
@op(
config_schema={"foo": StringSource}
)
def my_op_reads_from_env(context):
<http://context.log.info|context.log.info>(context.op_config["foo"])
op_configured = configured(my_op_reads_from_env, name="op_configured")({"foo": {"env": "FOO"}})
@job(
)
def my_job():
op_configured()
Jayme Edwards
09/01/2022, 3:45 PMdefault_config = {
"resources": {
"values": {
"config": {
"postgresql_host": {"env": "POSTGRESQL_HOST"},
"postgresql_app_database": {"env": "POSTGRESQL_APP_DATABASE"},
"postgresql_app_username": {"env": "POSTGRESQL_APP_USERNAME"},
"postgresql_app_password": {"env": "POSTGRESQL_APP_PASSWORD"},
"asset_files_dir": {"env": "ASSET_FILES_DIR"}
}
}
}
}
@job(
config=default_config,
resource_defs={
"values": make_values_resource(
postgresql_host=StringSource,
postgresql_app_database=StringSource,
postgresql_app_username=StringSource,
postgresql_app_password=StringSource,
asset_files_dir=StringSource),
"dbt": my_dbt_resource
}
)
def load_source_data_job():
@op(required_resource_keys={"values"})
jamie
09/01/2022, 4:06 PM