https://dagster.io/ logo
Title
s

saravan kumar

11/04/2022, 6:24 AM
In the resource when i use os.getenv("") , the db connection works well..I tried to move the details to run_config as per document..but they just get the string value instead of reading from the env
@job(resource_defs={"db_session": db_session})
def job_sample_db_test():
    run_sample_db_test()


job_sample_db_test.execute_in_process(
    run_config={
        "resources": {
            "db_session": {
                "config": {
                    "DB_USER": {"env": "DB_USER"},
                    "DB_PASSWORD": {"env": "DB_PASSWORD"},
                }
            }
        }
    }
)
@resource
@contextmanager
def met_db_session(context):
    try:
        user = context.resource_config["DB_USER"]
        password = context.resource_config["DB_PASSWORD"]
        db = os.getenv("DB_DB", "")
        host = context.resource_config["DB_HOST"]
        port = int(os.getenv("DB_PORT", 54332))
        db_connection = create_engine(f'postgresql://{user}:{password}@{host}:{port}/{db}')
        session_maker = sessionmaker(bind=db_connection)
        session = session_maker()
        yield session
    finally:
        db_connection.close()
Whatever is using os.getenv gets the proper values ,but context.resource_config just get the {"env": "DB_USER"} without actually reading the DB_USER from the environment..what i am missing? The secrets are kuberenets secrets and are available by os.getenv I am just looking for a resource which gets me sql alchemy db connection ,so if there is a better way ,i am down for it...
:dagster-bot-resolve: 1
z

Zach

11/04/2022, 3:03 PM
what does your resource config schema look like? are you using a StringSource for those config keys?
s

saravan kumar

11/04/2022, 3:12 PM
ob_sample_db_test.execute_in_process(
    run_config={
        "resources": {
            "db_session": {
                "config": {
                    "DB_USER": {"env": "DB_USER"},
                    "DB_PASSWORD": {"env": "DB_PASSWORD"},
                }
            }
        }
    }
)
This is what i have...I guess i am missing something? so intention is to supply the db_user from the env of the running container..
z

Zach

11/04/2022, 3:14 PM
oh I meant the actual config schema for the
db_session
resource. so what does the
db_session
resource definition look like? should be something like
@resource(config_schema={...})
def db_session(...)
    ...
s

saravan kumar

11/04/2022, 3:18 PM
so i think to my resource i have to pass the config schema which i am missing
@resource
@contextmanager
def met_db_session(context):
    try:
        user = context.resource_config["DB_USER"]
        password = context.resource_config["DB_PASSWORD"]
        db = os.getenv("DB_DB", "")
        host = context.resource_config["DB_HOST"]
        port = int(os.getenv("DB_PORT", 54332))
        db_connection = create_engine(f'postgresql://{user}:{password}@{host}:{port}/{db}')
        session_maker = sessionmaker(bind=db_connection)
        session = session_maker()
        yield session
    finally:
        db_connection.close()
I hoped i can just read the resource_config like this from the context
z

Zach

11/04/2022, 3:31 PM
hmm I would maybe try something like
from dagster import StringSource
@resource(config_schema={"DB_USER": StringSource, "DB_PASSWORD": StringSource})
@contextmanager
def met_db_session(context):
    ...
s

saravan kumar

11/04/2022, 3:31 PM
Sure..i will try that...to access it , it wud be same rite?context.resource_config["DB_USER"]
z

Zach

11/04/2022, 3:31 PM
correct
more generally, it might be helpful to review these docs on configuration management if you haven't already
s

saravan kumar

11/04/2022, 6:08 PM
I tried it , but when i run the job , it asks me for the config? I was hoping , it will read the env variable from container environment..
i guess the .execute_in_process() works and it gets the env vars, but when using launchpad it doesnt get the variables
s

sean

11/07/2022, 4:12 PM
Saravan, did you replace your
getenv
calls with
StringSource
as Zach suggested?
s

saravan kumar

11/07/2022, 4:12 PM
Yup..It works when i do it execute_in_process()
i went back to getEnv to solve it for now.. With the configuration , I have to manaully give it in the dagit when i run the job which is not desired.. I was hoping I could pass a default config
s

sean

11/07/2022, 4:20 PM
did you try setting the default config to a selector with the name of the env var you’re setting? I realize our docs here could definitely be more clear:
@op(
  config_schema={
        "foo": Field(StringSource, { "env": "FOO" })
    }
)
def foo_op(context) -> str:
    return context.op_config["foo"]
s

saravan kumar

11/07/2022, 4:22 PM
I can try this and will let you know...
on another note , is there any disadvantages of using env.get() than the config? Does the dag UI expose the env.get at some point? They are the db credentails and i want to use them at the most secure way..
s

sean

11/07/2022, 4:29 PM
do you mean the way you used
os.getenv()
in your initial code block inside your resource and stored the result in a local variable? No that won’t be exposed.
s

saravan kumar

11/07/2022, 5:04 PM
ok..I think i could go back to os.getEnv then..I will try the default config as well