Daniel Kim
05/05/2023, 11:52 PMdefs = Definitions(
assets=[
asset1,
asset2,
],
resources={
"io_manager": snowflake_pandas_io_manager.configured(
{
"account": {"env": "SNOWFLAKE_ACCOUNT"},
"warehouse": {"env": "SNOWFLAKE_WAREHOUSE"},
"database": {"env": "SNOWFLAKE_DATABASE"},
"schema": {"env": "SNOWFLAKE_SCHEMA"},
"role": {"env": "SF_ROLE"},
"user": {"env": "SNOWFLAKE_USER"},
"password": {"env": "SNOWFLAKE_PASSWORD"},
"authenticator": {"env": "SNOWFLAKE_AUTHENTICATOR"},
}
),
},
)
Now, I want to add a job that consists of just 1 job, pretty much I want to execute a Snowflake query like in this example in the docs :
@op(required_resource_keys={'snowflake'})
def get_one(context):
context.resources.snowflake.execute_query('SELECT 1')
@job(resource_defs={'snowflake': snowflake_resource})
def my_snowflake_job():
get_one()
my_snowflake_job.execute_in_process(
run_config={
'resources': {
'snowflake': {
'config': {
'account': {'env': 'SNOWFLAKE_ACCOUNT'},
'user': {'env': 'SNOWFLAKE_USER'},
'password': {'env': 'SNOWFLAKE_PASSWORD'},
'database': {'env': 'SNOWFLAKE_DATABASE'},
'schema': {'env': 'SNOWFLAKE_SCHEMA'},
'warehouse': {'env': 'SNOWFLAKE_WAREHOUSE'},
}
}
}
}
)
But not sure exactly, how can I add this job to my definition above. What I'm stuck on is how do I pass in the same Snowflake credentials/configs? I know in the definition, I can add the jobs=[my_snowflake_job], but not sure how to pass in the resource config for it also. So basically, I want my definition to incorporate both a job of assets and job of ops, sharing the same Snowflake credentials/configs.