Kirill Yakunin
02/13/2023, 3:06 PM@asset(
op_tags={
'dagster-k8s/config': {
'container_config': {
'image': BASIC_EXECUTOR_IMAGE,
"resources": {
"requests": {"cpu": "50m", "memory": "2Gi"},
}
},
"pod_spec_config": {
"nodeSelector": {
"role": "..."
}
}
},
}
)
def fake_init_data(context):
<http://context.log.info|context.log.info>("Test log!")
return [
{"date": "2022-01-01", "value": 5},
{"date": "2022-01-01", "value": 5},
{"date": "2022-01-01", "value": 5},
{"date": "2022-01-01", "value": 5},
{"date": "2022-01-01", "value": 5},
{"date": "2022-01-01", "value": 5},
{"date": "2022-01-01", "value": 5},
]
When I try to run it (materialize the asset), it actually get ran using my code-deployment image, and not the one specified, and nodeSelector/resources also are ignored.
If I try
fake_init_data_job = define_asset_job(
"fake_init_data_job", AssetSelection.keys(['fake_init_data']),
tags={
'dagster-k8s/config': {
'container_config': {
'image': BASIC_EXECUTOR_IMAGE,
"resources": {
"requests": {"cpu": "50m", "memory": "2"Gi},
}
},
"pod_spec_config": {
"nodeSelector": {
"role": "...."
}
}
},
}
)
and then run the job, the job gets executed correctly (with correct image, on correct node).
Any ideas on how to make asset materialization take the op_tags into account?
Would appreciate any help 🙏Kirill Yakunin
02/13/2023, 5:13 PM@repository(default_executor_def=k8s_job_executor)
Then, all of assets and jobs will be executed by k8s_job_executor by default, and op_tags will be taken into account.