https://dagster.io/ logo
#ask-community
Title
# ask-community
b

Bolin Zhu

08/30/2023, 6:50 AM
Hi team! Is there a way for us to modify asset level Kubernetes configuration? Let’s say I have an asset that is auto-materialising eagerly. It requires an abnormally high amount of memory to operate on a dataset. Is there a way to apply custom Kubernetes configuration to request for higher memory whenever it auto-materialises? So far from the doc, it appears that I need to materialise this asset with a job and use
k8s_job_executor
to run each step in its own pod with custom Kubernetes configuration.
D 1
cc: @Jonathan Phoon @Rudy Jariwala
👀 1
s

sandy

08/30/2023, 5:57 PM
Hi Bolin - if you're using the
k8s_job_executor
you can provide
op_tags
when you're defining your assets
b

Bolin Zhu

08/31/2023, 6:02 AM
Thanks Sandy! After providing
ops_tags
when defining the asset, the asset must be run by a job in order to those custom configurations to be applied right? If the asset were to auto-materialise, the supplied
op_tags
is doing nothing.
d

daniel

08/31/2023, 12:57 PM
For this to work you would need to specify the 'executor' field on your Definitions object in order to make it the default executor used by the jobs that are auto-materialized, yeah. Which would also have the effect of making each of your assets that are auto-materialized use that executor - we don't yet have support for mixing and matching the executors that are used by auto-materialized assets (i.e. run most assets in a subprocess, but just one in a k8s job) - but it's on our radar as something we'd like to support in the future
b

Bolin Zhu

08/31/2023, 1:27 PM
Thanks a lot Daniel! This is very helpful!
condagster 1