https://dagster.io/ logo
Title
s

Sean Han

03/03/2023, 10:46 PM
Hi team, just want to confirm that if I use
op_tags
like this and select to materialize the asset, the asset job will use this config, right? I’m currently not seeing this, not sure what is wrong
@multi_asset(
     ...
    op_tags={
        "dagster-k8s/config": {
            "container_config": {
                "resources": {
                    "requests": {"cpu": "5000m", "memory": "64Gi"}
                },
            },
        }
    }
)
def test_asset(context: OpExecutionContext) -> pd.DataFrame:
    pass
Not seeing it
Usually I see it with it set on jobs
d

daniel

03/03/2023, 10:58 PM
Hi Sean, this tag would need to go on the job that's being launched, yeah. Putting it on an individual asset (even if its multi-asset) isn't going to cause it to the affect the job that's being run
s

Sean Han

03/03/2023, 10:59 PM
I usually select the asset and click materialize, is there a way to let Dagster know that it needs the tag? Or I have to convert this multi asset to a job in code in order to apply the tag?
d

daniel

03/03/2023, 11:00 PM
I think for this you would need to call define_asset_job, yeah
s

Sean Han

03/03/2023, 11:03 PM
In this doc it says if I use
k8s_job_executor
, I can also use the
dagster-k8s/config
tag on a Dagster op to control the Kubernetes configuration for that specific op. Does this apply to asset? https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#instance-level-kubernetes-configuration
d

daniel

03/03/2023, 11:04 PM
yeah, in that case, applying op_tags would be applied to that op
to the k8s pod for that op, that is
s

Sean Han

03/03/2023, 11:06 PM
So my question is, I setup my repo as
*@repository*(default_executor_def=k8s_job_executor)
, is what I described above mean that tag is not applied? or applied and I can’t see it
d

daniel

03/03/2023, 11:06 PM
I'm not sure that's what you want for materializing a single asset though - in that case, it would spin up a k8s pod for the run (that doesn't do very much) and then a k8s pod just for running the single asset
s

Sean Han

03/03/2023, 11:06 PM
Yes, I want a pod with more memory to materialize that asset since that asset is big
d

daniel

03/03/2023, 11:08 PM
Yeah, I think this is exposing a thing that should be easier in the current system.... some kind of way to apply tags to jobs that materialize just this one asset could help potentially?
or if you're OK with spinning up two pods to materialize a single asset, then the k8s_job_executor solution could work too
s

Sean Han

03/03/2023, 11:11 PM
Could you explain more about the second solution? I don’t fully understand. Part of the reason I don’t want to use an asset job is that I want to be able to select a range of partition and materialize those, which I think job is not capable of.
d

daniel

03/03/2023, 11:12 PM
The k8s_job_executor solution you mean?
s

Sean Han

03/03/2023, 11:13 PM
Yes, is there anything I need to do differently for the tag to apply?
d

daniel

03/03/2023, 11:14 PM
No, I think op_tags would work with the k8s_job_executor. You wouldn't see it in the tags for the run because that doesn't include op tags, but i would expect the pods for each individual step/asset to have the resource limits applied
s

Sean Han

03/03/2023, 11:14 PM
Thanks! Good to know that it is applied