https://dagster.io/ logo
Powered by
o

Ohad Basan

03/01/2021, 10:23 PM
Hello there! I’m trying to use 
dagster-k8s/config
tag to apply tags to the k8s job that’s created by dagster I’m doing it like this:
Copy code
@solid(
    tags={
      'dagster-k8s/config':{
        'pod_template_spec_metadata': {
          'annotations': { "<http://iam.amazonaws.com/role|iam.amazonaws.com/role>": "Dist"}
        },
      }
    },
    required_resource_keys={"data_lake", "dataset_name"},
    config_schema=String,
)
the created job is being created without the above tags. I downloaded the run’s debug log from the dagster UI and i see that the tags do appear in the debug log. just not on the k8s job. am i doing something wrong? thank you!
a

alex

03/02/2021, 12:12 AM
just to confirm, you are you using 
celery-k8s
?
o

Ohad Basan

03/02/2021, 9:29 AM
I am using 
K8sRunLauncher
. is that a problem?
not using celery
a

alex

03/02/2021, 3:18 PM
Ah so in that case the only job pod is for the entire pipeline run so you should set the resource tags on the @pipeline
o

Ohad Basan

03/02/2021, 3:29 PM
I tried that too but it didn’t seem to work. will try again now.
thank you!
I still dont see the annotation in the pod.
Copy code
@pipeline(
    tags={
      'dagster-k8s/config': {
        'pod_template_spec_metadata': {
          'annotations': { "<http://iam.amazonaws.com/role|iam.amazonaws.com/role>": "Dist"}
        },
      }
    },
    mode_defs=[
      ModeDefinition(
          "local",
          resource_defs={
            "s3": s3_mock,
            # "snowflake": snowflake_resource,
            "data_lake": mock_data_lake,
            "dataset_name": dataset_name,
            "geocoder": mock_geocoder,
          },
      ),
      ModeDefinition(
          "prod",
          resource_defs={
            "s3": s3_resource,
            # "snowflake": snowflake_resource,
            "data_lake": s3_data_lake,
            "dataset_name": dataset_name,
            "geocoder": google_geocoder,
          },
      ),
    ],
)
did i make a mistake here?
a

alex

03/02/2021, 3:57 PM
hmm no this looks right to me - were you testing by launching in dagit? did you see the tags in the playground?
and just to verify what version are you on?
o

Ohad Basan

03/02/2021, 4:00 PM
I am launching with pipeline with dagit. if i am looking at the correct place, then no, I dont see the tags
I am on 0.10.4
if this is the place where the tags should be - then no 😞
a

alex

03/02/2021, 4:02 PM
should be right underneath the preset button - double check that your code change to add tags to @pipeline got deployed successfully and maybe hit “reload metadata” button in dagit
theres an example of a pipeline tag on the right and the reload button i am talking about on the left
o

Ohad Basan

03/02/2021, 4:06 PM
Hi. tried to reload the metadata. still no tags. connected to the userDeployment container to make sure that the code is there and it is… weird
btw I see the tags in the debug logs so dagster is aware of them
a

alex

03/02/2021, 4:09 PM
hmm very strange - not sure whats going awry here
can you show me the debug log entry
o

Ohad Basan

03/02/2021, 4:09 PM
sure
Copy code
"solid_name": "get_addresses", "tags": {"dagster-k8s/config": "{\"pod_template_spec_metadata\": {\"annotations\": {\"<http://iam.amazonaws.com/role\|iam.amazonaws.com/role\>": \"Dist\"}}}"}}, {"__class__": "SolidInvocationSnap", "input_dep_snaps": [{"__class__": "InputDependencySnap", "input_name": "addresses", "upstream_output_snaps": [{"__class__": "OutputHandleSnap", "output_name": "result", "solid_name": "geocode_addresses"}]}
here is a snippet
a

alex

03/02/2021, 4:11 PM
I believe thats the solid level tags not the pipeline level ones
OK heres a thing (which is fixed in newer versions of dagit)
make sure to open a new tab in the playgound
hmm there might be an issue here where we dont load the tags in to dagit unless you load a 
preset
or 
partition
o

Ohad Basan

03/02/2021, 4:15 PM
after opening a new tab I dont see the tags in the ui but I DO see them now at the “runs” table in dagit.
a

alex

03/02/2021, 4:15 PM
ah ok so they are on the run - but still not on the k8s pods
o

Ohad Basan

03/02/2021, 4:15 PM
exactly.
previously before opening the new tab they weren’t on the run too
a

alex

03/02/2021, 4:16 PM
did double check the job & pod via kubectl on the runs after the new tab?
o

Ohad Basan

03/02/2021, 4:18 PM
now it is working.
I missed the latest run
opening a new tab resolved it
🔥 thank you!
a

alex

03/02/2021, 4:18 PM
ya there is a “want to refresh your stale config” thing in dagit as of 0.10.X (5,6,7 I forget which)
o

Ohad Basan

03/02/2021, 4:19 PM
I’ll upgrade my version then
a

alex

03/02/2021, 4:19 PM
👍
💥 2
2 Views
Powered by