I'm looking to leverage Datadog for logging. There are ways for Datadog to auto-discover tags on pods launched by Dagster, by annotating the pods like

<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>: '{"<TAG_KEY>": "<TAG_VALUE>"}'

I see that Dagster is already labeling pods with job names, etc., and that is essentially what I'm trying to get into Datadog. Am I right in thinking that if I wanted all the pods generated from a certain job to bear the

<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>: '{"dagster/job-name": "my_job_name"}'

annotation, then I would need to pass this in during job construction?

You can have a look at 'dagster-k8s/config'. Search for it in the docs

yeah, i guess what i'm wondering is whether there's a way to make those tag configurations dynamic at runtime (i.e. tag all pods with the job-run-id) or whether i'll need to pass them in for every job i create. also, do you happen to know whether you have to provide all the keys available in

dagster-k8s/config

? getting errors when i try to override via the UI adding

dagster-k8s/config: {"pod_template_spec_metadata": {"annotations": {"<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>": {"FOO": "BAR"}}}}

as a tag, yields:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 538, in _process_api_request
    api_result = self._handle_api_request(request, instance, user_code_launcher)
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 424, in _handle_api_request
    launcher.launch_run(LaunchRunContext(pipeline_run=run, workspace=None))
  File "/dagster-k8s/dagster_k8s/launcher.py", line 298, in launch_run
    self._launch_k8s_job_with_args(job_name, args, run, pipeline_origin)
  File "/dagster-k8s/dagster_k8s/launcher.py", line 233, in _launch_k8s_job_with_args
    user_defined_k8s_config = get_user_defined_k8s_config(frozentags(run.tags))
  File "/dagster-k8s/dagster_k8s/job.py", line 150, in get_user_defined_k8s_config
    user_defined_k8s_config_value = json.loads(tags[USER_DEFINED_K8S_CONFIG_KEY])
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

Tagging with the run ID specifically is a bit tricky since you typically don't know the run ID yet during the hooks that we provide to supply tags as the run doesn't exist yet. Specifying tags as config in e.g. a schedule or sensor launch is definitely possible though. That error looks like a JSON parse error - you don't need to specify all the keys, but it thinks the value of the tag isn't valid JSON. Could it be an escaping issue?

yeah, i think so. in the launchpad ui, how should i pass in the key value for

dagster-k8s/config

? doesn't like it as a string `'`{"pod_template_spec_metadata": {"annotations": {"ad.datadoghq.com/tags": {"FOO": "BAR"}}}}'` or just as a dict

Here's a string that I pasted verbatim into the Tag editor right now and it was able to parse:

{
  "container_config": {
    "resources": {
      "requests": {
        "cpu": "500m",
        "memory": "4Gi"
      }
    }
  }
}

very weird. this is what's working for me:

dagster-k8s/config: {"pod_template_spec_metadata": {"annotations": {"<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>": "{\"dagster_job_name\": \"replicate_main_backend_orders\"}"}}}

. That's what I get when I load it programmatically using this:

def make_dagster_datadog_annotations(**kwargs):
    return {
        "dagster-k8s/config": {
            "pod_template_spec_metadata": {
                "annotations": {
                    "<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>": json.dumps(**kwargs)
                }
            }
        }
    }

very confusing... we're talking about this UI in cloud right?

ahh, that makes sense, i think im on .14.3 or something

Ah, unlucky

how do i upgrade dagster cloud agent? does a

helm upgrade --install user-cloud dagster-cloud/dagster-cloud-agent

do that?

^^ yeah, that's the upgrade command