I'm looking to leverage Datadog for logging. There...
# deployment-kubernetes
s
I'm looking to leverage Datadog for logging. There are ways for Datadog to auto-discover tags on pods launched by Dagster, by annotating the pods like
Copy code
<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>: '{"<TAG_KEY>": "<TAG_VALUE>"}'
I see that Dagster is already labeling pods with job names, etc., and that is essentially what I'm trying to get into Datadog. Am I right in thinking that if I wanted all the pods generated from a certain job to bear the
<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>: '{"dagster/job-name": "my_job_name"}'
annotation, then I would need to pass this in during job construction?
🤖 1
1
this area is all a bit new to me
a
You can have a look at 'dagster-k8s/config'. Search for it in the docs
👍 1
s
yeah, i guess what i'm wondering is whether there's a way to make those tag configurations dynamic at runtime (i.e. tag all pods with the job-run-id) or whether i'll need to pass them in for every job i create. also, do you happen to know whether you have to provide all the keys available in
dagster-k8s/config
? getting errors when i try to override via the UI adding
dagster-k8s/config: {"pod_template_spec_metadata": {"annotations": {"<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>": {"FOO": "BAR"}}}}
as a tag, yields:
Copy code
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 538, in _process_api_request
    api_result = self._handle_api_request(request, instance, user_code_launcher)
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 424, in _handle_api_request
    launcher.launch_run(LaunchRunContext(pipeline_run=run, workspace=None))
  File "/dagster-k8s/dagster_k8s/launcher.py", line 298, in launch_run
    self._launch_k8s_job_with_args(job_name, args, run, pipeline_origin)
  File "/dagster-k8s/dagster_k8s/launcher.py", line 233, in _launch_k8s_job_with_args
    user_defined_k8s_config = get_user_defined_k8s_config(frozentags(run.tags))
  File "/dagster-k8s/dagster_k8s/job.py", line 150, in get_user_defined_k8s_config
    user_defined_k8s_config_value = json.loads(tags[USER_DEFINED_K8S_CONFIG_KEY])
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
d
Tagging with the run ID specifically is a bit tricky since you typically don't know the run ID yet during the hooks that we provide to supply tags as the run doesn't exist yet. Specifying tags as config in e.g. a schedule or sensor launch is definitely possible though. That error looks like a JSON parse error - you don't need to specify all the keys, but it thinks the value of the tag isn't valid JSON. Could it be an escaping issue?
👍 1
s
yeah, i think so. in the launchpad ui, how should i pass in the key value for
dagster-k8s/config
? doesn't like it as a string `'`{"pod_template_spec_metadata": {"annotations": {"ad.datadoghq.com/tags": {"FOO": "BAR"}}}}'` or just as a dict
d
Here's a string that I pasted verbatim into the Tag editor right now and it was able to parse:
Copy code
{
  "container_config": {
    "resources": {
      "requests": {
        "cpu": "500m",
        "memory": "4Gi"
      }
    }
  }
}
And your string worked fine for me too, how strange:
Copy code
{
  "pod_template_spec_metadata": {
    "annotations": {
      "<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>": {
        "FOO": "BAR"
      }
    }
  }
}
Are you sure its not getting prepended with a single-quote or something?
You don't need to wrap it with ' on either side, maybe that's it?
😕 1
s
very weird. this is what's working for me:
dagster-k8s/config: {"pod_template_spec_metadata": {"annotations": {"<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>": "{\"dagster_job_name\": \"replicate_main_backend_orders\"}"}}}
. That's what I get when I load it programmatically using this:
Copy code
def make_dagster_datadog_annotations(**kwargs):
    return {
        "dagster-k8s/config": {
            "pod_template_spec_metadata": {
                "annotations": {
                    "<http://ad.datadoghq.com/tags|ad.datadoghq.com/tags>": json.dumps(**kwargs)
                }
            }
        }
    }
d
very confusing... we're talking about this UI in cloud right?
I'll double check that this isn't an agent version thing
Ah yep, I think I found it. https://github.com/dagster-io/dagster/pull/6205/files went out in 0.14.4 and had a bunch of fixes related to how we parse those tags
s
ahh, that makes sense, i think im on .14.3 or something
d
Ah, unlucky
s
how do i upgrade dagster cloud agent? does a
helm upgrade --install user-cloud dagster-cloud/dagster-cloud-agent
do that?
yeah, looks like that does it
🎉 1
d
^^ yeah, that's the upgrade command
123 Views