https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Adam Ward

04/19/2023, 3:10 PM
Hello - I am using Dagster 1.2.3. I have a scheduled job which materializes an asset and places 3 key/value pairs (a dict and two integers) in the Metadata. The keys are all strings. While the asset is materialized, the Dagit UI throws the error below when trying to view the run.
Copy code
Operation name: 
RunRootQueryMessage: ('cannot represent an object', InMetadataValue(value=13)
Path: ["pipelineRunOrError", "runConfigYaml"]
Locations:[{"line":14,"column"3}]
The dict is picked up and successfully serialized to JSON downstream so I know it's making it through. I just wonder if Dagit has a problem displaying it in the viewer? Or maybe I'm way off... Either way, can anyone shed some light on where this error is originating? Thanks in advance!
Correction: Dagit displays all of the metadata correctly for the materialized asset. 🤷🏻‍♂️
d

Daniel Gafni

04/19/2023, 3:58 PM
maybe there is a
None
there?
a

Adam Ward

04/19/2023, 10:00 PM
No nulls. Can confirm non-null values display in the asset. The view fails, however, when trying to open the run.
Based on debugging the stack trace from the error below, it looks like PyYAML is having trouble parsing the MetaData objects passed to the RunConfig. I theorize that Dagster may need to create a representer for the Dagster-specific objects for the
dump
function to use:
Copy code
Operation name: RunRootQuery

Message: ('cannot represent an object', TextMetadataValue(text='12'))

Path: [ "pipelineRunOrError","runConfigYaml" ]

Locations: [ { "line": 14,"column": 3 } ]

Stack Trace:
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\graphql\execution\execute.py", line 521, in execute_field
    result = resolve_fn(source, info, **args)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\dagster_graphql\schema\pipelines\pipeline.py", line 488, in resolve_runConfigYaml
    return dump_run_config_yaml(self._pipeline_run.run_config)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\dagster\_utils\yaml_utils.py", line 150, in dump_run_config_yaml
    return yaml.dump(
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\__init__.py", line 253, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\__init__.py", line 241, in dump_all
    dumper.represent(data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 27, in represent
    node = self.represent_data(data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:<http://yaml.org|yaml.org>,2002:map', data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:<http://yaml.org|yaml.org>,2002:map', data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:<http://yaml.org|yaml.org>,2002:map', data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:<http://yaml.org|yaml.org>,2002:map', data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:<http://yaml.org|yaml.org>,2002:map', data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 58, in represent_data
    node = self.yaml_representers[None](self, data)
  File "C:\Users\<REDACTED>\Miniconda3\envs\<REDACTED>\lib\site-packages\yaml\representer.py", line 231, in represent_undefined
    raise RepresenterError("cannot represent an object", data)
I was able to figure out the issue here and thought I'd share in case anyone else ran into it. In short, I had an asset_sensor where I was grabbing the metadata from the asset materialization event log and using
serialize_value
function to pass it to the op_config for the job triggered by the asset_sensor. The values of the
EventLogEntry.asset_materialization.metadata
are contained in Dagster NamedTuple types such as
TextMetadataValue
and
JsonMetadataValue
. Serializing that metadata object with
serialize_value
resulted in those Dagster-specific types being passed to the downstream job and - because I was trying to deserialize them using
json.loads
- Dagit didn't know how to display those class instances in the job config, resulting in the error and stack trace above. I resolved the issue by replacing the use of
serialize_value
in the sensor to the snippet below:
Copy code
asset_event_metadata = {
        key: val.value
        for key, val in asset_event.asset_materialization.metadata.items()
    }
I hope this is helpful for others folks in the future. I know our use case of passing metadata from an asset to a downstream job is probably non-standard, so - with my upfront apologies for the direct callout - I'm going to tag @daniel, who has helped me in the past to review this for any unexpected behavior. Please let me know if I can provide any further context or details here.
4 Views