https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jacob Marcil

03/15/2023, 3:32 PM
Hello everyone, I am querying an API that returned a JSON payload. What I would like is to convert that JSON to a dict and then use _`dagster_type` in my asset configuration to validate that the payload has the expected format._ _The problem is that when I save the asset with fs_io_manager, it save the dict to file._ _Is there a way I could keep the JSON format so it save the content to disk properly while still validating the payload with
dagster_type
?_
dagster bot responded by community 1
a

Andras Somi

03/15/2023, 4:21 PM
You could use a Pydantic model for parsing and validating the JSON into a Python object and decorate that class with @usable_as_dagster_type to be able to use it as a Dagster type in assets.
j

Jacob Marcil

03/15/2023, 4:26 PM
That’s what I’m doing, but if at the moment of storing the results to file. If I don’t convert the dict to JSON I Will store a python dict object instead of the JSON payload. This will fail the validation check since json is a string
Copy code
@asset(
dagster_type=MyPydanticType
)
def my_asset():
     json = get_json_payload_from_api()
     return json
This will store the object as a dict on file
Copy code
@asset(
dagster_type=MyPydanticType
)
def my_asset():
     json = get_json_payload_from_api()
     obj = MyPydanticType.parse_obj(json)
     return obj
a

Andras Somi

03/15/2023, 6:02 PM
It won't store it as a dict, I think it will be pickled if you use the default io manager. But if it's important to store json and also have the proper dagster type you can define your own io manager and save the json representation there (and provide logic how to load that for downstream assets).
Or it isn't used downstream (so the only purpose is saving the json) you don't have to worry about the dagster type at all (Pydantic does the validation anyway), you should return Nothing.
3 Views