https://dagster.io/ logo
#ask-ai
Title
# ask-ai
d

Darryl Ng

02/22/2024, 10:49 AM
How can I return the previous DataVersion in observable_source_asset so that it is marked as unchanged
s

Scout

02/22/2024, 10:49 AM
In the context of
observable_source_asset
, the purpose of the observation function is to return a
DataVersion
that represents the current state of the source data. If the data has not changed, the observation function should return the same
DataVersion
as it did previously. This would indicate to Dagster that the data version has not changed, and therefore the source asset is unchanged. The documentation does not provide a direct example of returning the previous
DataVersion
in an
observable_source_asset
. However, it does describe how to define an observation function that computes and returns a
DataVersion
based on the content of a file, for example, by hashing the file contents. If the file content has not changed, the hash (and thus the
DataVersion
) would remain the same. Here is a conceptual approach based on the documentation's description of how
observable_source_asset
works: 1. Define an
observable_source_asset
with an observation function. 2. In the observation function, compute the
DataVersion
based on the current state of the source data. 3. If the source data has not changed since the last observation, the computed
DataVersion
should be identical to the previous one. 4. Return the computed
DataVersion
from the observation function. If you have access to the previous
DataVersion
(e.g., stored externally or retrievable through some mechanism), you could compare the newly computed
DataVersion
with the previous one. If they match, you return the previous
DataVersion
, indicating that the data has not changed. However, the documentation does not provide explicit code examples or mechanisms for retrieving and returning the previous
DataVersion
. If you need to implement such functionality, you may need to devise a custom solution that fits your specific use case and data source.
2 Views