Darryl Ng
02/22/2024, 10:49 AMScout
02/22/2024, 10:49 AMobservable_source_asset
, the purpose of the observation function is to return a DataVersion
that represents the current state of the source data. If the data has not changed, the observation function should return the same DataVersion
as it did previously. This would indicate to Dagster that the data version has not changed, and therefore the source asset is unchanged. The documentation does not provide a direct example of returning the previous DataVersion
in an observable_source_asset
. However, it does describe how to define an observation function that computes and returns a DataVersion
based on the content of a file, for example, by hashing the file contents. If the file content has not changed, the hash (and thus the DataVersion
) would remain the same. Here is a conceptual approach based on the documentation's description of how observable_source_asset
works: 1. Define an observable_source_asset
with an observation function. 2. In the observation function, compute the DataVersion
based on the current state of the source data. 3. If the source data has not changed since the last observation, the computed DataVersion
should be identical to the previous one. 4. Return the computed DataVersion
from the observation function. If you have access to the previous DataVersion
(e.g., stored externally or retrievable through some mechanism), you could compare the newly computed DataVersion
with the previous one. If they match, you return the previous DataVersion
, indicating that the data has not changed. However, the documentation does not provide explicit code examples or mechanisms for retrieving and returning the previous DataVersion
. If you need to implement such functionality, you may need to devise a custom solution that fits your specific use case and data source.