Tyler Jenkins
09/05/2023, 11:53 AMdaniel
09/05/2023, 7:47 PMTyler Jenkins
09/06/2023, 11:59 AMdaniel
09/06/2023, 12:03 PMdaniel
09/06/2023, 12:04 PMTyler Jenkins
09/06/2023, 12:06 PMTyler Jenkins
09/06/2023, 12:31 PM@asset(required_resource_keys={'requests_session'},
io_manager_key='pg_truncate',
freshness_policy=FreshnessPolicy(maximum_lag_minutes=24 * 60)
)
def generated_dictionary(context):
#Do some work here
return Output(value=generated_model, data_version=DataVersion(hashed_value_of_model))
And this successfully runs according to the asset's freshness policy.
In another code location we have:
@asset(
required_resource_keys={'minio'},
config_schema = {'bucket': Field(str, default_value='TeamBucket')},
freshness_policy = FreshnessPolicy(maximum_lag_minutes=60),
io_manager_key='pg'
auto_materialize_policy=AutoMaterializePolicy.eager()
)
def get_files(context: OpExecutionContext):
#Get files from s3 here
return Output(value=files_array, data_version=DataVersion(hashed_value_of_files))
This asset does not auto-materialize based on the freshness policy. However, I can still manually materialize the asset successfully.
Both assets are the top most asset in their respective lineages, and are running through dagster-webserver 1.4.10 (corresponding with both asset's dagster versions)