Charles
03/30/2023, 3:32 PM@asset
that periodically loads data to BigQuery but I'd like to run it three times based on these inputs ["one", "two", "three"]
- what's the best way to do this without duplicating code?sean
03/30/2023, 4:14 PMCharles
03/30/2023, 4:39 PMBigQueryDataframeIOManager
- it fetches daily data and also runs on a daily cron schedule.
Using exactly the same code inside this @asset
function I'd also like to run it on an hourly and minutely cron schedule, aggregating data at the respective resolution.
So to summarise, in total I'm trying to update three BigQuery table:
1. bigquery_minutely_table
updated every minute and aggregated on a minute timeframe
2. bigquery_hourly_table
updated every hour and aggregated on an hourly timeframe
3. bigquery_daily_table
updated every day and aggregated on a daily timeframe
Is this possible?config.yaml
to pass in some run configurations. sean
03/30/2023, 7:54 PM@asset(partitions_def=HourlyPartitionsDefinition...)
def hourly_asset():
shared_code()
@asset(partitions_def=DailyPartitionsDefinition...)
def daily_asset():
shared_code()
@asset(partitions_def=TimeWindowPartitionsDefinition(..., cron_schedule="* * * * *", ...)
def minutely_asset():
shared_code()
Charles
03/31/2023, 9:29 AM