I have one `@asset` that periodically loads data t...
# ask-community
c
I have one
@asset
that periodically loads data to BigQuery but I'd like to run it three times based on these inputs
["one", "two", "three"]
- what's the best way to do this without duplicating code?
🤖 1
s
c
I currently have an asset that fetches some data and uploads it to BigQuery using a
BigQueryDataframeIOManager
- it fetches daily data and also runs on a daily cron schedule. Using exactly the same code inside this
@asset
function I'd also like to run it on an hourly and minutely cron schedule, aggregating data at the respective resolution. So to summarise, in total I'm trying to update three BigQuery table: 1.
bigquery_minutely_table
updated every minute and aggregated on a minute timeframe 2.
bigquery_hourly_table
updated every hour and aggregated on an hourly timeframe 3.
bigquery_daily_table
updated every day and aggregated on a daily timeframe Is this possible?
I'm still wrapping my head around your docs but it looks like I could define my timeframes inside a
config.yaml
to pass in some run configurations.
@sean would really appreciate your guidance as to how the above is best achieved - thanks!
s
IIUC I think you want three separate partitioned assets, since you’re generating results at 3 different granularities, maybe something like this:
Copy code
@asset(partitions_def=HourlyPartitionsDefinition...)
def hourly_asset():
    shared_code()

@asset(partitions_def=DailyPartitionsDefinition...)
def daily_asset():
    shared_code()

@asset(partitions_def=TimeWindowPartitionsDefinition(..., cron_schedule="* * * * *", ...)
def minutely_asset():
    shared_code()
c
This is really helpful, I'll revert if I have any Q's