🌼 New for the BigQuery I/O manager in 0.18.2 🌼 This release highlight comes to you a few days late, but here’s what’s new in last week’s release. The BigQuery I/O manager supports self-dependent assets! If you have a partitioned asset that depends on a prior partition of itself, the BigQuery I/O manager will load that partition as a DataFrame. For the first partition in the series, an empty DataFrame will be returned

@asset(
    partitions_def=DailyPartitionsDefinition(start_date="2023-01-01"),
    ins={
        "self_dependent_asset": AssetIn(
            key=AssetKey(["self_dependent_asset"]),
            partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1),
        ),
    },
    metadata={
        "partition_expr": "date",
    },
)
def self_dependent_asset(context, self_dependent_asset: pd.DataFrame) -> pd.DataFrame:
    date = pd.Timestamp(context.asset_partition_key_for_output())

    if self_dependent_asset.empty:
        return pd.DataFrame({"date": [date], "value": [1]})
    return pd.DataFrame({"date": [date], "value": [self_dependent_asset["value"][0] + 1]})

The BigQuery I/O manager now accepts

timeout

configuration. Currently, this configuration will only be used when working with Pandas DataFrames. It will set the number of seconds to wait for a request before using a retry.