Qwame
05/09/2023, 7:28 PMhandle_output
section of the pandas type handler file that skips the creation of tables if the data frame has 0 rows or some sort of a check to ensure that 0 rows data frames are not passed as output to the IO managerif obj.shape[0] > 0: # or table_slice.job_config.schema:
job = connection.load_table_from_dataframe(
dataframe=with_lowercase_cols,
destination=f"{table_slice.schema}.{table_slice.table}",
project=table_slice.database,
location=context.resource_config.get("location")
if context.resource_config
else None,
timeout=context.resource_config.get("timeout")
if context.resource_config
else None,
# job_config=table_slice.job_config if table_slice.job_config else None,
)
job.result()
elif context.has_partition_key:
context.log.warn(f"The partition has {obj.shape[0]} rows")
else:
check.failed(
f"The object has {obj.shape[0]} rows. Please provide a `job_config` since auto-detection of data types can be wrong without data"
)
sean
05/09/2023, 10:03 PMQwame
05/10/2023, 2:15 PM