19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
I have a partitioned asset by day that calls an AP...
# ask-communityd
Danny Steffy
08/23/2023, 1:59 PMI have a partitioned asset by day that calls an API and uses the BigQuery IO manager. I have a schedule that runs every morning and runs the job for that partition. There isn't always data that is returned every day, and if I return an empty dataframe to the IO manager, it appears to error out. However, if I instead yield the Output and make the asset
output_required=Falsej
jamie
08/23/2023, 2:36 PMwhat error do you get when it returns an empty dataframe?
d
Danny Steffy
08/23/2023, 2:57 PM Copy code
dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "schedule_raw":
The above exception was caused by the following exception:
google.api_core.exceptions.BadRequest: 400 Table schedule_raw_c8d7df7e_36c9_4bd1_8ec2_69c7b6d316b7_source does not have a schema.Danny Steffy
08/23/2023, 3:25 PMmaybe that was a mistake though? Let me revert my changes back to returning an empty dataframe and confirm that's the error
Danny Steffy
08/23/2023, 3:27 PMyep, just confirmed that's the error that is returned
j
jamie
08/23/2023, 3:27 PMok - it’s been a minute since i’ve looked at some of the bigquery stuff, but i remember that kind of error showing up when an empty dataframe was the first thing materialized by an asset. the IO manager would try to make a table from the empty dataframe, and it would error because there was no column schema. i don’t recall it happening if you had a successful materialization and then later tried to materialize an empty dataframe
d
Danny Steffy
08/23/2023, 3:29 PMI can confirm that I have successful materializations before and the landing table exists in BigQuery, but I'm still getting this error. It looks like maybe it tries to create a temporary landing table, but if there's an empty dataframe there's no schema for it to create?
Danny Steffy
08/23/2023, 3:30 PMimage.png
j
jamie
08/23/2023, 3:30 PMok the temporary table wouldn’t be surprising. thinking on this…
14 Views