https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Danny Steffy

08/23/2023, 1:59 PM
I have a partitioned asset by day that calls an API and uses the BigQuery IO manager. I have a schedule that runs every morning and runs the job for that partition. There isn't always data that is returned every day, and if I return an empty dataframe to the IO manager, it appears to error out. However, if I instead yield the Output and make the asset
output_required=False
, it says it never materialized the asset for the partition. What's the correct way to handle this case?
j

jamie

08/23/2023, 2:36 PM
what error do you get when it returns an empty dataframe?
d

Danny Steffy

08/23/2023, 2:57 PM
Copy code
dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "schedule_raw":
The above exception was caused by the following exception:
google.api_core.exceptions.BadRequest: 400 Table schedule_raw_c8d7df7e_36c9_4bd1_8ec2_69c7b6d316b7_source does not have a schema.
maybe that was a mistake though? Let me revert my changes back to returning an empty dataframe and confirm that's the error
yep, just confirmed that's the error that is returned
j

jamie

08/23/2023, 3:27 PM
ok - it’s been a minute since i’ve looked at some of the bigquery stuff, but i remember that kind of error showing up when an empty dataframe was the first thing materialized by an asset. the IO manager would try to make a table from the empty dataframe, and it would error because there was no column schema. i don’t recall it happening if you had a successful materialization and then later tried to materialize an empty dataframe
d

Danny Steffy

08/23/2023, 3:29 PM
I can confirm that I have successful materializations before and the landing table exists in BigQuery, but I'm still getting this error. It looks like maybe it tries to create a temporary landing table, but if there's an empty dataframe there's no schema for it to create?
image.png
j

jamie

08/23/2023, 3:30 PM
ok the temporary table wouldn’t be surprising. thinking on this…
6 Views