https://dagster.io/ logo
#integration-bigquery
Title
# integration-bigquery
w

won

02/25/2023, 3:26 PM
why we need to uppercase columns? find error when dataframe like this
pd.DataFrame(['510365552', '564032331', '505153022', '547927024', '556154751', '569685159'])
Copy code
TypeError: descriptor 'upper' for 'str' objects doesn't apply to a 'int' object
...

  File "....virtualenvs\dagster\lib\site-packages\dagster_gcp_pandas\bigquery\bigquery_pandas_type_handler.py", line 39, in handle_output
    with_uppercase_cols = obj.rename(str.upper, copy=False, axis="columns")
j

jamie

02/27/2023, 4:38 PM
the columns are uppercase so that if you materialize the asset with any of the database io managers (particularly snowflake) the same table will be created. iirc there’s an issue with snowflake tables having lowercase column names that can sometimes cause querying issues so we convert everything to uppercase for consistency
q

Qwame

05/05/2023, 9:02 AM
@jamie Is there a way to set a flag or key in the Metadata for lower case columns? I really want lower case columns and I don't think that's going to be an issue for BigQuery
j

jamie

05/05/2023, 1:20 PM
there isn’t but i think it should be reasonable to add. do you mind opening a github issue?
👍 1
q

Qwame

05/05/2023, 6:01 PM
@jamie issue here
ty spinny 1
5 Views