I noticed that the dagster-gcp-pandas library uses upper case column names. is there a specific reason for this? if not I'll do a PR, so that the library uses the column names as defined in the DataFrame.

Hi! Let me check with someone from the team and get back to you on this! Thanks for offering to raise the PR.

This is largely done for consistency with other io managers that store data in databases. Some of those databases require upper case column names in order for queries to work correctly, so we uppercase all column names across the board to preserve the ability to swap io manager implementations

I came accross the issue with snowflake and did a internal patch to have case-sensitive column names working correctly, I think users should have the option to choose the behaviour they prefer here : In many cases when we introduce Dagster in existing pipelines we don't want to have to fix/change columns because io_managers normalise them to uppercase - so if the backend IO manager supports it, it is good to have.

That was also my intention, existing datasets are lower case. I ended up with my own

BigQueryPandasIOManager

variant, which also allows me to control the table schema.

thanks for the input! I’ll keep this in mind for future work on these io managers