Is there a way to pass Dagster Redshift connection to pandas read_sql() function? I saw that there was a function __get__conn() but I want to respect its privacy (naming convention). The reason, if it helps to show my purpose of the use is that before I had a connection setup that I was passing into

df = pd.read_sql(some_query, my_redshift_connection)

I would then get back a DataFrame with column names. Now using

my_data = context.resources.redshift.execute_query(some_query, fetch_results=True)

I have to do something like

df = pd.DataFrame(my_data, columns=['my_first_column', 'second_column', 'third'])

I am looking for a way not to declare column names and just get it off my query or DataFrame creation.

I'm not familiar with the dagster redshift connection, but it's entirely possible to create your own connection and define it as a resource. You potentially wouldn't even need to write your own code really, just dig into the dagster redshift resource, i wouldn't be surprised if there's something already there but I'm unsure if this is supported by pd.read_sql. TLDR: Yes, but i haven't done this exactly so i can't say if it's easy or may require a few extra lines of code.

I'm confused. Your response sounds like the exact setup that I have done and not what I'm wanting to do.

I feel like it's pretty reasonable for us to allow you to key into the connection itself, this is likely just an oversight. Mind filing an issue?

👍