https://dagster.io/ logo
#integration-snowflake
Title
# integration-snowflake
c

clay

02/09/2023, 6:28 PM
Does
SnowflakePandasTypeHandler
know how to convert a python datetime object (
datetime.datetime.now()
) into a timestamp column in Snowflake when materializing a DataFrame?
j

jamie

02/09/2023, 6:33 PM
I don’t think i have a unit test for that specifically (there’s one for pandas.Timestamp). are you seeing unexpected behavior? some context - there’s a known issue with storing pandas timestamps in snowflake so the pandas snowflake io manager has some work-arounds written in. those might be affecting datetime as well
c

clay

02/09/2023, 6:33 PM
I ran into an error so i'm just `strftime`'ing it out for now and can cast it to a timestamp as needed
j

jamie

02/09/2023, 6:34 PM
ok. I’ll write up a unit test for it to see what’s up
if you have the error + stack trace that would be helpful too
c

clay

02/09/2023, 6:35 PM
it's probably related to the same issue as the pandas datetime. I did some googling and it looks like the non-dagster snowflake connector for python has issues with datetime objects, too. Unfortunately, I don't have the error or stack trace any longer
j

jamie

02/10/2023, 4:54 PM
I made a quick example that by-passes the workaround we have in the io manager. in it, the asset has datetime objects that get stored as
invalid date
and then when a downstream asset tries to load the data i get the error
Copy code
sqlalchemy.exc.InterfaceError: (snowflake.connector.errors.InterfaceError) 252005: Failed to convert current row, cause: year 47005047 is out of range
(Background on this error at: <https://sqlalche.me/e/14/rvf5>)
which is the same error i get with pandas timestamps. However, the workaround in the io manager is to turn date-like objects into strings and then store them in the db. And that is working as expected with
datetime
objects. if you happen to run into the issue again definitely let me know
👍 1
21 Views