Has there been any work on using snowpark as an alternative to the snowflake python connector/ sqlalchemy in the IOManager backend? We’ve been using it internally and have seen some significant performance benefits. It’s also quite expressive when it comes to metadata operations. An example might be the flow control for what to do when the schema of an asset changes between code versions. Currently I believe the response is “panic”. It would be fairly easy in this framework to establish some lightweight checks like “do the schemas match. Yes? Cool. No? Check whether we’re supposed to throw an exception or replace the table” Doing so also opens up the possibility of using snowpark dataframes instead of pandas data frames. For operations which are computationally light, but extremely bandwidth intensive (add 1 to every row in this terabyte of data, that sort of thing), it would be great to have the ability to leverage dataframe abstractions on snowflake compute rather than needing to pull the data down locally.

hey @Matt Clarke we haven’t talked about this much on our end, but we’re working on some developer improvements that should make doing this kind of work easier (ie without having to write an entire I/O manager)