I am trying to use `snowflake io manager` with partitions Bu

Question

I am trying to use `snowflake io manager` with partitions But I m getting an error saying no `partition expr` is given even though I pass it to the Output metadata my asset ``` asset io manager key= snowflake io manager required resource keys= eldermark partitions def=WeeklyPartitionsDefinition start date= 2022 11 01 key prefix= snowflake eldermark proxy def resident context gt Output DataFrame start end = context asset partitions time window for output filter str = f LastMod Stamp gt = start timestamp AND LastMod Stamp lt end timestamp records = context resources eldermark fetch obj obj= Resident filter=filter str df = pd DataFrame list records columns= src dtype= string yield Output df metadata= partition expr PARSE JSON SRC LASTMOD STAMP TIMESTAMP ``` error thrown ```dagster core errors DagsterExecutionHandleOutputError Error occurred while handling output result of step snowflake eldermark proxy resident File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute plan py line 266 in dagster event sequence for step for step event in check generator step events File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute step py line 394 in core dagster event sequence for step for evt in type check and store output step context user event input lineage File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute step py line 451 in type check and store output for evt in store output step context step output handle output input lineage File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute step py line 636 in store output for elt in iterate with context File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster utils init py line 462 in iterate with context return File Users binhcongpham pyenv versions 3 9 15 lib python3 9 contextlib py line 137 in exit self gen throw typ value traceback File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan utils py line 82 in solid execution error boundary raise error cls The above exception was caused by the following exception ValueError Asset AssetKey snowflake eldermark proxy resident has partitions but no partition expr metadata value so we don t know what column to filter it on File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan utils py line 52 in solid execution error boundary yield File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster utils init py line 460 in iterate with context next output = next iterator File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute step py line 626 in gen fn gen output = output manager handle output output context output value File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core storage db io manager py line 90 in handle output table slice = self get table slice context context File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core storage db io manager py line 183 in get table slice raise ValueError ```

Sean Lopp · Answer

Have you tried adding the metadata w the partition expr to the asset decorator not the output

Binh Pham · Answer

Thank you that worked But why would metadata not be recognized in `Output`

Binh Pham · Answer

Now I m getting a schema error in sqlalchemy The conn is saying that no schema was specified I have defined my asset with a key prefix updated asset ``` asset io manager key= snowflake io manager required resource keys= eldermark partitions def=WeeklyPartitionsDefinition start date= 2022 11 01 key prefix= snowflake eldermark proxy metadata= partition expr PARSE JSON SRC LASTMOD STAMP TIMESTAMP def resident context gt DataFrame start end = context asset partitions time window for output filter str = f LastMod Stamp gt = start timestamp AND LastMod Stamp lt end timestamp records = context resources eldermark fetch obj obj= Resident filter=filter str df = pd DataFrame list records columns= src dtype= string return df``` error ```dagster core errors DagsterExecutionHandleOutputError Error occurred while handling output result of step snowflake eldermark proxy resident File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute plan py line 266 in dagster event sequence for step for step event in check generator step events File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute step py line 394 in core dagster event sequence for step for evt in type check and store output step context user event input lineage File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute step py line 451 in type check and store output for evt in store output step context step output handle output input lineage File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute step py line 636 in store output for elt in iterate with context File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster utils init py line 462 in iterate with context return File Users binhcongpham pyenv versions 3 9 15 lib python3 9 contextlib py line 137 in exit self gen throw typ value traceback File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan utils py line 82 in solid execution error boundary raise error cls The above exception was caused by the following exception sqlalchemy exc ProgrammingError snowflake connector errors ProgrammingError 090106 22000 Cannot perform CREATE TABLE This session does not have a current schema Call USE SCHEMA or use a qualified name SQL CREATE TABLE resident SRC TEXT Background on this error at <https sqlalche me e 14 f405> File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan utils py line 52 in solid execution error boundary yield File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster utils init py line 460 in iterate with context next output = next iterator File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core execution plan execute step py line 626 in gen fn gen output = output manager handle output output context output value File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster core storage db io manager py line 103 in handle output self handlers by type obj type handle output context table slice obj or File Users binhcongpham Repos data operations venv lib python3 9 site packages dagster snowflake pandas snowflake pandas type handler py line 94 in handle output with uppercase cols to sql File Users binhcongpham Repos data operations venv lib python3 9 site packages pandas core generic py line 2951 in to sql return sql to sql File Users binhcongpham Repos data operations venv lib python3 9 site packages pandas io sql py line 698 in to sql return pandas sql to sql File Users binhcongpham Repos data operations venv lib python3 9 site packages pandas io sql py line 1732 in to sql table = self prep table File Users binhcongpham Repos data operations venv lib python3 9 site packages pandas io sql py line 1631 in prep table table create File Users binhcongpham Repos data operations venv lib python3 9 site packages pandas io sql py line 843 in create self execute create File Users binhcongpham Repos data operations venv lib python3 9 site packages pandas io sql py line 829 in execute create self table create bind=self pd sql connectable File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy sql schema py line 962 in create bind run ddl visitor ddl SchemaGenerator self checkfirst=checkfirst File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine base py line 3228 in run ddl visitor conn run ddl visitor visitorcallable element kwargs File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine base py line 2211 in run ddl visitor visitorcallable self dialect self kwargs traverse single element File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy sql visitors py line 524 in traverse single return meth obj kw File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy sql ddl py line 895 in visit table self connection execute File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine base py line 1380 in execute return meth self multiparams params EMPTY EXECUTION OPTS File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy sql ddl py line 80 in execute on connection return connection execute ddl File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine base py line 1472 in execute ddl ret = self execute context File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine base py line 1943 in execute context self handle dbapi exception File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine base py line 2124 in handle dbapi exception util raise File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy util compat py line 210 in raise raise exception File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine base py line 1900 in execute context self dialect do execute File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine default py line 736 in do execute cursor execute statement parameters File Users binhcongpham Repos data operations venv lib python3 9 site packages snowflake connector cursor py line 804 in execute Error errorhandler wrapper self connection self error class errvalue File Users binhcongpham Repos data operations venv lib python3 9 site packages snowflake connector errors py line 276 in errorhandler wrapper handed over = Error hand to other handler File Users binhcongpham Repos data operations venv lib python3 9 site packages snowflake connector errors py line 331 in hand to other handler cursor errorhandler connection cursor error class error value File Users binhcongpham Repos data operations venv lib python3 9 site packages snowflake connector errors py line 210 in default errorhandler raise error class The above exception was caused by the following exception snowflake connector errors ProgrammingError 090106 22000 Cannot perform CREATE TABLE This session does not have a current schema Call USE SCHEMA or use a qualified name File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine base py line 1900 in execute context self dialect do execute File Users binhcongpham Repos data operations venv lib python3 9 site packages sqlalchemy engine default py line 736 in do execute cursor execute statement parameters File Users binhcongpham Repos data operations venv lib python3 9 site packages snowflake connector cursor py line 804 in execute Error errorhandler wrapper self connection self error class errvalue File Users binhcongpham Repos data operations venv lib python3 9 site packages snowflake connector errors py line 276 in errorhandler wrapper handed over = Error hand to other handler File Users binhcongpham Repos data operations venv lib python3 9 site packages snowflake connector errors py line 331 in hand to other handler cursor errorhandler connection cursor error class error value File Users binhcongpham Repos data operations venv lib python3 9 site packages snowflake connector errors py line 210 in default errorhandler raise error class ```

Sean Lopp · Answer

what does your configured call look like for the snowflake io manager

Binh Pham · Answer

```snowflake io manager = build snowflake io manager SnowflakePandasTypeHandler configured snowflake io manager = snowflake io manager configured account env SNOWFLAKE ACCOUNT user env SNOWFLAKE USER password env SNOWFLAKE PASSWORD database env SNOWFLAKE LIFESPRK DATABASE role env SNOWFLAKE ROLE warehouse env SNOWFLAKE WAREHOUSE ```

Sean Lopp · Answer

that is odd it all looks correct what version of the packages FWIW this code snippet is very similar and I just confirmed it works <https gist github com slopp 6b2a6a6342630fe945bc1e5498d01bff> You might try `key prefix = ELDERMARK PROXY ` sometimes capitalization gets a little wonky I d also double check the schema exists in that database the io manager creates tables but it wont create the schema it just uses it

Binh Pham · Answer

I see that schema doesnt exist yet Thanks for the help