https://dagster.io/ logo
#integration-snowflake
Title
# integration-snowflake
t

Tobias Otte

03/02/2023, 4:12 PM
Hello, i just started to try out dagster by using the quickstart_snowflake template with dagster cloud, but the materilization fails, because it seems like the tables cant be created. The the following log. The role inside snowflake has the necssary permissions to operate on the schema and create new tables. If i login with the user and role i use with dagster and execute "create table HACKERNEWS_TOPSTORY_IDS(item_ids variant)" then the hackernews_topstory_ids step works but the next step fails. There just creating the table doesn't seem to fix it, because one column seems to be named "by", which is also a sql keyword and leads to an error. Have you been running the quickstart_snowflake template in dagster cloud successfully? Thank you and kind regards Tobias
dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "hackernews_topstory_ids":
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py", line 269, in dagster_event_sequence_for_step
for step_event in check.generator(step_events):
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 386, in core_dagster_event_sequence_for_step
for evt in _type_check_and_store_output(step_context, user_event):
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 439, in _type_check_and_store_output
for evt in _store_output(step_context, step_output_handle, output):
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 640, in _store_output
for elt in iterate_with_context(
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_utils/__init__.py", line 473, in iterate_with_context
return
File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_core/execution/plan/utils.py", line 85, in op_execution_error_boundary
raise error_cls(
The above exception was caused by the following exception:
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 002003 (42S02): SQL compilation error:
Object 'THEMENSTEUERUNG.TEST_SCHEMA.HACKERNEWS_TOPSTORY_IDS' does not exist or not authorized.
[SQL: DELETE FROM THEMENSTEUERUNG.TEST_SCHEMA.hackernews_topstory_ids]
(Background on this error at: <https://sqlalche.me/e/14/f405>)
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_core/execution/plan/utils.py", line 55, in op_execution_error_boundary
yield
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_utils/__init__.py", line 471, in iterate_with_context
next_output = next(iterator)
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 630, in _gen_fn
gen_output = output_manager.handle_output(output_context, output.value)
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster/_core/storage/db_io_manager.py", line 136, in handle_output
self._db_client.delete_table_slice(context, table_slice, conn)
File "/venvs/1791569dd264/lib/python3.8/site-packages/dagster_snowflake/snowflake_io_manager.py", line 173, in delete_table_slice
connection.execute(_get_cleanup_statement(table_slice))
File "/venvs/1791569dd264/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1365, in execute
return self._exec_driver_sql(
File "/venvs/1791569dd264/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1669, in _exec_driver_sql
ret = self._execute_context(
File "/venvs/1791569dd264/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
self._handle_dbapi_exception(
File "/venvs/1791569dd264/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
util.raise_(
File "/venvs/1791569dd264/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/venvs/1791569dd264/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/venvs/1791569dd264/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
File "/venvs/1791569dd264/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 827, in execute
Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
File "/venvs/1791569dd264/lib/python3.8/site-packages/snowflake/connector/errors.py", line 275, in errorhandler_wrapper
handed_over = Error.hand_to_other_handler(
File "/venvs/1791569dd264/lib/python3.8/site-packages/snowflake/connector/errors.py", line 330, in hand_to_other_handler
cursor.errorhandler(connection, cursor, error_class, error_value)
File "/venvs/1791569dd264/lib/python3.8/site-packages/snowflake/connector/errors.py", line 209, in default_errorhandler
raise error_class(
The above exception was caused by the following exception:
snowflake.connector.errors.ProgrammingError: 002003 (42S02): SQL compilation error:
Object 'THEMENSTEUERUNG.TEST_SCHEMA.HACKERNEWS_TOPSTORY_IDS' does not exist or not authorized.
File "/venvs/1791569dd264/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/venvs/1791569dd264/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
File "/venvs/1791569dd264/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 827, in execute
Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
File "/venvs/1791569dd264/lib/python3.8/site-packages/snowflake/connector/errors.py", line 275, in errorhandler_wrapper
handed_over = Error.hand_to_other_handler(
File "/venvs/1791569dd264/lib/python3.8/site-packages/snowflake/connector/errors.py", line 330, in hand_to_other_handler
cursor.errorhandler(connection, cursor, error_class, error_value)
File "/venvs/1791569dd264/lib/python3.8/site-packages/snowflake/connector/errors.py", line 209, in default_errorhandler
raise error_class(
y

yuhan

03/02/2023, 9:55 PM
Hi Tobias, I’ve run this template successfully. But let me take a look and see if I can repro this issue now.
j

jamie

03/02/2023, 10:43 PM
hi! this is a bug that will be fixed in this weeks release!
🙏 1
t

Tobias Otte

03/03/2023, 7:50 AM
Thank you. I see a Bugfix for dagster-snowflake in 1.1.21 that was published today. Is that the bug you are referring to? As of now the run in dagster cloud still fails. I am not sure where i can see the dagster version used by dagster cloud but this should be 1.1.21 as well?
j

jamie

03/03/2023, 3:31 PM
yep, it should be in the 1.1.21 release. you may need to redeploy your docker file so that it has the latest dagster version
t

Tobias Otte

03/03/2023, 3:55 PM
Thank you, i am using the Dagster + Snowflake starter kit on dagster-cloud. I have redeployed my github project and as far as i can see there is no explicit version in the dependencies, so it should use the lastest.
j

jamie

03/03/2023, 4:04 PM
and just to confirm - after redeploying you still see the bug?
t

Tobias Otte

03/03/2023, 4:11 PM
Yes. I also just tried the iris example shown here, but used the same project: https://docs.dagster.io/integrations/snowflake/using-snowflake-with-dagster I get errors that the tables IRIS_DATASET and IRIS_CLEANED don't exist or are not authroized. If i create them manualy in snowflake, then the Iris example works.
j

jamie

03/03/2023, 4:13 PM
ok - i just tried running some examples locally and they are working for me. are you able to pull down the docker image and exec into it to see the version of the dagster-snowflake library?
3 Views