https://dagster.io/ logo
Powered by
Title
c

clay

03/08/2023, 5:14 PM
I'm going to have to roll back to an earlier version if there's not a way to toggle that off
dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "crm_jira_tickets_reduced":

  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_plan.py", line 269, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 386, in core_dagster_event_sequence_for_step
    for evt in _type_check_and_store_output(step_context, user_event):
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 439, in _type_check_and_store_output
    for evt in _store_output(step_context, step_output_handle, output):
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 648, in _store_output
    handle_output_gen,
  File "/usr/local/lib/python3.7/site-packages/dagster/_utils/__init__.py", line 473, in iterate_with_context
    return
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/utils.py", line 90, in op_execution_error_boundary
    ) from e

The above exception was caused by the following exception:
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 003001 (42501): SQL access control error:
Insufficient privileges to operate on database 'SANDBOX'
[SQL: create schema if not exists CRM_GLOBAL_RESTRICTED;]
(Background on this error at: <https://sqlalche.me/e/14/f405>)

  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/utils.py", line 55, in op_execution_error_boundary
    yield
  File "/usr/local/lib/python3.7/site-packages/dagster/_utils/__init__.py", line 471, in iterate_with_context
    next_output = next(iterator)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 630, in _gen_fn
    gen_output = output_manager.handle_output(output_context, output.value)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/storage/db_io_manager.py", line 134, in handle_output
    self._db_client.ensure_schema_exists(context, table_slice, conn)
  File "/usr/local/lib/python3.7/site-packages/dagster_snowflake/snowflake_io_manager.py", line 168, in ensure_schema_exists
    connection.execute(f"create schema if not exists {table_slice.schema};")
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1370, in execute
    future=False,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1676, in _exec_driver_sql
    distilled_parameters,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/cursor.py", line 839, in execute
    Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 294, in errorhandler_wrapper
    error_value,
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 345, in hand_to_other_handler
    cursor.errorhandler(connection, cursor, error_class, error_value)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 231, in default_errorhandler
    cursor=cursor,

The above exception was caused by the following exception:
snowflake.connector.errors.ProgrammingError: 003001 (42501): SQL access control error:
Insufficient privileges to operate on database 'SANDBOX'

  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/cursor.py", line 839, in execute
    Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 294, in errorhandler_wrapper
    error_value,
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 345, in hand_to_other_handler
    cursor.errorhandler(connection, cursor, error_class, error_value)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 231, in default_errorhandler
    cursor=cursor,
here's the error @jamie
j

jamie

03/08/2023, 5:19 PM
yeah that was a change in .19. I can investigate a different way to check if the schema exists before trying to create it
c

clay

03/08/2023, 5:20 PM
Thanks! I'll roll back to 1.1.18
do you want me to open a ticket about it?
j

jamie

03/08/2023, 5:21 PM
yeah that’d be great. i may not have time to get to it this week, so that’ll help as a reminder
❤️ 1
c

clay

03/08/2023, 5:21 PM
Last question: does dagit version always match dagster version?
so dagit 1.1.18 would be required for the rollback, too?
j

jamie

03/08/2023, 5:25 PM
if it’s not a pain to roll back dagit, or there isn’t a new feature you need, rolling back is probably a good idea. We do backcompat testing, so it’s most likely fine to keep the versions different, but there’s always a chance there could be issues
c

clay

03/08/2023, 5:25 PM
great, thanks!
j

jamie

03/08/2023, 5:47 PM
lol fix was super easy. i’ll have a pr up soon
❤️ 1
c

clay

03/08/2023, 5:47 PM
i created a ticket btw
but then fell down a rabbit hole of trying to figure out how to fix it but keep running into privileges blocks. How did you end up checking?
j

jamie

03/08/2023, 5:49 PM
there’s a 
show schemas like <pattern> in database <db>
that i’m using. It worked for me in an account with locked down privileges, but if you’re able to confirm that works for you that’d be great
c

clay

03/08/2023, 5:50 PM
I can run show schemas like. Does that return a table to dagster? The Snowflake docs point out that you have to use RESULT_SCAN if you want to select from the results, which is where I was running into weirdness
If it works for you then that works for me. 🙂
SHOW SCHEMAS LIKE 'CRM_GLOBAL_RESTRICTED' IN DATABASE SANDBOX
works for the case where it was failing for me
j

jamie

03/08/2023, 5:51 PM
cool. with the python connector api i was just able to slap a 
fetchall()
on the query and then it returns a list containing some info about the schema if it exists, otherwise the list is empty
c

clay

03/08/2023, 5:52 PM
awesome!
Thanks, Jaimie!
#dagster-snowflakeJoin Slack
Powered by