I m going to have to roll back to an earlier version if ther dagster #integration-snowflake

Join Slack

I'm going to have to roll back to an earlier versi...

# integration-snowflake

clay

03/08/2023, 5:14 PM

I'm going to have to roll back to an earlier version if there's not a way to toggle that off

clay

03/08/2023, 5:15 PM

Copy code

dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "crm_jira_tickets_reduced":

  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_plan.py", line 269, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 386, in core_dagster_event_sequence_for_step
    for evt in _type_check_and_store_output(step_context, user_event):
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 439, in _type_check_and_store_output
    for evt in _store_output(step_context, step_output_handle, output):
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 648, in _store_output
    handle_output_gen,
  File "/usr/local/lib/python3.7/site-packages/dagster/_utils/__init__.py", line 473, in iterate_with_context
    return
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/utils.py", line 90, in op_execution_error_boundary
    ) from e

The above exception was caused by the following exception:
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 003001 (42501): SQL access control error:
Insufficient privileges to operate on database 'SANDBOX'
[SQL: create schema if not exists CRM_GLOBAL_RESTRICTED;]
(Background on this error at: <https://sqlalche.me/e/14/f405>)

  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/utils.py", line 55, in op_execution_error_boundary
    yield
  File "/usr/local/lib/python3.7/site-packages/dagster/_utils/__init__.py", line 471, in iterate_with_context
    next_output = next(iterator)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 630, in _gen_fn
    gen_output = output_manager.handle_output(output_context, output.value)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/storage/db_io_manager.py", line 134, in handle_output
    self._db_client.ensure_schema_exists(context, table_slice, conn)
  File "/usr/local/lib/python3.7/site-packages/dagster_snowflake/snowflake_io_manager.py", line 168, in ensure_schema_exists
    connection.execute(f"create schema if not exists {table_slice.schema};")
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1370, in execute
    future=False,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1676, in _exec_driver_sql
    distilled_parameters,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/cursor.py", line 839, in execute
    Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 294, in errorhandler_wrapper
    error_value,
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 345, in hand_to_other_handler
    cursor.errorhandler(connection, cursor, error_class, error_value)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 231, in default_errorhandler
    cursor=cursor,

The above exception was caused by the following exception:
snowflake.connector.errors.ProgrammingError: 003001 (42501): SQL access control error:
Insufficient privileges to operate on database 'SANDBOX'

  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/cursor.py", line 839, in execute
    Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 294, in errorhandler_wrapper
    error_value,
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 345, in hand_to_other_handler
    cursor.errorhandler(connection, cursor, error_class, error_value)
  File "/usr/local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 231, in default_errorhandler
    cursor=cursor,

clay

03/08/2023, 5:18 PM

here's the error @jamie

jamie

03/08/2023, 5:19 PM

yeah that was a change in .19. I can investigate a different way to check if the schema exists before trying to create it

clay

03/08/2023, 5:20 PM

Thanks! I'll roll back to 1.1.18

clay

03/08/2023, 5:20 PM

do you want me to open a ticket about it?

jamie

03/08/2023, 5:21 PM

yeah that’d be great. i may not have time to get to it this week, so that’ll help as a reminder

❤️ 1

clay

03/08/2023, 5:21 PM

Last question: does dagit version always match dagster version?

clay

03/08/2023, 5:22 PM

so dagit 1.1.18 would be required for the rollback, too?

jamie

03/08/2023, 5:25 PM

if it’s not a pain to roll back dagit, or there isn’t a new feature you need, rolling back is probably a good idea. We do backcompat testing, so it’s most likely fine to keep the versions different, but there’s always a chance there could be issues

clay

03/08/2023, 5:25 PM

great, thanks!

jamie

03/08/2023, 5:47 PM

lol fix was super easy. i’ll have a pr up soon

❤️ 1

clay

03/08/2023, 5:47 PM

i created a ticket btw

clay

03/08/2023, 5:48 PM

but then fell down a rabbit hole of trying to figure out how to fix it but keep running into privileges blocks. How did you end up checking?

jamie

03/08/2023, 5:49 PM

there’s a

show schemas like <pattern> in database <db>

that i’m using. It worked for me in an account with locked down privileges, but if you’re able to confirm that works for you that’d be great

clay

03/08/2023, 5:50 PM

I can run show schemas like. Does that return a table to dagster? The Snowflake docs point out that you have to use RESULT_SCAN if you want to select from the results, which is where I was running into weirdness

clay

03/08/2023, 5:50 PM

If it works for you then that works for me. 🙂

clay

03/08/2023, 5:51 PM

SHOW SCHEMAS LIKE 'CRM_GLOBAL_RESTRICTED' IN DATABASE SANDBOX

works for the case where it was failing for me

jamie

03/08/2023, 5:51 PM

cool. with the python connector api i was just able to slap a

fetchall()

on the query and then it returns a list containing some info about the schema if it exists, otherwise the list is empty

clay

03/08/2023, 5:52 PM

awesome!

clay

03/08/2023, 5:52 PM

Thanks, Jaimie!

2 Views

Open in Slack

Previous Next