https://dagster.io/ logo
Powered by
Title
c

Charles Lariviere

05/18/2021, 5:06 PM
Hey 👋 I’ve been getting the same “framework error” on a pipeline I tried running twice. Curious if you know of ways to debug this? This is running on Kubernetes — the pod did not reach the CPU or Memory limits, and is marked as 
Status:Terminated
and 
Reason: Completed
. 
An exception was thrown during execution that is likely a framework error, rather than an error in user code.
STEP_FAILURE 
dagster.core.errors.DagsterExecutionInterruptedError
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 190, in _dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 310, in core_dagster_event_sequence_for_step
    for user_event in check.generator(
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 69, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 598, in _user_event_sequence_for_step_compute_fn
    for event in iterate_with_context(
  File "/usr/local/lib/python3.8/site-packages/dagster/utils/__init__.py", line 362, in iterate_with_context
    next_output = next(iterator)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 126, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn):
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 92, in _yield_compute_results
    user_event_generator = compute_fn(SolidExecutionContext(step_context), inputs)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/definitions/decorators/solid.py", line 308, in compute
    result = fn(context, **kwargs)
  File "/analytics/solids/database.py", line 58, in load_tables
    context.resources.snowflake.execute_queries(
  File "/usr/local/lib/python3.8/site-packages/dagster_snowflake/resources.py", line 121, in execute_queries
    cursor.execute(sql, parameters)  # pylint: disable=E1101
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 610, in execute
    ret = self._execute_helper(query, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 433, in _execute_helper
    ret = self._connection.cmd_query(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/connection.py", line 942, in cmd_query
    ret = self.rest.request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 381, in request
    return self._post_request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 676, in _post_request
    ret = self._get_request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 580, in _get_request
    ret = self.fetch(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 719, in fetch
    ret = self._request_exec_wrapper(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 762, in _request_exec_wrapper
    return_object = self._request_exec(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 926, in _request_exec
    raw_ret = session.request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1344, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/contrib/pyopenssl.py", line 325, in recv_into
    if not util.wait_for_read(self.socket, self.socket.gettimeout()):
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/util/wait.py", line 146, in wait_for_read
    return wait_for_socket(sock, read=True, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/util/wait.py", line 107, in poll_wait_for_socket
    return bool(_retry_on_intr(do_poll, timeout))
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/util/wait.py", line 43, in _retry_on_intr
    return fn(timeout)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/util/wait.py", line 105, in do_poll
    return poll_obj.poll(t)
  File "/usr/local/lib/python3.8/site-packages/dagster/utils/interrupts.py", line 78, in _new_signal_handler
    raise error_cls()
PIPELINE FAILURE 
dagster.core.errors.DagsterExecutionInterruptedError
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/api.py", line 762, in pipeline_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
  File "/usr/local/lib/python3.8/site-packages/dagster/core/executor/in_process.py", line 38, in execute
    yield from iter(
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/api.py", line 841, in __iter__
    yield from self.iterator(
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 62, in inner_plan_execution_iterator
    for step_event in check.generator(
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 262, in _dagster_event_sequence_for_step
    raise interrupt_error
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 190, in _dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 310, in core_dagster_event_sequence_for_step
    for user_event in check.generator(
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 69, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 598, in _user_event_sequence_for_step_compute_fn
    for event in iterate_with_context(
  File "/usr/local/lib/python3.8/site-packages/dagster/utils/__init__.py", line 362, in iterate_with_context
    next_output = next(iterator)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 126, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn):
  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 92, in _yield_compute_results
    user_event_generator = compute_fn(SolidExecutionContext(step_context), inputs)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/definitions/decorators/solid.py", line 308, in compute
    result = fn(context, **kwargs)
  File "/analytics/solids/database.py", line 58, in load_tables
    context.resources.snowflake.execute_queries(
  File "/usr/local/lib/python3.8/site-packages/dagster_snowflake/resources.py", line 121, in execute_queries
    cursor.execute(sql, parameters)  # pylint: disable=E1101
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 610, in execute
    ret = self._execute_helper(query, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 433, in _execute_helper
    ret = self._connection.cmd_query(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/connection.py", line 942, in cmd_query
    ret = self.rest.request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 381, in request
    return self._post_request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 676, in _post_request
    ret = self._get_request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 580, in _get_request
    ret = self.fetch(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 719, in fetch
    ret = self._request_exec_wrapper(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 762, in _request_exec_wrapper
    return_object = self._request_exec(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/network.py", line 926, in _request_exec
    raw_ret = session.request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1344, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/contrib/pyopenssl.py", line 325, in recv_into
    if not util.wait_for_read(self.socket, self.socket.gettimeout()):
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/util/wait.py", line 146, in wait_for_read
    return wait_for_socket(sock, read=True, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/util/wait.py", line 107, in poll_wait_for_socket
    return bool(_retry_on_intr(do_poll, timeout))
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/util/wait.py", line 43, in _retry_on_intr
    return fn(timeout)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/vendored/urllib3/util/wait.py", line 105, in do_poll
    return poll_obj.poll(t)
  File "/usr/local/lib/python3.8/site-packages/dagster/utils/interrupts.py", line 78, in _new_signal_handler
    raise error_cls()
ENGINE EVENT 
An exception was thrown during execution that is likely a framework error, rather than an error in user code.
dagster.check.CheckError: Invariant failed. Description: Pipeline run etl_creative_market_pipeline (b6e4faec-1ee8-49ae-bbe7-be282a79ed85) in state PipelineRunStatus.FAILURE, expected NOT_STARTED or STARTING

Stack Trace:
  File "/usr/local/lib/python3.8/site-packages/dagster/grpc/impl.py", line 88, in core_execute_run
    yield from execute_run_iterator(recon_pipeline, pipeline_run, instance)
,  File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/api.py", line 74, in execute_run_iterator
    check.invariant(
,  File "/usr/local/lib/python3.8/site-packages/dagster/check/__init__.py", line 167, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")
I get these 3 errors successively.
a

alex

05/18/2021, 6:05 PM
Do you have any external autoscaling systems enabled for the cluster? These interrupt exceptions are likely the 
sigterm
k8s sends when it’s spinning down a pod. Other users have seen this problem when using autoscaling systems that assume all pods can be moved mid computation - which the dagster jobs pods (mostly the run orchestrating process) do not yet support graceful recovery from unexpected termination
c

Charles Lariviere

05/18/2021, 6:21 PM
Ah interesting! We do indeed have an autoscaler — will look into that. Thanks 🙏
#dagster-supportJoin Slack
Powered by