https://dagster.io/ logo
#ask-community
Title
# ask-community
g

Gustavo Carvalho

06/06/2023, 4:29 PM
Hi! Have you ever encountered this error:
dagster._check.CheckError: Invariant failed. Description: Parent pipeline snapshot id out of sync with passed parent pipeline snapshot
? How can I overcome it?
a

alex

06/06/2023, 5:14 PM
in what context did you encounter this error?
g

Gustavo Carvalho

06/06/2023, 5:16 PM
Hi @alex I am running dagit locally and changing some configs in one of my assets and requesting backfills. It was working until I changed "something", then I started getting the error
I am trying to revert the changes and checking what caused it, I am starting to guess it is related to my asset's metadata
here's my asset definition code:
Copy code
@asset(
        name="Way2_ETL",
        description=description or "Way2 ETL",
        partitions_def=partitions_def,
        compute_kind="Way2 API",
        key_prefix=key_prefix,
        required_resource_keys={
            way2_api_connector_key,
            sqlalchemy_engine_key,
        },
        # metadata={
        #     "scada_id": scada_id,
        #     "pontoId_mapping": pontoId_mapping,
        #     "grandezas_mapping": grandezas_mapping,
        #     "resource_key_mapping": {
        #         "way2_api_connector_key": way2_api_connector_key,
        #         "sqlalchemy_engine_key": sqlalchemy_engine_key,
        #     },
        # },
    )
    def way2_etl(context: OpExecutionContext):
        ...
It works with
metadata
commented out, but stops working when I uncomment it
a

alex

06/06/2023, 5:18 PM
what precisely stops working? the backfills?
g

Gustavo Carvalho

06/06/2023, 5:19 PM
yeah,
a

alex

06/06/2023, 5:19 PM
what version are you on?
g

Gustavo Carvalho

06/06/2023, 5:19 PM
let me check
dagster 1.3.7 dagit 1.3.7 dagster-graphql 1.3.7
I am uncommenting part of my metadata to isolate what's causing the error
a

alex

06/06/2023, 5:22 PM
are you using
dagster dev
or manually running dagit and the daemon? You could try restarting the daemon
g

Gustavo Carvalho

06/06/2023, 5:22 PM
dagster dev
a

alex

06/06/2023, 5:23 PM
a full stack trace would be helpful if you are receiving one
g

Gustavo Carvalho

06/06/2023, 5:23 PM
sure
just a sec
btw,I guess I found the problem
I am passing a
dict[int, int]
to my metadata
No errors when running as below
Copy code
dagster._check.CheckError: Invariant failed. Description: Parent pipeline snapshot id out of sync with passed parent pipeline snapshot

  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster_graphql/implementation/utils.py", line 126, in _fn
    return fn(*args, **kwargs)
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster_graphql/implementation/utils.py", line 57, in _fn
    result = fn(self, graphene_info, *args, **kwargs)
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster_graphql/schema/roots/mutation.py", line 281, in mutate
    return create_execution_params_and_launch_pipeline_exec(graphene_info, executionParams)
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster_graphql/schema/roots/mutation.py", line 259, in create_execution_params_and_launch_pipeline_exec
    return launch_pipeline_execution(
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 32, in launch_pipeline_execution
    return _launch_pipeline_execution(graphene_info, execution_params)
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 66, in _launch_pipeline_execution
    run = do_launch(graphene_info, execution_params, is_reexecuted)
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 49, in do_launch
    dagster_run = create_valid_pipeline_run(graphene_info, external_job, execution_params)
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster_graphql/implementation/execution/run_lifecycle.py", line 79, in create_valid_pipeline_run
    dagster_run = graphene_info.context.instance.create_run(
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster/_core/instance/__init__.py", line 1395, in create_run
    dagster_run = self._construct_run_with_snapshots(
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster/_core/instance/__init__.py", line 1118, in _construct_run_with_snapshots
    self._ensure_persisted_job_snapshot(job_snapshot, parent_job_snapshot)
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster/_core/instance/__init__.py", line 1165, in _ensure_persisted_job_snapshot
    check.invariant(
  File "/home/gustavo/miniconda3/lib/python3.10/site-packages/dagster/_check/__init__.py", line 1654, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")
Full stack trace
a

alex

06/06/2023, 5:27 PM
hmm, i find that surprising if its just a regular
dict[int,int]
is it something slightly more complex than that?
g

Gustavo Carvalho

06/06/2023, 5:27 PM
not at all
Copy code
with open(root / "json_definitions/circuits.json") as file:
    circuits = json.load(file)

pontoId_mapping = {circuit["tag"]: circuit["id"] for circuit in circuits}
here's the code defining it
Copy code
[
 {
  "id": 100,
  "tag": 5174
 },
 {
  "id": 101,
  "tag": 5173
 },
 {
  "id": 102,
  "tag": 5172
 },
 ...
]
this is
circuits.json
i will try to build a quick MWE
ok, the MWE did not report the error
I must clarify that I made changes to
pontoId_mapping
, btw. Before (the asset could backfill even with the
dict[int, int]
in the metadata)
Copy code
pontoId_mapping = {circuit["id"]: circuit["tag"] for circuit in circuits}
After (the asset backfill started giving error)
Copy code
pontoId_mapping = {circuit["tag"]: circuit["id"] for circuit in circuits}
a

alex

06/06/2023, 5:38 PM
that before and after looks the same unless im missing something
g

Gustavo Carvalho

06/06/2023, 5:38 PM
sry
I swapped
tag
and
id
(edited above accordingly)
a

alex

06/06/2023, 5:42 PM
hmm, does`circuits.json` have any collisions in tag values?
g

Gustavo Carvalho

06/06/2023, 5:43 PM
let me check, it should not
a

alex

06/06/2023, 5:44 PM
if you had a string value and int value for the same key, that would cause issues going through serialize->deserialize, something like this:
Copy code
>>> {4: 'x', '4': 'y'}
{4: 'x', '4': 'y'}
>>> d = {4: 'x', '4': 'y'}
>>> import json
>>> json.dumps(d)
'{"4": "x", "4": "y"}'
>>> json.loads(json.dumps(d))
{'4': 'y'}
g

Gustavo Carvalho

06/06/2023, 5:45 PM
got it, but there are no collisions in the data
a

alex

06/06/2023, 5:46 PM
got it. I havent been able to produce a repro either
g

Gustavo Carvalho

06/06/2023, 5:46 PM
neither for
id
or
tag
, each field is "all uniques"
I believe it is something that is persisted somewhere and is giving me the snapshot id conflict
however, I already have cleared all my $DAGSTER_HOME directory (multiple times actually), but the error still happening
can this be stored somewhere else?
a

alex

06/06/2023, 5:52 PM
I believe the desync is happening between how this is loaded in code server process that loads the definitions directly and the dagit/daemon host processes that operate on serialized representations that are fetched from the code server
g

Gustavo Carvalho

06/06/2023, 5:52 PM
hmmm
that would make sense in the clashing between str and int
but i couldnt reproduce the mwe
a

alex

06/06/2023, 5:53 PM
are you unblocked by using
dict[str,int]
?
g

Gustavo Carvalho

06/06/2023, 5:54 PM
yes
if do this:
"pontoId_mapping": {str(k): v for k, v in pontoId_mapping.items()},
it works
a

alex

06/06/2023, 5:54 PM
would you be willing to file an issue for this ? https://github.com/dagster-io/dagster/issues
g

Gustavo Carvalho

06/06/2023, 6:25 PM
Yeah, I would! What kind of data should I provide, since I couldn't isolate a MWE?
thank you box 1
a

alex

06/06/2023, 7:31 PM
including what you know and what you tried will be helpful
👍 1
g

Gustavo Carvalho

06/07/2023, 12:46 AM
13 Views