Hi All, I am seeing this error which started coming in our deployment. 2022-03-24 001846 +0530 - dagster.daemon.SensorDaemon - INFO - Sensor jira_test_seq_issue_recreate_asset_sensor skipped: Sensor function returned an empty result I have looked at the previous threads. DAGSTER_HOME is set in my environment and I am not using virtual environment. Python version is 3.7. System was running fine and now after we restart the System or the DAGIT Demon nothing helps.

Hi Vishal, is it possible to post or DM the code of jira_test_seq_issue_recreate_asset_sensor?

@asset_sensor(asset_key = AssetKey("jira_test_seq_issue_recreate_asset_key"), job = sensor_based_issue_recreate_for_jira_entity_job, minimum_interval_seconds = 10) def jira_test_seq_issue_recreate_asset_sensor(context, asset_event): issue_recreate_asset_key = asset_event.dagster_event.asset_key.path # issue_recreate_asset_metadata = asset_event.dagster_event.event_specific_data issue_recreate_asset_metadata = asset_event.dagster_event.step_materialization_data.materialization.metadata_entries issue_recreate_pipeline = asset_event.pipeline_name print(f"HI###############################################################{issue_recreate_asset_metadata}") jiraTestSequenceForIssueRepro = None if issue_recreate_asset_metadata and len(issue_recreate_asset_metadata) >= 0: for metadatEntry in issue_recreate_asset_metadata: if metadatEntry.label == "jira_test_sequence_for_issue_recreate": jiraTestSequenceForIssueRepro = metadatEntry.entry_data.data break else: yield SkipReason(f"No Asset Event found") if jiraTestSequenceForIssueRepro: jiraTestSequenceForIssueRepro = JsonSerializeDeserializeUtility.serialize(jiraTestSequenceForIssueRepro) print(f"#########################{jiraTestSequenceForIssueRepro}######################") yield RunRequest( run_key = context.cursor, run_config={ "ops": { "start_issue_repro": { "config": { "jiraTestSequenceForIssueRepro": jiraTestSequenceForIssueRepro } } } }) else: yield SkipReason(f"The Asset materialized does not have any Event with label 'jira_test_sequence_for_issue_recreate'.")

Have you been materializing assets with that key? The sensor will skip until it detects new assets

Yes, We are materializing with the Key. And in one run multiple assets are materialized. Yes, the Sensor is skipping them although the Keys are present.

Does Dagit show a cursor value on the page for the sensor?

yes, It shows same way !!!

and If you click through to the asset listed there (from the "Monitored assets" row, do you see recent materialized assets there)?

yes, do see them.

Under the hood, the asset sensor is querying the event log… would you mind running the following in Python?

from dagster import DagsterInstance, DagsterEventType, EventRecordsFilter, AssetKey
instance = DagsterInstance.get()
cursor = 1498647
event_records = instance.get_event_records(
    EventRecordsFilter(
        event_type=DagsterEventType.ASSET_MATERIALIZATION,
        asset_key=AssetKey("jira_test_seq_issue_recreate_asset_key"),
        after_cursor=cursor,
    ),
    ascending=False,
    limit=1,
)

I want to see if your storage layer has a materialization for that asset key after the given cursor

C:\Users\Administrator\Desktop>python slack.py [] C:\Users\Administrator\Desktop> it returns empty value, in end we printed event_records

if you set

cursor

to be

None

, do you get a value?

We are getting the value. C:\Users\Administrator\Desktop>python slack.py [EventLogRecord(storage_id=1500986, event_log_entry=EventLogEntry(error_info=None, level=10, user_message='Materialized value jira_test_seq_issue_recreate_asset_key.', run_id='e356837c-31ec-44ee-a469-ce73ebca885d', timestamp=1648066530.4770663, step_key='materialize_asset_invoke_issue_recreate_op[605630]', pipeline_name='sensor_based_identify_jira_failed_instances_job', dagster_event=DagsterEvent(event_type_value='ASSET_MATERIALIZATION', pipeline_name='sensor_based_identify_jira_failed_instances_job', step_handle=ResolvedFromDynamicStepHandle(solid_handle=NodeHandle(name='materialize_asset_invoke_issue_recreate_op', parent=None), mapping_key='605630', key='materialize_asset_invoke_issue_recreate_op[605630]'), solid_handle=NodeHandle(name='materialize_asset_invoke_issue_recreate_op', parent=None), step_kind_value='COMPUTE', logging_tags={'pipeline_name': 'sensor_based_identify_jira_failed_instances_job', 'pipeline_tags': '{\'.dagster/grpc_info\': \'{"host": "localhost", "port": 55751}\', \'.dagster/scheduled_execution_time\': \'2022-03-23T201500+00:00\', \'dagster/schedule_name\': \'sensor_based_identify_run_issue_repro_01\'}', 'resource_fn_name': 'None', 'resource_name': 'None', 'run_id': 'e356837c-31ec-44ee-a469-ce73ebca885d', 'solid_name': 'materialize_asset_invoke_issue_recreate_op', 'step_key': 'materialize_asset_invoke_issue_recreate_op[605630]'}, event_specific_data=StepMaterializationData(materialization=AssetMaterialization(asset_key=AssetKey(['jira_test_seq_issue_recreate_asset_key']), description=None, metadata_entries=[MetadataEntry(label='jira_test_sequence_for_issue_recreate', description=None, entry_data=JsonMetadataValue(data={'entity_id': 54, 'entity_workflow_id': 49, 'execution_cycle_id': 55245, 'execution_instance_id': 3246408, 'execution_instance_vyom_ref': '5421812255047035580', 'execution_result_nm': None, 'execution_run_vyom_ref': None, 'execution_status_nm': None, 'external_test_case_id': None, 'issue_reproduction': {'final_issue_status': None, 'iterations': None}, 'jira_external_ref': 'WINT-341', 'jira_test_sequence_id': 190, 'test_case_id': None, 'test_nm': None, 'workflow_config_json': {'config_values': [{'config_name': 'IsolateFl', 'config_value': 'Y'}, {'config_name': 'FWVersionPathUrl', 'config_value': None}, {'config_name': 'TestIterationNbr', 'config_value': 5}, {'config_name': 'SequenceIterationNbr', 'config_value': 5}, {'config_name': 'ParentOrNewRandomSeedFl', 'config_value': 'parent'}, {'config_name': 'SequenceTestCaseCountNbr', 'config_value': 5}, {'config_name': 'OverallTimeoutInSecondsNbr', 'config_value': 5}]}, 'workflow_execution_id': None, 'workflow_id': 1}))], partition=None, tags={}), asset_lineage=[]), message='Materialized value jira_test_seq_issue_recreate_asset_key.', pid=18428, step_key='materialize_asset_invoke_issue_recreate_op[605630]')))]

What storage are you using? Is it the default sqlite storage, or postgres?

Postgres

Very puzzled here… there’s clearly something wrong here but not sure what it could be. Is there anyway you could share a partial dump of your

event_log

table? Something that would include that row (with storage_id

1500986

)…

{"__class__": "EventLogEntry", "dagster_event": {"__class__": "DagsterEvent", "event_specific_data": {"__class__": "StepMaterializationData", "asset_lineage": [], "materialization": {"__class__": "AssetMaterialization", "asset_key": {"__class__": "AssetKey", "path": ["jira_test_seq_issue_recreate_asset_key"]}, "description": null, "metadata_entries": [{"__class__": "EventMetadataEntry", "description": null, "entry_data": {"__class__": "JsonMetadataEntryData", "data": {"entity_id": 54, "entity_workflow_id": 49, "execution_cycle_id": 55245, "execution_instance_id": 3246408, "execution_instance_vyom_ref": "5421812255047035580", "execution_result_nm": null, "execution_run_vyom_ref": null, "execution_status_nm": null, "external_test_case_id": null, "issue_reproduction": {"final_issue_status": null, "iterations": null}, "jira_external_ref": "WINT-341", "jira_test_sequence_id": 190, "test_case_id": null, "test_nm": null, "workflow_config_json": {"config_values": [{"config_name": "IsolateFl", "config_value": "Y"}, {"config_name": "FWVersionPathUrl", "config_value": null}, {"config_name": "TestIterationNbr", "config_value": 5}, {"config_name": "SequenceIterationNbr", "config_value": 5}, {"config_name": "ParentOrNewRandomSeedFl", "config_value": "parent"}, {"config_name": "SequenceTestCaseCountNbr", "config_value": 5}, {"config_name": "OverallTimeoutInSecondsNbr", "config_value": 5}]}, "workflow_execution_id": null, "workflow_id": 1}}, "label": "jira_test_sequence_for_issue_recreate"}], "partition": null, "tags": {}}}, "event_type_value": "ASSET_MATERIALIZATION", "logging_tags": {"pipeline_name": "sensor_based_identify_jira_failed_instances_job", "pipeline_tags": "{'.dagster/grpc_info': '{\"host\": \"localhost\", \"port\": 55751}', '.dagster/scheduled_execution_time': '2022-03-23T201500+00:00', 'dagster/schedule_name': 'sensor_based_identify_run_issue_repro_01'}", "resource_fn_name": "None", "resource_name": "None", "run_id": "e356837c-31ec-44ee-a469-ce73ebca885d", "solid_name": "materialize_asset_invoke_issue_recreate_op", "step_key": "materialize_asset_invoke_issue_recreate_op[605630]"}, "message": "Materialized value jira_test_seq_issue_recreate_asset_key.", "pid": 18428, "pipeline_name": "sensor_based_identify_jira_failed_instances_job", "solid_handle": {"__class__": "SolidHandle", "name": "materialize_asset_invoke_issue_recreate_op", "parent": null}, "step_handle": {"__class__": "ResolvedFromDynamicStepHandle", "key": "materialize_asset_invoke_issue_recreate_op[605630]", "mapping_key": "605630", "solid_handle": {"__class__": "SolidHandle", "name": "materialize_asset_invoke_issue_recreate_op", "parent": null}}, "step_key": "materialize_asset_invoke_issue_recreate_op[605630]", "step_kind_value": "COMPUTE"}, "error_info": null, "level": 10, "message": "", "pipeline_name": "sensor_based_identify_jira_failed_instances_job", "run_id": "e356837c-31ec-44ee-a469-ce73ebca885d", "step_key": "materialize_asset_invoke_issue_recreate_op[605630]", "timestamp": 1648066530.4770663, "user_message": "Materialized value jira_test_seq_issue_recreate_asset_key."}

Was hoping something more of a

pg_dump

file that contains the

event_log

table row with

id

values from

1498647

to

1500986

. Also, the debug file from the run

e356837c-31ec-44ee-a469-ce73ebca885d

could be useful.

Hi prha, I am from Vishal's team. We found few more instances relating to this issue.