I ran into an issue where I accidentally ran a job where an dagster #ask-community

I ran into an issue where I accidentally ran a job...

John Boyle

05/24/2022, 6:16 AM

I ran into an issue where I accidentally ran a job where an assetmaterialization had an asset_key="". This results in the following error when running the job in dagit (edit: and navigating to the /instance/assets/ page)

Copy code

TypeError: '<' not supported between instances of 'str' and 'NoneType'
  File "D:\dev\dagster\dagster\venv\lib\site-packages\dagster_graphql\implementation\utils.py", line 49, in _fn
    return fn(*args, **kwargs)
  File "D:\dev\dagster\dagster\venv\lib\site-packages\dagster_graphql\implementation\fetch_assets.py", line 38, in get_assets
    materialized_keys = instance.get_asset_keys(
  File "D:\dev\dagster\dagster\venv\lib\site-packages\dagster\utils\__init__.py", line 616, in inner
    return func(*args, **kwargs)
  File "D:\dev\dagster\dagster\venv\lib\site-packages\dagster\core\instance\__init__.py", line 1335, in get_asset_keys
    return self._event_storage.get_asset_keys(prefix=prefix, limit=limit, cursor=cursor)
  File "D:\dev\dagster\dagster\venv\lib\site-packages\dagster\core\storage\event_log\sql_event_log.py", line 833, in get_asset_keys
    asset_keys = [AssetKey.from_db_string(row[1]) for row in sorted(rows, key=lambda x: x[1])]

You can easily recreate this by running the materializations.py tutorial

Copy code

dagster\examples\docs_snippets\docs_snippets\intro_tutorial\advanced\materializations

if you run it one time, going to http://127.0.0.1:3000/instance/assets results in the usual result where you can see the sorted_cereals_csv asset defined in the example. However, if you change asset_key="sorted_cereals_csv" to asset_key="" and then rerun the job, you get the error at the top of this post. the bummer is that if you fix the error and then re-run the job, going to http://127.0.0.1:3000/instance/assets still results in the error (presumably breaking on reading the asset with the blank key). I tried deleting the offending run but it didn't seem to affect the assets. and since I can't load up the assets page, I can't wipe the bad assets. also, I can go to the working job and still load up the properly labeled assets, but it's basically broken the repo for now. is there a way to fix this so my repo will work properly again? (i.e. load up the /assets/ page again).

🤖 1

sean

05/25/2022, 4:56 PM

Thanks for this report. We shouldn’t be allowing blank strings as asset keys, I’ll open up an issue for that. We also may be able to include a patch in this week’s release (going out tmrw) to allow reading the blank asset key and fix your broken repo. cc @sandy for any recommendations on how John can delete the reference to the invalid asset key so that his repo works without us needing to patch

sandy

05/25/2022, 4:58 PM

@prha - what do you think?

prha

05/25/2022, 5:23 PM

I think the only way to do this at the moment is to manually delete the row from your event log DB

sean

05/25/2022, 5:24 PM

PR to disallow empty asset keys: https://github.com/dagster-io/dagster/pull/8069

prha

05/25/2022, 5:25 PM

delete from asset_keys where asset_key is null;

John Boyle

05/25/2022, 5:26 PM

Thanks!

sean

05/25/2022, 5:43 PM

@John Boyle just to confirm, were you able to delete the asset key and fix your repo using prha’s advice?

John Boyle

05/25/2022, 7:09 PM

@sean hm, I tried and seem to be having issues. might be mysql 101 issues.

Copy code

select * from asset_keys where asset_key is null;

seems to find the null keys, when I run the delete command, it doesn't seem to find the null keys and nothing is deleted. I'm using python to do this

Copy code

select_result = [a for a in cur.execute('select * from asset_keys where asset_key is null;')]
delete_result = [a for a in cur.execute('delete from asset_keys where asset_key is null;')]

print(select_result)
print(delete_result)

select_result looks correct. [(0, None, '{"__ class __" : "EventLogHistory ... ] delete_result = []

sean

05/25/2022, 7:12 PM

Hmm-- if you run the select again do you get the same? It’s possible that delete just doesn’t return anything (not an expert here).

John Boyle

05/25/2022, 7:15 PM

doh, I got it working. I forgot to

Copy code

cur.commit()

John Boyle

05/25/2022, 7:16 PM

after doing that, then running again returns [] and [] for both queries

John Boyle

05/25/2022, 7:16 PM

and dagit is happy again 🙂 thanks again for the help and follow-up

sean

05/25/2022, 7:20 PM

Great, thanks for your report, we merged a patch that prevents empty asset keys so this won’t happen to anyone else!

Open in Slack

Previous Next