What is the best way to monitor "dynamic assets". ...
# ask-community
d
What is the best way to monitor "dynamic assets". An example of what I am talking about would be monitor an S3 bucket for new/updated files, and then put them into database tables (potentially, a new file will mean a new database table). How can I monitor this kind of process? It seems that if I just emit AssetObservations/Materializations without declaring the asset those records just disappear
c
Hi Daniel. Wanted to clarify what you mean by emitting asset events without declaring the asset--do you mean you are yielding asset events from within a sensor?
d
Yes, or within a Job/Op
c
It's currently not supported to yield observations or materializations from within a sensor. You could define a sensor that triggers a job that generates the asset observations/materializations. What do you mean by the records disappearing?
d
Just that they don't get recorded (as there isn't an asset defined for them). I think that is really the core issue. The sensor/job needs some way to "create assets" so that it can be tracked from a job, but I don't see a way to do that dynamically
c
Within an op in the job started by the sensor, you should be able to do something like this:
Copy code
yield AssetMaterialization(asset_key="my_asset_key")
You should then be able to view all assets and their materializations.
d
even if that asset doesn't exist prior?
c
yes
d
H'm so for some reason that doesn't seem to work for me. That should up with the rest of the asssets on the asset window?
c
yes this page, if you specified "foo" as the asset_key
Was the materialization successfully yielded from within the job run?
d
I see it in the log in dagit, but it doesn't seem to show up anywhere
I am doing context.log_event(materialization)
c
Thats odd, I'm able to see the asset on the catalog page when when I log a materialization. What version of dagster are you running?
In your run logs, are you able to see a link to the asset:
d
I do see the link
but not on the catalog
Just figured it out. It just doesn't show up on the graph view, but if you show the list view they do show up. Thank you!
c
Oh yes, got it, that makes sense! Only assets that exist in your repository (defined via
@asset
) will show up on the asset graph, but all other assets will display in the asset catalog (list view)