https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Api RM

03/17/2023, 9:33 AM
I am implementing an hourly scheduled background job that process data from IoT sensors. The data from each sensor is trivial but combined it can be several gigabytes. Should I avoid using software-defined assets to save space?
dagster bot responded by community 1
It seems like using assets (instead of raw ops/jobs) can lead to storage issues which are not addressed in the documentation.
z

Zach

03/17/2023, 5:48 PM
I don't think there's much of a difference between ops / jobs and assets with regards to storage, they both will storage any inputs/outputs using IOManagers. If you use something like the s3_pickle_io_manager then all the outputs will just get sent to S3 so you don't have to worry about local storage.
4 Views