The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

I am implementing an hourly scheduled background job that process data from IoT sensors. The data from each sensor is trivial but combined it can be several gigabytes. Should I avoid using software-defined assets to save space?

It seems like using assets (instead of raw ops/jobs) can lead to storage issues which are not addressed in the documentation.

I don't think there's much of a difference between ops / jobs and assets with regards to storage, they both will storage any inputs/outputs using IOManagers. If you use something like the s3_pickle_io_manager then all the outputs will just get sent to S3 so you don't have to worry about local storage.