Fran Sanchez

05/21/2020, 5:27 PM
I have a number of solids that require some information from the same source, in this case a file. Opening the file is expensive because an index needs to be built but once extracted each part can be serialised to disk as small parquet files. What is the best way to implement this type of data source in dagster? A solid? A resource? How to make it work efficiently with multiprocessing and avoid loading the same file many times?