sandy06/07/2020, 9:12 PM
Does that answer your question? There are a bunch of concepts involved - I can explain in more detail if it would be helpful, which is a
dagster_pyspark.DataFrameknows how to save and load PySpark DataFrames.
DagsterTypetells Dagster that when a solid has a return type annotation for a,
make_python_type_usable_as_dagster_type, then Dagster should use the
pyspark.sql.DataFrameto save and load it.
David06/08/2020, 5:23 AM
sandy06/08/2020, 3:04 PM
David06/08/2020, 3:18 PM