Hi, I want to loop through some files so using static partition definition. Is there a way to read the file content in function

data_asset

and simply output to as a csv?

from dagster import StaticPartitionsDefinition, AssetExecutionContext, asset
import pandas as pd

dataset_partitions_def = StaticPartitionsDefinition(
    ["weather", "rain"]
)

@asset(partitions_def=dataset_partitions_def)
def data_asset(context: AssetExecutionContext):
    # read in the content

Example I/O manager

class MyPartitionedIOManager(IOManager):
    def _get_path(self, context) -> str:
        if context.has_partition_key:
            return (f"/sfo_q2_{context.asset_partition_key}_sample.csv")
        else:
            return "/".join(context.asset_key.path)

    def handle_output(self, context, obj):
        obj.to_csv(f"/output/sfo_q2_{context.asset_partition_key}_sample.csv")

    def load_input(self, context):
        return pd.read_csv(self._get_path(context))

Hi Lien - you can also do this without IO managers by saving it in the asset itself, or setting up a resource that is used to save the file

Hi @Odette Harary if I want to use the data further down the pipeline, is it recommended to pass it through IO manager?