https://dagster.io/ logo
#ask-community
Title
# ask-community
l

Lien

09/05/2023, 3:48 PM
Hi, I want to loop through some files so using static partition definition. Is there a way to read the file content in function
data_asset
and simply output to as a csv?
Copy code
from dagster import StaticPartitionsDefinition, AssetExecutionContext, asset
import pandas as pd

dataset_partitions_def = StaticPartitionsDefinition(
    ["weather", "rain"]
)

@asset(partitions_def=dataset_partitions_def)
def data_asset(context: AssetExecutionContext):
    # read in the content
Example I/O manager
Copy code
class MyPartitionedIOManager(IOManager):
    def _get_path(self, context) -> str:
        if context.has_partition_key:
            return (f"/sfo_q2_{context.asset_partition_key}_sample.csv")
        else:
            return "/".join(context.asset_key.path)

    def handle_output(self, context, obj):
        obj.to_csv(f"/output/sfo_q2_{context.asset_partition_key}_sample.csv")

    def load_input(self, context):
        return pd.read_csv(self._get_path(context))
o

Odette Harary

09/05/2023, 8:52 PM
Hi Lien - you can also do this without IO managers by saving it in the asset itself, or setting up a resource that is used to save the file
l

Lien

09/14/2023, 8:28 AM
Hi @Odette Harary if I want to use the data further down the pipeline, is it recommended to pass it through IO manager?