Jake Kagan
03/14/2023, 4:26 PMExperimentalWarning: "resource_defs" is an experimental argument to function "asset". It may break in futur
e versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at <https://docs.python>.
org/3/library/warnings.html#describing-warning-filters.
i'm trying to give an io manager to my asset; kinda confused about this:
the io manager uploads a parquet file into s3. so what other way would i be able to do this with an asset if not through a resource def like so:
# ----------------------------------------------- VARS ----------------------------------------------- #
RESOURCE_DEFS = {"io_df_to_s3": common_io_df_to_s3,
}
# ----------------------------------------------- JOB ----------------------------------------------- #
@asset(resource_defs={**RESOURCE_DEFS}, io_manager_key="io_df_to_s3")
def beg_inv_excel_to_s3():
# INTO S3:
BUCKET = "my_bucket/"
FILE_PATH = "my_path/"
FILE_FORMAT = 'PARQUET'
FILE_NAME = 'my_excel_file'
FILE_NAME = f"{FILE_NAME}.{FILE_FORMAT.lower()}"
EXCEL_FILE = r'L:my_excel_file.xlsx'
EXCEL_SHEET = r'my_sheet'
excel_data = pl.read_excel(file=EXCEL_FILE, sheet_name=EXCEL_SHEET)
return {"file_name": FILE_NAME, "file_path": BUCKET + FILE_PATH, "polars_dataframe": excel_data}
yuhan
03/17/2023, 12:44 AM@asset(required_resource_keys={"api"})
def beg_inv_excel_to_s3():
and then thread it in using with_resources
with_resources(
definitions=[beg_inv_excel_to_s3],
resource_defs={"api": foo_resource},
)
or directly apply it on definition level:
defs = Definitions(
assets=[asset_requires_resource],
resources={"foo": foo_resource},
)
yuhan
03/17/2023, 12:45 AM