Hello, I am trying to get an AssetMaterialization test running based on the website, but am running into the following after using:

from dagster import job, op, Output

what's the error you're seeing? just based on the import statement, you probably want to import

AssetMaterialization

from dagster as well, if you haven't already

dagster.core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "my_materialization_op": The above exception was caused by the following exception: NameError: name 'AssetMaterialization' is not defined

as written, this op will just output the object that is inside

remote_storage_path

, which in this case is the string

'.csv'

. if you want dagster to write the contents of

df

to a remote storage system, you have a few options. What remote storage system are you using, and how do you usually store stuff there?

Ah, that makes sense. At this time, I am just trying to save locally (same folder as the read_csv). Eventually, we'll use Azure.

ah ok! what does the pipeline that you're trying to build look like? will you be using this output (or the remote_storage_path) in other ops? or do you just want to read the data in this file, modify it, and store it remotely all in one op?

For this scenario, I was hoping to read, modify, and store remotely all in one. This is my first try with materializations, so still exploring all available options.

for the local case, you can just add a

<http://df.to|df.to>_csv("your_chosen_path.csv")

before your AssetMaterialization event, which will store it to whatever local path you want. to store remotely, you'll probably want to get the contents of the df as a csv-formatted string (which would be

csv_string = <http://df.to|df.to>_csv()

), at which point you'd have to call the azure API to upload that string to a specific location

Sounds good...appreciate the help!

no problem 👍

Sorry, a couple questions from one of my coworkers: Do I have it correct that AssetMaterializations are for persisting op outputs for other downstream op’s to be able to use? Also, is it best to use AssetMaterializations in prod only, and skip them for dev/testing?

AssetMaterializations are just events that let Dagster know what's going on inside your job. They'll be stored in the event database, and will help you visualize/track how assets are changed/updated over time. Depending on your needs and how you deploy your jobs, it might be useful to emit AssetMaterializations in all environments, or just prod, but there's no single right answer there.

Hi Owen, I'm trying to use fs_io_manager for local testing and am getting the following error. Am I approaching the base path in the incorrect way? (Dagster documentation snapshot included too)

when you call configured, you're setting specific configurable fields on the fs_io_manager. One of those configurable fields is "base_path", which is base directory under which it will store all of its files. The argument to this

configured

function is a dictionary where each key is the name of a configurable field, and the value is the value you want to set it to. So in this case you'd want this to be

fs_io_manager.configured({"base_path": "/mnt/c/psf/dagstertesting/"})

(i.e. "set base_path to be /mnt/c/...")

Okay, thank you...I originally had it set up like what you suggested, but got the same error message that I just sent. Anything else I might be missing from this?

ah sorry, this should be "base_dir", not "base_path". apologies for the confusion/bad docs example

Ah, that worked...very exciting, thanks!

@Joe Schmid there is, but not a built-in one. you would need to write your own IOManager that implements this functionality. This would probably look very similar to the default PickledObjectFilesystemIOManager , but the handle_output function would write to a csv directly (rather than serializing the object using pickle and writing it to a generic file).

Got it, thanks!

hi @Joe Schmid! In general, this seems like a pretty reasonable setup. I'm not familiar with ODBC, but do you need to pass that opened connection object to the ops that interface with your database? if so, I'd recommend using a resource for that functionality, and adding it as a

required_resource_key

on the ops that talk to the database. This has a few benefits, including allowing you to substitute this resource for a mock one if you want to test your ops. As for the AssetMaterializations, you can optionally yield an AssetMaterialization every time that you save a dataframe to a SQL table. This won't impact the functionality of your job in any way, and is totally optional, but it will help Dagster keep track of changes to that table over time, which might be useful. You could also yield AssetMaterializations for those saved csv files if you want those to be tracked as well.

Thanks, Owen! We'll try it out.