In a dagster operation I want to download a file a...
# ask-community
l
In a dagster operation I want to download a file and write it to a particular file location. The code runs without errors, but the file does not appear in the location. Using a docker to deploy. Any idea why this happenes?
dagster bot responded by community 1
Could somebody help?
dagster bot responded by community 1
In fact a find indicates that the file does not exist anywhere
z
I don't think yielding AssetMaterializations actually writes out any data anywhere, it's pretty much just a way to emit a metadata event to track some info about an asset. I think you'll need to either manually write your file to `target_path`within that op (then yielding the asset materialization is just a nice way to show the the path in the dagit UI), or refactor to use the SDA approach with the
@asset
decorator and an IOManager that will persist your file where you want it to go
I'm not sure dynamic outputs work with
@asset
though, so you'll probably need to just save the file manually within your op
l
the
client.download
function should manually save it to the
target_path
This works outside of dagster. any idea why it wont in this case?
z
what run launcher are you using?
l
Could the io_manager be intercepting it?
z
unlikely / no
l
dagster_docker DockerRunLauncher
maybe need to specify the output path as volume?
z
so the docker run launcher runs each dagster job in a new docker container which is terminated at the end of the job run. so anything you save "locally" within the job will be saved to the container, which is killed when the job run ends. so you need to either write to a volume mount or write to a remote storage location, like S3
l
thx! I will try this and get back to you
it worked thx Zach
🎉 1