https://dagster.io/ logo
#ask-community
Title
# ask-community
k

Kazushi Nagayama

08/15/2023, 12:04 PM
i was successful running a job locally but when i upload it to dagster cloud, i get:
Copy code
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp9u6bywne/storage/my_projects'
what am i doing wrong here?
z

Zach

08/15/2023, 3:59 PM
The default IO manager writes to
/tmp/
, local to where the asset was materialized. If downstream assets are not materialized as part of the same job they won't be able to find upstream outputs, because they'll have been saved to the ephemeral ECS container that ran the upstream materialization. I would suggest configuring your assets to use the
ConfigurablePickledObjectS3IOManager
from
dagster-aws
so the outputs get saved to S3 and are retrievable by subsequent materializations. This doc has an example of using the
ConfigurablePickledObjectS3IOManager
https://docs.dagster.io/concepts/io-management/io-managers#applying-io-managers-to-assets
k

Kazushi Nagayama

08/16/2023, 12:46 AM
Thanks for the reply! I see, I guess I’ll just deploy to my own instance and not use Cloud
t

Tim Castillo

08/16/2023, 5:36 PM
Hi there! Is there a particular reason you're choosing to use instance instead of Cloud, given what Zach said? Depending on how you deploy in production, you'll get the same error message. Dagster is made to orchestrate assets being made and not to act as storage, and the local file IO manager is made for convenience. I'd recommend that you store your assets into somewhere permanent, like S3 or a database.
k

Kazushi Nagayama

08/17/2023, 3:15 AM
these 'assets' are just tmp files that are being created because of how the pipeline is made. overall, dagster focusing on assets seem to not fit our use case, so we've decided to consider other orchestration pipelines. thanks!
z

Zach

08/17/2023, 4:08 PM
Sounds like you've made up your mind, but for posterity here are the docs on ops and graphs, which is Dagster's other core framework for organizing non-asset pipelines (and which assets are built on top of). Here's a discussion on when to use ops and graphs vs. when to use assets. Good luck!
❤️ 1