Is there an example of how to use the

s3_resource

from

dagster_aws

?

the airline demo uses it here https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/airline_demo/cache_file_from_s3.py#L48

Thanks! How does system_storage or file_manager come into play in this?

so you can have a system_storage with

required_resource_keys

including

s3

and then use the

s3_resource

to provide that resource. You can see in the airline demo that resource is provided under the key s3 https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/airline_demo/pipelines.py#L53

Ok. I’ve been looking over that and other pieces of code, but I’m having a really tough time understanding how to actually use it. The ObjectStore APIs also look interesting, but I’m even more lost on how that hooks in. 🙂

ya you've certainly bumped in to one of the less polished corners of the system

I’m not sure if I missed something in the tutorial, but I feel like I didn’t learn anything about dealing with files and objects.

Thats good feedback, we could certainly do better. What exactly is your end goal?

Generally speaking, I want to read an object from S3, do some text processing, and write a new object to S3.

So, I think for now you can ignore the system_storage and file_manager since those are hooks for controlling how dagsters internally handles the intermediates. I think the airline demo is probably the best example to follow.

So far, I’ve successfully gotten some existing code to run that ignores resources, FileHandles, etc, and just directly handles files and objects. But, I’m trying (unsuccessfully) to use Dagster’s APIs.

the documentation for the airline demo needs to be rewritten but the code it self is in decent shape

Ah, ok!

the README i think is just referring to itself being out of date - language we should improve (or just do the rewrite)

K.