https://dagster.io/ logo
Title
l

Lucas Santiago Cardoso

05/25/2023, 1:22 PM
Hey team, Me and my team are new to dagster... and we are trying to adjust our code to dagster style... Some of our etls starts with a list of files from s3, so we thought that implementing a resource to accomplish. But I'm getting some errors with pydantic when doing something similar to this
I Have a Resource class with the S3 listing logic, and created the resource as well in the link. However when defining in the
Definitions()
its not working.
d

David Weber

05/25/2023, 1:37 PM
Hi Lucas, can you show the error message and the respective code implementation? I was struggling with the pythonic resources as well, maybe I am able to help you
l

Lucas Santiago Cardoso

05/25/2023, 1:41 PM
After some adjustments, I'm getting this error:
Untitled.cpp
d

David Weber

05/25/2023, 1:52 PM
How does your
aws_s3_per_file_resource
function look like?
l

Lucas Santiago Cardoso

05/25/2023, 1:52 PM
image.png
d

David Weber

05/25/2023, 1:55 PM
Hm so it looks like that dagster is unhappy with you specifying the
s3_resource
argument with the type
S3Resource
It must be any of the type configs that the error message is suggesting.
Try the things in the last sentence: ResourceDependency[Union]
as a type
l

Lucas Santiago Cardoso

05/25/2023, 2:04 PM
It worked!!! Thank you so much @David Weber
:rainbow-daggy: 1