https://dagster.io/ logo
Title
e

Eduardo Santizo

04/30/2021, 7:28 PM
Hey! Is there a way to use a resource inside a pipeline itself? Something like this:
@pipeline(
   mode_defs = [basic_mode]
)
def test_pipeline(context, input1, input2):
   resources.basic_mode.run()
d

daniel

04/30/2021, 7:31 PM
Hey Eduardo - this isn't possible because the pipeline doesn't actually perform any execution - it just describes the structure of the graph made up of solids. Any actual execution happens inside a solid.
e

Eduardo Santizo

04/30/2021, 7:37 PM
Thanks. And inside a solid factory?
d

daniel

04/30/2021, 7:41 PM
You can use a resource inside the @solid function inside a solid factory, but since a solid factory is just a function that returns solids and isn't something that dagster handles in any special way, using a resource outside of the @solid function would have the same problem as using it inside a pipeline
👍 1
e

Eduardo Santizo

05/03/2021, 5:55 PM
Another question: I'm creating a solid factory that generates a group of "downloader solids" based on the contents of a JSON file. Each downloader solid has a different URL to download, and I define this URL programmatically as part of a loop during the solid factory execution. The solids generate exactly as expected, but during their execution, they all use the same URL. I asume this is because only the structure of a pipeline is defined when Dagster loads, and not necessarily its contents. Is this right?
d

daniel

05/03/2021, 5:57 PM
Would you mind making a new post for this? That'll ensure the right people on the core team see the question
e

Eduardo Santizo

05/03/2021, 6:01 PM
Ok, thank you Daniel