Hello, quick question about executing python scripts within Dagster. I have a python script that runs some logic, and want to integrate that script in my job / pipeline. I understand the basics of Software Defined Assets, but can anyone point me to some example code on how to load the python script into Dagster and execute it as an op?
dagster bot responded by community 1
09/11/2022, 7:15 PM
Where do you want to execute it (EG: Kubernetes, spark cluster, etc.)? The answer will probably vary a lot depending on that. Also, you may want to consider integrating your scripts into Dagster itself so you can have it all in one place. Dagsters resources and contexts make this very powerful.
09/11/2022, 8:23 PM
@Zach P I want to execute the script from within Dagster itself and have it all in one place. I would like to use resources and contexts as a means to execute my python script. Do you know how I can do that? (or point to documentation / examples that could guide me?)