I have another silly question. I am a bit confused about ops,jobs,assets,pipelines and solids. My understanding though is that assets have ops running under the hood anyway and offer more functionalities. Jobs are used in combination with schedules to run materialise the assets when needed (not sure how should I decide whether to use pipelines or jobs).
So far I have structured my pipeline as assets with dependencies and then there is a job and a schedule to materialise them.
Now, If I want to change a date filter based on the scheduled execution time, what would the best approach be? I am considering splitting the "trasform" asset into 3 "parallel" assets (with same dependencies) and then label them differently (maybe using groups?) so that the schedule knows what to run at what time (or maybe setting up more than 1 schedule).
But maybe it could be done using different configurations?
How would you tackle this?
Attached is how my pipeline looks at the moment with the highlighted asset that I would like to behave differently based on the schedule:
03/24/2023, 4:44 PM
hey @Dario De Stefano to start with, pipeline and solid are deprecated. pipelines are now jobs and solids are now ops.
you are correct that if you want to run a set of assets on a schedule you can make a job from those assets and have the schedule run that job.
as for whether to split up the transform asset, I think I would lean toward using setting different configuration for the job based on the time of the schedule and then having the transform asset respond according to the configuration it receives
Dario De Stefano
03/24/2023, 4:45 PM
Thank you so much I will look into how to achieve that!
@jamie These two schedules are identical right? Since I did not specify anything in RunRequest.