Simon
04/14/2022, 3:05 PMsandy
04/14/2022, 6:50 PMasset_sensor
decorator currently doesn't support partitions, but here's an example of of a custom partition-based asset sensor: https://github.com/dagster-io/dagster/discussions/7306.
The example uses software-defined assets, but the sensor implementation in it doesn't depend on software-defined assets to function.
I'm curious to hear when you'd want your asset sensor to trigger, with respect to partitions.Simon
04/14/2022, 9:30 PMSimon
04/14/2022, 9:31 PMDaniel Gafni
07/10/2022, 9:34 PMsandy
07/10/2022, 11:41 PMDaniel Gafni
07/11/2022, 5:53 AMsandy
07/11/2022, 5:19 PMIs the snippet above still the recommended way to check if a partition has been materialized?It is (though I just made a small edit). Let me know if it doesn't work for you.
I don't want to make a complete job for data materialization + model inference since it will unnecessarily trigger the data part. As I understand, I need to make 2 separate jobs (actually even 3 with another one being model training) and bind them together with a sensor.Exactly
Daniel Gafni
07/18/2022, 12:43 PMPartitionsMapping
inside sensors. I'm not sure how to achieve this with the example above since the AssetKey
doesn't have this information. What would be a good way to do this?sandy
07/18/2022, 3:37 PMAssetsDefinition
object, you can call its get_partition_mapping
method: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/definitions/assets.py#L491. Would that work for you?Daniel Gafni
07/18/2022, 3:37 PM