In general I feel dagster features the concept of

asset

but it’s strange schedules do not support native asset but we have to convert them to jobs. Sensors recently added support for asset, shall we do the same for schedules?

I've seen a lot of our users start doing:

from dagster import define_asset_job, ScheduleDefinition, AssetSelection

schedule = ScheduleDefinition(
   name = "run_asset_group",
   job = define_asset_job(AssetSelection.groups("my_group"),
   cron_schedule = "..."
)

I don't love the asset selection thing, but it does make it pretty lightweight to spin up a custom asset schedule

hey @William - I recently filed an issue to track adding this: https://github.com/dagster-io/dagster/issues/11055

I don't know how to use it 😂

Hmm good point, lists are poorly documented. I'll fix that. Here's how you do it:

AssetSelection.assets(*my_assets_list)

if i have two assets:

asset_a_key = AssetKey(["snowflake", "db", "schema", "table_a"])
asset_b_key = AssetKey(["snowflake", "db", "schema", "table_b"])

how do i select those two assets?

fwiw @Stephen Bailey my experience matches yours. Trying to select assets (esp with schemas as prefixes) is always a PITA I am loathe to do often @sandy when I use dbt to load assets I don't have a python object to reference, so I have to do weird things like

AssetSelection.keys([AssetKey["schema", "dbt_model"],AssetKey["schema", "dbt_model2"]) | AssetSelection.assets(python_asset_object)

And I never get it right the first, second, or third time. And I never remember if keys or assets is the string or the object. And whether or not the arguments are a list or a list of lists. The IDE tries to help but the type hint is basically "thing coercible to keys" and IDK what thing or coerce means necessarily in this context The

upstream

,

downstream

stuff is cool

@Sean Lopp - anywhere in particular you would expect to see more docs for this?

Maybe worth its own section in the assets concept page? BTW Stephen your use case could be:

asset_a_key = AssetKey(["snowflake", "db", "schema", "table_a"])
asset_b_key = AssetKey(["snowflake", "db", "schema", "table_b"])

AssetSelection.assets(asset_a_key, asset_b_key) 

#OR 

AssetSelection.assets(asset_a_key) |
AssetSelection.assets(asset_a_key)

Note that in this context

&

is restrictive

|

or is additive You could also probably do some invocation of

AssetSelection.keys

though I'd have to play around to see if I could figure out whether keys is a list of AssetKeys or a list of lists or something else

cc @sandy

hmmmmm. I would expect

AssetSelection.keys(*asset_key_list)

to work is it possible that one of your assets used to be partitioned, but is no longer partitioned? or vice versa?

hmm, that's possible. I was playing around with specifying different assets, but using the same sensor and just reloading it after changes. One of those was a partitioned asset. Are you thinking that the cursor may have become partitioned, then when it tried to submit new runs, it was parsing it incorrectly?

Exactly (though still a case we should handle more gracefully)

Ok, well, in that case I'll retract my asset selection complaint... but is there a way for me to test that it's working as expected? Something like

def test_selection():
    selection = AssetSelection.group("my_group_with_four_assets")

    assert selection.asset_count == 4

One option would be

from my_project import my_repo

def test_selection():
    selection = AssetSelection.groups("my_group")
    
    selected_keys = selection.resolve(my_repo.asset_graph)
    assert len(selected_keys) == 4

(Sorry to keep derailing the original question.) I'm also frequently having troubles finding the right way to select assets because I'm confused between all the different ways to call them, definitions, keys, asset groups, key prefix... I think it needs some simplification work. I also don't like using hard coded string based relations like

AssetSelection.groups("my_group")

because strings are hard to refactor automatically and if I make a typo, my IDE cannot detect, at coding time, that the object doesn't actually exist. So instead, I try to use things like:

asset_keys=[asset.key for asset in my_assets]

Similarly, I try to avoid those hard coded op/asset names in my job config, so it doesn't break if I decide to change my op/asset names. So I try to use things like:

my_op.name: { "config": {...}}

This is useful feedback. FYI, instead of doing

AssetSelection.keys(*[asset.key for asset in my_assets])

, you can do

AssetSelection.assets(*my_assets)

.

This is a good example of the difficulty of discovering the best way to select assets. Maybe we could have only one parameter

asset_selection

instead of both

asset_selection

and

asset_keys

, where we can pass anything that clearly points to assets: AssetSelection, a list of Assets, a list of AssetKeys, or just strings, and then Dagster handles the interpretation of it.

Maybe we could have only one parameter

asset_selection

instead of both

asset_selection

and

asset_keys

, where we can pass anything that clearly points to assets: AssetSelection, a list of Assets, a list of AssetKeys, or just strings, and then Dagster handles the interpretation of it

Are there particular functions / classes that you're talking for this?

define_asset_job

?

My most recent use case is selecting the assets in an

@multi_asset_sensor()

, but my point is general for all signatures where assets need to be selected.

got it - yes, the

@multi_asset_sensor

params especially are a bit of a mess. I filed an issue for addressing this: https://github.com/dagster-io/dagster/issues/11558

Thank you for the tickets! Continuing on the idea of removing abstractions, instead of having to call a specific class

AssetSelection

like you told me:

@multi_asset_sensor(
    asset_selection=AssetSelection.assets(*my_asset_sequence),
    job=run_assets,
)

I would expect to be able to just pass my sequence of assets:

@multi_asset_sensor(
    asset_selection=my_asset_sequence,
    job=run_assets,
)

Why does my sequence of assets need to go through another abstraction for a sensor to understand it ? If the sensor needs a specific attribute of the asset, it's its job to look for it.

The abstraction is useful because other functions use it (

define_asset_job

), and it does allow the mixing and matching selecting via

groups

and

assets

, which could be really useful for more complex cases. (It could also be useful, for example, to have tag matching in a future version.)

define_asset_job

could also allow to directly use a sequence of assets. Ok for having an additional abstraction if one requires additional complexity like mixing groups and keys in the selection. But I think the syntax for the default usage of providing a sequence of assets could be easier.

here's a proposed change for multi_asset_sensor: https://github.com/dagster-io/dagster/pull/11567