https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
s

Stephen Bailey

01/05/2023, 8:28 PM
The Dagster AssetReconciliationSensor does this nifty thing where it yields a RunRequest but doesn't create a global job definition (AFAIK). Is it is simple as returning the following rather than a pre-defined job?
Copy code
return RunRequest(asset_selection=list(asset_keys))
🤖 1
o

owen

01/05/2023, 9:15 PM
yep that'll work! as long as those assets are possible to run within the same job (so for example, a list containing two assets, one hourly partitioned and the other daily partitioned would result in an error)
🌈 1
under the hood, dagster creates a "mega job" (or multiple mega jobs, in the case of different partitions definitions) when the repository is created
s

Stephen Bailey

01/05/2023, 9:38 PM
do i need to specify the mega job in the
@multi_asset_sensor(job=...)
syntax?
o

owen

01/05/2023, 9:46 PM
I believe you should be able to just leave the job out entirely
s

sandy

01/05/2023, 9:50 PM
I believe you should be able to just leave the job out entirely
it's worth trying, but I suspect this does not work the asset reconciliation sensor uses the
asset_selection
argument on
@sensor
to target a bunch of assets, and then the RunRequest subselects within that set.
@multi_asset_sensor
does not yet support that argument (confusingly, the
asset_selection
argument on
@multi_asset_sensor
refers to the assets to monitor). I am going to try to remedy this. as a workaround in the meantime, you could create a job that targets all your assets and point at it in the multi-asset sensor.
👍 2
o

owen

01/05/2023, 9:52 PM
ah that's right, good catch
s

Stephen Bailey

01/13/2023, 2:20 AM
been playing with this this week. two gotches to add -- 1.
RunRequest(asset_selection=..)
does not accept
AssetSelection
as an accepted type:
dagster._check.ParameterCheckError: Param "asset_selection" is not one of ['Sequence'].
2. The "mega job", which is best most easily defined as
define_asset_job(selection=AssetSeleciton.all())
, fails if there are different partitions. I can see the tehnical complexity here and I finially understand why there were those ASSET_JOB_0 - 5 showing up in the reconicilation sensor. But, a pain nonetheless