Philippe Laflamme
01/12/2023, 1:17 PM@asset
that queries an endpoint when materializing. This endpoint has a concurrency limit of n
, which means I need to only materialize n
assets concurrently. I’ve setup the QueuedRunCoordinator
with tag_concurrency_limits
, say http/limit: n
which limits to n
. How do I apply this tag to my @asset
? I’ve tried using op_tags
but that doesn’t seem to apply it when materializing; and for backfilling, I’ve found that:
• in the simple popup in the UI, I can specify a range, but I cannot specify this tag
• in the “shift-click” popup, I cannot specify a range, but can specify the tag
• on the command line, I can do both, i.e.: --from x --to y --tags n
But at the end of the day, I always want this tag to be applied when materializing this asset. Is this possible?Job
with the necessary tags on that, i.e.: define_asset_job("the_job", "the_asset", {"http/limit": "n"})
That then allows me to backfill the asset by launching runs of this new job.
That said, the @asset
still allows “ad-hoc materialization” which means that those could conflict with the ones running through the Job
. It’d be nice to have more control over this “ad-hoc” job, either disabling it or perhaps pointing at a Job
definition.sandy
01/12/2023, 9:59 PM