I have an ` asset` that queries an endpoint when materializi dagster #ask-community

I have an `@asset` that queries an endpoint when m...

Philippe Laflamme

01/12/2023, 1:17 PM

I have an

@asset

that queries an endpoint when materializing. This endpoint has a concurrency limit of

, which means I need to only materialize

assets concurrently. I’ve setup the

QueuedRunCoordinator

with

tag_concurrency_limits

, say

http/limit: n

which limits to

. How do I apply this tag to my

@asset

? I’ve tried using

op_tags

but that doesn’t seem to apply it when materializing; and for backfilling, I’ve found that: • in the simple popup in the UI, I can specify a range, but I cannot specify this tag • in the “shift-click” popup, I cannot specify a range, but can specify the tag • on the command line, I can do both, i.e.:

--from x --to y --tags n

But at the end of the day, I always want this tag to be applied when materializing this asset. Is this possible?

Philippe Laflamme

01/12/2023, 4:23 PM

Digging into this, I discovered that what I needed was a

Job

with the necessary tags on that, i.e.:

define_asset_job("the_job", "the_asset", {"http/limit": "n"})

That then allows me to backfill the asset by launching runs of this new job. That said, the

@asset

still allows “ad-hoc materialization” which means that those could conflict with the ones running through the

Job

. It’d be nice to have more control over this “ad-hoc” job, either disabling it or perhaps pointing at a

Job

definition.

sandy

01/12/2023, 9:59 PM

Hey @Philippe Laflamme - what you've discovered is the current best way to do this. However, I agree with you that it's incomplete. Here's an issue where we're tracking adding this capability: https://github.com/dagster-io/dagster/issues/8979

🙏 1

12 Views

Open in Slack

Previous Next