https://dagster.io/ logo
Title
p

peay

09/27/2022, 9:33 AM
Hello, is there a way to automatically add a key prefix to all assets in a given repository and repository location, without having to add it to each individual asset? Say, for all assets in
repoA
at repository location
location
I'd like to pre-pend
location
,
repoA
to
prefix_list
. I use different repository locations for different branches of my code (e.g., one repository location for the user code deployment for
develop
, one repository location for the user code deployment for
feature-1234
, etc.), and it'd be great to allow these to coexist easily. I've seen utilities like prefix_assets but I am wondering whether there's a way to do this automatically for all repositories, taking the location into account as well.
j

Joseph McCartin

09/27/2022, 10:19 AM
I think I do this for my pipelines. I take the
__file__
property of the module where the asset is defined, and feed it to a function that returns the path, formatted with underscores instead of slashes. I do this for the
asset_name
field in the asset decorator, but you could do this to the prefix list too
p

peay

09/27/2022, 11:42 AM
This would work fine for the repo name part, but I am especially interested in getting the repository location name, which is what is set in
workspace.yaml
and is only known when Dagster loads a (possibly remote) user code deployment. Ideally, if there was a way in
workspace.yaml
to be able to set an option along those lines:
load_from:
- grpc_server:
    location_name: "locationA"
    prefix_asset_keys_with_location: true
    ...
that would prefix all asset key with
locationA
j

Joseph McCartin

09/27/2022, 12:10 PM
Hmm, if that workspace file exists in the deployment, it must be possible to read it from the path in your own code. Then you could use it as an input to the function i mentioned above.
c

claire

09/27/2022, 4:40 PM
Yeah, there isn't an automatic way to do this at the moment. Joseph's approach above of reading the workspace file to fetch the location name should work
👍 1