Alec Koumjian
03/01/2023, 4:33 PMdefs.load_asset_value
that can be accessed inside an op/job? I'm aware of the very recent implementation that lets you load an asset into an op using .to_source_asset()
, with the big caveat that it does not allow for partitioned assets. Secondly, it's a real bear to have to send in to the job all the resource_defs
, the config
that maps all of those resources, then in the ops
containing the assets you have to include required_resource_keys
and a noop ins
. Hopefully I'm wrong about all that.sandy
03/01/2023, 7:50 PMI'm aware of the very recent implementation that lets you load an asset into an op usingthis is fixed in our release that's landing today, with the big caveat that it does not allow for partitioned assets.to_source_asset()
sandy
03/01/2023, 7:51 PM@op
def my_op():
from .somewhere import defs
asset_value = defs.load_asset_value(...)
Alec Koumjian
03/01/2023, 7:52 PMAlec Koumjian
03/01/2023, 8:18 PMdagster._core.errors.DagsterInvariantViolationError: $DAGSTER_HOME "/opt/dagster/dagster_home" is not a directory or does not exist. Dagster requires this environment variable to be set to an existing directory in your filesystem
Hm, obviously there is some kind of context set when accessing this inside the op vs outside the dagster environment as a whole.sandy
03/01/2023, 8:30 PM@op
def my_op(context):
from .somewhere import defs
asset_value = defs.load_asset_value(..., instance=context.instance)
Rasmus Bonnevie
03/02/2023, 8:24 AMop->job->def
as an import chain, but then you also have def->op
now, creating a loop.
Maybe you make it work if you only use the op in another code location, but that seems restrictive.sandy
03/02/2023, 4:09 PMRasmus Bonnevie
03/03/2023, 4:55 AM