https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
a

Alec Koumjian

03/01/2023, 4:33 PM
Would it be possible to get an imperative function like
defs.load_asset_value
that can be accessed inside an op/job? I'm aware of the very recent implementation that lets you load an asset into an op using
.to_source_asset()
, with the big caveat that it does not allow for partitioned assets. Secondly, it's a real bear to have to send in to the job all the
resource_defs
, the
config
that maps all of those resources, then in the
ops
containing the assets you have to include
required_resource_keys
and a noop
ins
. Hopefully I'm wrong about all that.
s

sandy

03/01/2023, 7:50 PM
I'm aware of the very recent implementation that lets you load an asset into an op using
.to_source_asset()
, with the big caveat that it does not allow for partitioned assets
this is fixed in our release that's landing today
👍 2
could you do something like?
Copy code
@op
def my_op():
    from .somewhere import defs
    asset_value = defs.load_asset_value(...)
a

Alec Koumjian

03/01/2023, 7:52 PM
I was going to look into doing just that but had a feeling it wouldn't "just work". I'll give it a try, thank you
Copy code
dagster._core.errors.DagsterInvariantViolationError: $DAGSTER_HOME "/opt/dagster/dagster_home" is not a directory or does not exist. Dagster requires this environment variable to be set to an existing directory in your filesystem
Hm, obviously there is some kind of context set when accessing this inside the op vs outside the dagster environment as a whole.
s

sandy

03/01/2023, 8:30 PM
Try this:
Copy code
@op
def my_op(context):
    from .somewhere import defs
    asset_value = defs.load_asset_value(..., instance=context.instance)
r

Rasmus Bonnevie

03/02/2023, 8:24 AM
isn't this likely to cause a circular dependency? If you want to use the above op in a job, you have
op->job->def
as an import chain, but then you also have
def->op
now, creating a loop. Maybe you make it work if you only use the op in another code location, but that seems restrictive.
1
s

sandy

03/02/2023, 4:09 PM
that's why you need to do the import inside the body of the op instead of at module-level
r

Rasmus Bonnevie

03/03/2023, 4:55 AM
ah, fair, but I think non-top imports are not exactly best practice and I'd personally be uncomfortable relying on a solution like this one, but maybe I'm being irrational 🙂
2 Views