The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

*Weekly Release Highlights: 1.0.11* :sparkles:

:new: Assets can be loaded outside of a run, e.g. in notebooks
:handshake: Snowflake IO Manager now works with dbt assets
:linked_paperclips: Python dependency: latest version of Alembic is supported

<@UULA0R2LV> can you explain `Assets can be loaded outside of a run, e.g. in notebooks` in more detail or link to a description?

here's the documentation: <https://docs.dagster.io/master/concepts/assets/software-defined-assets#loading-asset-values-outside-of-dagster-runs>

Interesting - but where would Jupyter get the repository definition from?

you would need your Dagster python code to be installed in the same python environment. is that difficult in your setup?

I think this should be doable - I will have to experiment with this.

<@U011CET83FG> how to handle the types if the associated IO manager can work with i.e. pandas and spark dataframes as the output? How can the desired type of the loaded asset be specified in:
```repo.load_asset_value(AssetKey("asset1"))```


In any case I get: `Attempting to access asset_key, but it was not provided when constructing the OutputContext`  when trying to execute `repo.load_asset_value(AssetKey(["prefix", "key"]))`

<https://dagster.slack.com/archives/C01U5LFUZJS/p1665073045992489> might be related

here's a PR that should fix this issue: <https://github.com/dagster-io/dagster/pull/9914>

&gt; How can the desired type of the loaded asset be specified
`load_asset_value` accepts a `python_type` argument that allows you to specify that type that you want to load into. details here: <https://docs.dagster.io/_apidocs/repositories#dagster.RepositoryDefinition.load_asset_value>

When I pass a spark dataframe as the type: `DagsterInvariantViolationError: Attempting to access op_def, but it was not provided when constructing the InputContext`  is the error message I am greeted with.

This is due to: `meta_config = context.op_def.output_defs[0].metadata` trying to access the metadata which is part of the assets definition.

how can an `op_def` be provided to this input context?

hey <@U01PP2ED4ES> - we can include the `op_def` on there in a future change, but why not just use `context.metadata`?

probably because in an old version of dagster this was mandatory - it is certainly much nicer to use context.metadata. I somehow overlooked this change. Let me try this