The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

In general, is there any imperative instruction on how to use Libraries? The docs list them, but I wasn’t able to find a working implementation tutorial.

Hi Jan!

Dagster has quite a number of pluggable components, so each library is potentially invoked in a different way, depending on what functionality you’re trying to achieve.  Are there some libraries that you’re particularly interested in where the specific documentation could be improved?

Hi <@UM49TQ8EB>, I’m not quite thinking of what you’re describing, I think.
And I gotta say, possibly this is just me not having studied the docs deeply enough.
So I’m not looking for functional documentation of these components, they often already exist as these are standalone projects, right? What I ran into (and from studying the Slack archives some people also experienced this, for others it seemed to work automagically) is how to properly operationalize the modules in Dagster so they can be used in a file.

Basically I ran into
1. pip install component, i.e. dagster-airbyte: Check
2. ?????
3. use the component as `from dagster_airbyte import foo`
4. `Module not found` error when trying `dagit -f airbyte_foo.py`
So then it turns out files must not be in the same dir as the virtual env holding the python installed component for modules to be parsed properly. This, to me, was just not transparent.

<@UM49TQ8EB> to give you a more specific example: I just cloned the dagster repo and went into the `examples/hacker_news` directory.
It has a high level `setup.py` as well as a `workspace.yaml` .

So again, my steps are
1. create venv, run `pip install -e .` , setup.py gets executed.
2. run `dagit`
3. Error: `Module not found: dagster_dbt` 
So again, dagster seems not to be able to parse modules correctly despite a present setup.py containing dagster-dbt as well as a workspace.yaml and no further context as to what’s the rootcause of this mistake is provided.