think I may have asked this before but can't find the post here. Is there an example dagster repo that we can use for guidance on project structure best practices? similar to what Jaffle Shop is for dbt: https://github.com/fishtown-analytics/jaffle_shop

I'm looking to create a solid that is just an abstraction of another solid (adds a little bit of application-specific logic to bash command solid). Is the best way to do this to use composite solids? It feels like composite solids are really meant to be used to abstract sub-DAGs instead of single solids

Hi All, I'm looking to create a sql_solid similar to the one in the air demo. Essentially just a solid that executes arbitrary sql statements. The tutorial does a great job explaining how I can have an input parameter and specify that in the config yaml file. However, further into the tutorial it goes over hydrating inputs. My understanding is I could make a custom data type, say for example, SqlDataFrame that uses a sqlalchemy engine to execute a sql statement (e.g "select * from datawarehouse_table") and wraps a pandas DataFrame. With this couldn't I avoid having to create a "sql_solid" and instead just have a parameter of a solid of type SqlDataFrame ?

Can someone help clarify the new Repository thing ? I'm getting very confused comparing the docs against the demos in the git repo. Throughout the beginning of the documentation it uses the regular "@pipeline" decorator with no parameters and can run dagit to view pipelines and yaml config files like normal. However, when comparing to the demo files the pipelines have "mode_defs" and "preset_defs" predefined (also noticed they have names and other properties like resources and config). When defining a repo I would expect to be able to simply import and list pipelines just like the pipeline_defs in the pandas_hello_world demo. However, when dagit is run it's throwing errors saying the pipeline doesn't have a "name" property set when only using the plain "@pipeline" decorator like the documentation. This is strange since viewing the definition of the "@pipeline" decorator it states the "name" parameter is optional ? Also, what's the purpose of the lambas in the "pipeline_dict" dictionaries for the bay_bikes and airline demos ? I understand what intent for Repositories and pipeline modes is for. However, during these early stages with simple tests I'm trying to avoid having to define local vs test vs prod pipeline modes. Perhaps a little clarification on which source to believe to be able to run dagit would be helpful.

Is it possible to reuse a single resource more than once in a pipeline with an alias or something ? For instance, the DbInfo resource in the examples works great when specifying a single database to connect to because the solids that use the DbInfo resource have lines that execute like

context.db_info.engine.execute(...)

. But how would you specify multiple of a same resource if one pipeline needed to connect to a mysql and postgres server ? Something along the lines of:

resources:
  db_info_mysql:
    config:
      username: mysqluser
      ...
  db_info_postgres:
    config:
      username: pguser
      ...

Max Gasner just published a new version: 0.6.8.

Hi Team I am getting error while installing dagster for gevent package .Can anybody from the team please help

I am using Python 3.8.1

prha just published a new version: 0.6.9.

where does dagster surface invocation metadata (run ID, etc.)? Is it attached to

context

?

@dwall context.pipeline_run gives you a

PipelineRun

what's the currently recommended way of handling secrets in YAML config

Are there any near-term plans for caching?

following up on this a little bit, I think the consensus here was to use the

Nothing

input/output type to model dependencies between solids that don't rely on input/output. This seems to work well most of the time, but I'm getting stuck on how to leverage this for reusable solids that have a dynamic amount of upstream and/or downstream dependencies. An example of this is a

dbt_run

solid we have. This

dbt_run

solid can theoretically be dependent on 1 to n number of upstream solids. Currently, it feels like the only way to model this in Dagster is to create the solid with n number of

Nothing

input definitions. This doesn't feel intuitive to me so I'm wondering if there is something I'm missing here.

I know I've asked this before, but I'm not sure we landed on something definitive. Is there an established pattern for creating solids that wrap other solids?

I know I've seen the Dagster team use factory functions for this, but is it possible just for a solid to be an abstraction of another solid?

planning any support to inject env vars into yaml?

getting a PyObjC error when trying to install dagit on linux. Have y'all run into this?

huh! can we see a full traceback?

Hello! The

README.md

in dagster-ge says:

This library is not meant for current use, as it is built on deprecated abstractions. It will be rebuilt.

Is this accurate? It looks like things are being updated. Should I not be trying to use great expectations yet?

Hello All, I'm trying to configure dagster in Google Cloud together with distributed dask, but unfortunately it don't want to work without S3 storage that a little bit strange. As I noticed this storage hardcoded in dagster-dask, but why? There is dagster-gcp that can provides GCS, so maybe possible add it to gaster-dask (or even better make som general requirements)?

Latin modern fonts for Latext: https://www.ctan.org/tex-archive/fonts/lm/

Hi all, I'm getting an error while installing the requirements.txt for my dagster project. everything works great on windows, cloned the repo from gitlab to the linux server and can't install the dependency for pywin32 on linux (because why would you?). The python package "pipdeptree" shows it as a dependency for dagit. This can't be windows only right ?

Hi all. I'm trying to understand how, through the python api, I can execute a subset of pipeline solids. I can use execute_solids_within_pipeline(), but this returns a list of SolidExecutionResults. Ideally, what I want is a PipelineExecutionResult, which has a run_id and a context I can inspect. Is there a way to do this?

Hi all! New user here. I'm really liking the philosophy and documentation of the framework and am trying it out on a client project. I have some n00b questions: Looking at the last couple of weeks of chat history, it sounds like lots of folks are making feature engineering / ML pipelines (as am I) and trying to get intermediate memoization to fit their use case. I find the current UI for this somewhat inconvenient since my source data does not updating frequently and I'm not running jobs interactively or though dagit; re-using intermediates across jobs and configs seems to fit my use case better. If I understand correctly this is possible, but requires specifying a particular job run intermediate output. Someone suggested splitting the pipeline into parts, but this is also inconvenient since experimentation on the feature engineering parts of the pipeline generate lots of different possible sub-pipelines, so this ends up being close to making a pipeline per solid. The best options I could think of are either: a. Build caching logic into solids; this seems unfortunate since most of the work for what I want is already built in the intermediates framework b. Hack IntermediateStore paths to: (i) not incorporate job_id, (ii) have versioned solids (to invalidate caches when logic changes), (iii) pass along upstream solid dependencies (and their versions) to incorporate into the intermediate path key (so upstream version changes will bump the downstream path) Before going further down either path, since this seems like a pretty common use case, I wanted to check if there are other better options (or if either of these is a bad idea).

Who's paying for slack?

can a

ScheduleDefinition

use a pipeline preset?

we are consistently getting the following error when running a pipeline in a docker container. Have y'all seen something like this before?:

An exception was thrown during execution that is likely a framework error, rather than an error in user code.
Original error message: Exception: Pipeline execution process for f8960a24-22b1-4149-96d7-806be3f228d2 unexpectedly exited

Stack Trace:
  File "/home/docker/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.8/site-packages/dagster_graphql/implementation/pipeline_execution_manager.py", line 212, in _generate_synthetic_error_from_crash
    raise Exception(