Hi All, I'm just starting diving into dagster and hoping to use it to refactor some of our (simple) data pipelines at for my place of work. I'm trying to create a reusable solid that would connect to our database and execute a sql query which would return the data in the form of a pandas dataframe. However, I'm receiving an error DagsterInvalidDefinitionError which states "Must pass the output from previous solid invocations or inputs to the composition function as inputs when invoking solids during composition.". Is it possible to pass just a regular data type (e.g string, int, etc) as a parameter to a solid instead of having it be the return type of another solid ?

Here is the code I have thus far for reference

When using

dagster

on command-line, how do I enable the console logging to print the same error detail message that Dagit shows for a step that failed? I set

log_level

for console logging to

DEBUG

but it has no effect.

should `Dict`s be a valid dagster type in a Field config? I'm trying to create a

Field

with a Dict type, but getting

DagsterInvalidDefinitionError

. tried both the dagster Dict (

from dagster import Dict

) and the typing Dict. the error message seems to imply Dicts and NamedDicts are valid, though

Hi everyone, I have lots of repeating params in config, can anyone help me refactor? Don't understand how to share a parameter between different aliases of solids. Code in thread

Is it possible to group pipeline definitions together into a single pipeline or is it good practice to use composite solids?

So, enabling

dagster-aws

to work with in accounts without a default VPC is something I can spend some time on. Skimming the code for the

cli

package, I think it would make sense to add the ability to use a given vpc/security group pair. Any advice or warning for pitfalls from anyone who may have already thought this through more than I have?

Hi all Just started investigating Dagster last week and trying to figure out a couple best practices for challenges I was hoping to solve. First, is there a beat practice for dynamically generating instances of solids based on the output of an upstream solid? Use case is wanting to be able to map a common (for us) pattern over a dynamic number of outputs. Would prefer not to have to update a config file for each run. Second, is there a good way to make Dagster Date aware with dagster-cron? Basically want to run on a schedule, but have the current date as a parameter that gets passed in. Realize this is encroaching heavily on airflow territory. Maybe this is a situation where airflow is actually the best tool for the job. Curious to get feedback/opinions.

Hi, I'm curious if contributions to dagster-pandas would be helpful. I use pandas extensively in my application and fleshed out dagster-pandas to support Series, Index, etc.

Yes, definitely!

Oh, awesome. Okay, I'll put together a pr request

alangenfeld just published a new version: 0.6.4.

it's not a great error message, but try

return

Hi all, I am new here and wondering what is the difference between Dagster and Prefect (https://docs.prefect.io/)?

hey folks - would love to chat with some folks that have done any serious work on dagster. can't find any real world blogs/examples

Is there a nice way to use enums (as strings) in configs? I.e. I'd like to expose several options to the user via a config, e.g. picking one of many algorithms. Those could be identified by a string that is then used for a lookup in a dict. There are two downsides: - Errors (i.e. wrong spelling, option does not exist) are only caught at run time when the actual lookup happens - There's no (easy?) way of showing the options in the nice config editor in dagit

Hey everyone! I started looking at Dagster recently and fell in love with its amazing UI. Part of that feeling comes from the fact that we're currently using Luigi at my place of work and its UI is too minimalistic. However, I didn't find any comparisons of Dagster vs. Luigi over the internet so asking here. My question is: In Luigi, Targets -- data artifacts produced by Tasks (pipelines), are first-class citizens and each time you execute a pipeline, Luigi unfolds all its dependencies and for each of them decides whether it should be executed or not based on presence of output targets (a task is considered as completed if all its output targets exist). How does Dagster fit into this picture? If I have a parametrized dagster pipeline and some of its solids have already been executed and produced an output (in my case, it's hive partition in s3 written by pyspark), will dagster skip corresponding subgraphs of these finished solids and execute only what's needed? Thanks in advance, will be happy to hear any inputs on this!

HI Team I have a question. Can Solids in Dagster have conditions loops. I am working on a project where there might be condtional loops.It might not strictly follow DAG concept.Can anyone please specify and give some information about DAG flexibility and behaviour in Dagster.It would be helpful for my research

Hey everyone,

I have a simple question we currently are operating our airflow cluster on kubernetes using the kubernetes executor would it be possible to use dagster to generate our DAG ? Are their any equivalent in Dagster to the KubernetesPodOperator ?

Hi, Noticed that the example for dagster-pandas is dependent on dagstermill - you may want to consider to split the examples

Hi, We are running dagit in an offline environment. we have noticed that the browser is unresponsive after running an execution plan. (I cannot see any errors). Does anyone have any ideas about this issue?

Hi, I was wondering if anybody is using Dagster with an instance of airflow running on a docker container ? I am trying to do the https://dagster.readthedocs.io/en/stable/sections/learn/tutorial/intro_airflow.html tutorial but cannot find a way to make my DAG appear in the airflow UI . Any ideas ?

Hi All, I'm going through the tutorial and have a question. In the section "Parametrizing solids with config" it explains the difference between inputs and configs. The "don't do it like this it's bad practice" example it uses says not to hoist all the parameters of the to_csv function into the arguments of the solid. I completely agree that having to rewrite all the permutations of arguments for all the underlying function(s) calls would be a nightmare. However, the "proper" solution putting them into the config seems it's doing exactly that. Would it not be easier (specifically easier not necessarily better since I understand the point of defining the config options) to just have a parameter of **kwargs to be passed in and avoid having to redefine options in the functions being used ?

I was just listening to Nick on the software engineering podcast. https://podcasts.google.com/?feed=aHR0cHM6Ly9zb2Z0d2FyZWVuZ2luZWVyaW5nZGFpbHkuY29tL2ZlZWQvcG9kY2FzdC8&episode=aHR0cDovL3NvZnR3YXJlZW5naW5lZXJpbmdkYWlseS5jb20vP3A9ODI3Ng He acknowledged that the current iteration of Dagster is only a piece of the vision. Curious if there's any time of product outline for what a 1.0 release might look like?

Are there any other examples using the

Selector

type to limit input values in a config file? I'm trying to configure a solid with multiple parameters, but where one of the parameters is necessarily fixed to be one of a set of possible values

Hey everyone, searched this slack, couldn't find an answer somehow. How can I deploy dagit with some password on top (like jupyter does for example)? I created a pipeline and launch it from cli now, but want all this interactive things but don't like that anyone can go to ip and have access to it

Hi! I enjoyed the dagster podcasts. Is there an example of how one could use dagster as a way to model an ML pipeline down to featurization (say replacing sklearn or Dask ML pipelines)? I see ML pipelines as spanning from coarse (Airflow) to being fine grained (featurization or business logic pipeline). My interest in Dagster is that it provides a single abstraction for both coarse and fine grained pipelines, along with guaranteed execution, low latency execution (say the DAG is to be executed in a serving environment), and parameterization support.

Hello everyone, I am a user of dagster and I couldn't find a way to inject environment variables in my resources. Is there any documentation about this ? Thank you 🙂