Hi all, I am still learning and this may not be possible or may be super simple but I am trying to understand how to properly link two solids where the internal solid uses a

Selector

. Because this solid is an internal to the dag (non a start node) I am not able to (or just don't know how to) pass in the solid config because the value is outputted by the previous solid. I have downloaded a file from s3 and the output is file path string to a csv file. Once I have this, I would then like to use the example

sum_solid

solid used in the airline_demo code that takes in this file path and using the dagster_pandas.Dataframe type. I was looking at the source code and see the Dataframe type uses a

Selector

to determine how to handle the data. Is there a way to properly do this?

newbie question: When would one use a

@lambda_solid

vs a

@solid

From the documentation of `lambda_solid`:

This shortcut allows the creation of simple solids that do not require configuration and whose implementations do not require a context.

So tl;dr: if you don't need a context or config fields,

lambda_solid

it is

Thanks Kevin!

Are there

retry

semantics in Dagster? I can't find anything in the docs referencing that. Is it down to the author of the solid to code that?

prha just published a new version: 0.5.7.

Hi Dagster! Quick Q. Is it possible to specify the config in environment_dict to pass to a resource, based on the current Mode?

Hi everyone, I ran into an error that I am having trouble understanding: The error occurs when I yield an OutputDefinition with a dagster_type of DataFrame (from dagster_pandas) from a solid. For example, when executing the following solid:

@solid(output_defs=[OutputDefinition(name='df',dagster_type=DataFrame)])
def get_df(context):
  yield OutputDefinition(pd.DataFrame([1, 2, 3]), 'df')

The following error occurs:``` ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). File "/Users/noahtrueblood/.local/share/virtualenvs/dappr-Fw8AAVno/lib/python3.7/site-packages/dagster/core/errors.py", line 104, in user_code_error_boundary yield File "/Users/noahtrueblood/.local/share/virtualenvs/dappr-Fw8AAVno/lib/python3.7/site-packages/dagster/core/engine/engine_inprocess.py", line 568, in _user_event_sequence_for_step_compute_fn for event in gen: File "/Users/noahtrueblood/.local/share/virtualenvs/dappr-Fw8AAVno/lib/python3.7/site-packages/dagster/core/execution/plan/compute.py", line 75, in _execute_core_compute for step_output in _yield_compute_results(compute_context, inputs, compute_fn): File "/Users/noahtrueblood/.local/share/virtualenvs/dappr-Fw8AAVno/lib/python3.7/site-packages/dagster/core/execution/plan/compute.py", line 52, in _yield_compute_results for event in user_event_sequence: File "/Users/noahtrueblood/.local/share/virtualenvs/dappr-Fw8AAVno/lib/python3.7/site-packages/dagster/core/definitions/decorators.py", line 343, in compute for item in result: File "/Users/noahtrueblood/dappr/data/transform/derive.py", line 57, in get_df yield OutputDefinition(pd.DataFrame([1, 2, 3]), 'df') File "/Users/noahtrueblood/.local/share/virtualenvs/dappr-Fw8AAVno/lib/python3.7/site-packages/dagster/core/definitions/output.py", line 31, in init self._runtime_type = check.inst(resolve_to_runtime_type(dagster_type), RuntimeType) File "/Users/noahtrueblood/.local/share/virtualenvs/dappr-Fw8AAVno/lib/python3.7/site-packages/dagster/core/types/runtime.py", line 538, in resolve_to_runtime_type dagster_type = remap_python_type(dagster_type) File "/Users/noahtrueblood/.local/share/virtualenvs/dappr-Fw8AAVno/lib/python3.7/site-packages/dagster/core/types/mapping.py", line 13, in remap_python_type if type_annotation == int: File "/Users/noahtrueblood/.local/share/virtualenvs/dappr-Fw8AAVno/lib/python3.7/site-packages/pandas/core/generic.py", line 1478, in nonzero .format(self.class.name))

I see in `dagster/python_modules/dagster/dagster/core/types/mapping.py` that an equality is performed. For example: `type_annotation == int`. Which makes me think that type_annotation is of type pd.DataFrame instead of type DataFrame. Any ideas?
Interestingly, the error does not occur when I use a return instead of a yield:

@solid(output_defs=[OutputDefinition(name='df',dagster_type=DataFrame)]) def get_df(context): return pd.DataFrame([1, 2, 3]) ```

would love to hear more about this!!! https://getdbt.slack.com/archives/C0VLZM3U2/p1566588090016400?thread_ts=1566558444.011800&cid=C0VLZM3U2

Hi again Dagsters! Another question for you. Is it possible to programmatically instantiate solids - for example, if

solid_a

outputs a list of

n

files, and we want to create

n

aliases of solid_b each with a file as input, can we do this?

Hi all, I trying to create a custom type that uses the Union type so that I can accept a mixed list with two types. I get an error when trying to create a custom dagster type and am not sure if this is the right approach. Any pointers or suggestions?https://docs.python.org/3/library/typing.html#typing.Union

Hi Team, My name is Achin .I am new to Dagster and have started using it for workflow implemenation.I tested normal hello_world program.The CLI response was success but for Dagit GUI I am getting an error AttributeError: module 'os' has no attribute 'setsid' I am using command dagit -f hello_world.py -n hello_world_pipeline

Can anyone help me to understand where I am going wrong

Issue is solved I used dagit-cli instead

Hi Team, I am try to add data validations and creating user defined data type.I want to perform a check if the file read is csv from one node and the next node also receives csv . I am getting error dagster.core.errors.DagsterInvalidDefinitionError: Input "pdd" in solid "Input1" is not connected to any outputs and can not be hydrated from configuration, creating an impossible to execute pipeline Sample code from dagster import execute_pipeline, pipeline, solid,as_dagster_type,lambda_solid,input_hydration_config import pandas as pd

DataFrame = as_dagster_type(

pd.DataFrame,

name='PandasDataFrame',

description='''Two-dimensional size-mutable, potentially heterogeneous

tabular data structure with labeled axes (rows and columns).

See <http://pandas.pydata.org/>''',

)

@lambda_solid

def Input1(pdd: DataFrame) -> DataFrame:

r = pdd.read_csv('file1.csv')

return r

@lambda_solid

def Merge(r: DataFrame,r2: DataFrame,pdd: DataFrame) -> DataFrame:

r3=pdd.concat([r,r2], axis=1)

return

@lambda_solid

def Input2(pdd: DataFrame) -> DataFrame:

r2 = pdd.read_csv('file2.csv')

return r2

@lambda_solid

def Result_output(y: DataFrame) -> DataFrame:

y3=y

return

@pipeline

def actual_dag_pipeline() :

y=Merge(Input1(),Input2())

Result_output(y)

Hi Team , I am facing issues in using ipynb files in dagstermill I am trying to execute a simple ipynb file The pipeline is created but when I click on execute I am getting error • Missing required field "solids" at document config root. Available Fields: "['execution', 'expectations', 'loggers', 'resources', 'solids', 'storage']".*

The second issue is i am not able to add boilerplate in the ipynb file dm.register_repository is failing as I see there is no such module in the package installed Could anyone from the team please help me

DM Code

import collections
import dagstermill as dm
from dagster import pipeline, execute_pipeline, lambda_solid
from dagster import PipelineDefinition,InputDefinition, OutputDefinition, Int
from dagster import Any, Field, lambda_solid, solid, pipeline, as_dagster_type
from dagster import InputDefinition, OutputDefinition, Int
import dagstermill as dm

my_notebook_solid = dm.define_dagstermill_solid(
                            name='DM1',
                            notebook_path='DM1.ipynb',
                            input_defs = [
                                InputDefinition(name='a'),
                                InputDefinition(name='b')
                            ],
                            output_defs  = [OutputDefinition()]
                        )
#def notebook_pipeline():
   # return PipelineDefinition(name='julia_pipeline', solid_defs=[my_notebook_solid])
def notebook_pipeline():
    return PipelineDefinition(name='pipeline', solid_defs=[my_notebook_solid])

ipynb file

import dagstermill as dm

from dagstermill.examples.repository import define_example_repository

a = 1

b = 2

result= a+b

dm.yield_result(result)

Hi Team , The above mentioned issue still exists I even tried a simple jupyter notebook with just a print hello message and I am getting an execution error

notebook_solid = dm.define_dagstermill_solid(
    name='hello_world1',
    notebook_path='hello_world1.ipynb')
def notebook_pipeline():
    return PipelineDefinition(name='pipeline', solid_defs=[notebook_solid])

There seems to be some issue in executing notebook cells in dagstermill. I would appreciate if anyone could help me to resolve this issue .

I've got a colleague who is evaluating Dagster for some data manipulation in one of our applications, but he needs to be able to generate nested subtasks based on inputs to a given solid. I feel like that's a conversation that has happened on here previously, but a cursory search didn't turn it up. Is that a capability that Dagster currently possesses?

What’s the

config_cls

param in the

output_materialization_config

decorator?

I’m getting this error when I try the output materialization

Max Gasner just published a new version: 0.5.8.

Hi @max and Team ,the research on which I am working now requires julia script files to be processed simultaneously with python .I can see that dagstermill solids do not accept .jl files and accepts only ipynb files.Could you please let me know if there is a way to run julia scripts in dagstermill .

Hi @max I have a important query If I run julia via python script ,will that be supported by Dagster as I see Dagster just needs to run the python code which It supports . Below example explains what I mean. Currently I have environment issue I am unable to test my condition and below code works and as a normal python script and I have tested it .Could you help me to understand If dagster will support this in the form of a solid definition

#file.py 
 import julia
 j = julia.Julia()
x = j.include("test.jl")

I tried it worked this way in dagster