dagster #announcements

King Chung Huang

07/25/2019, 11:23 PM

When declaring a pipeline, can I also set some config values on solids? For example, with the following solid and pipeline, can I set the value of the

prefix

config to something like

_foo_

as part of the pipeline, and still optionally set it via the environment when executing the pipeline?

Copy code

@solid(
    config={'prefix': Field(str, is_optional=True, default_value='_')}
)
def prefix_value(context, v):
    return f'{context.solid_config["prefix"]}{v}'

@pipeline
def test_normalize():
    prefix_value.alias('prefix_foo')() # set prefix to _foo_ somehow?
    prefix_value.alias('prefix_bar')() # and, set prefix to _bar_ somehow?

braunk

07/28/2019, 9:47 PM

Hello! Is it possible to set slack notification on a pipeline run error? I was having trouble finding this in the docs? Also I know Airflow and Dask are options for execution, but couldn’t Cloudwatch event triggering an ECS task work just as well?

King Chung Huang

07/28/2019, 10:41 PM

I've been looking through the examples and source code, but I can't figure out how to specify a file as input. The error I'm getting says, “input input_file must get a value either (a) from a dependency or (b) from the inputs section of its configuration.” How do I do the latter (b)?

Copy code

@solid()
def io_solid(context, input_file: FileHandle):
    <http://context.log.info|context.log.info>(f'Read from {input_file.path_desc}.')


@pipeline
def test_io_solid():
    io_solid()

King Chung Huang

07/29/2019, 7:52 PM

Is there an example of how to use the

s3_resource

from

dagster_aws

user

07/29/2019, 11:03 PM

Max Gasner just published a new version: 0.5.5.

Oleg Agapov

07/31/2019, 1:35 PM

Hi Dagsters, I know that Dagster doesn't have scheduler out-of-the-box. But can you recommend any schedulers which will be easy to pick up and integrate?

Justin

07/31/2019, 5:30 PM

Hello hello. I’m trying out the initial hello_world tutorial and it’s not working. I don’t get any output from running the pipeline from the CLI, and in dagit things seem like they may work, but I don’t get any “result” from the actual step. Also, when add the name== ‘__main__’ and run the python hello_world.py it just returns/outputs nothing =/

Justin

07/31/2019, 5:33 PM

The

pipeline

and the

solid

do execute (or at least, print statements I put into them do print things to the console), but my output does not match the output from the docs, so I’m confused

Justin

07/31/2019, 5:35 PM

Specifically, I’m concerned by the lack of an emitted output in my execution (right)

schrockn

07/31/2019, 5:36 PM

we could display the output value for scalar types

schrockn

07/31/2019, 5:36 PM

but we don’t do it in the general case

schrockn

07/31/2019, 5:36 PM

(we used to do this because repr can be VERY slow on certain things)

schrockn

07/31/2019, 5:36 PM

but it is “working”

schrockn

07/31/2019, 5:36 PM

just the hello wolrd output is confusing

Justin

07/31/2019, 5:37 PM

Ok. Great. The docs are just out of date then?

schrockn

07/31/2019, 5:37 PM

yes

Justin

07/31/2019, 5:37 PM

👍

schrockn

07/31/2019, 5:37 PM

@nate filing issue

nate

07/31/2019, 5:37 PM

https://github.com/dagster-io/dagster/issues/1635

schrockn

07/31/2019, 5:37 PM

i think if you proceed to next tutorial stage things will make more sense

schrockn

07/31/2019, 5:37 PM

what you could do is add config to “materialize” the output to a file somewhere

Justin

07/31/2019, 5:37 PM

Got it. I like to take small steps to ensure sanity at first 😃

schrockn

07/31/2019, 5:37 PM

to prove to yourself that things are working

schrockn

07/31/2019, 5:37 PM

gimme one sec

schrockn

07/31/2019, 5:38 PM

Copy code

solids:
  hello_world:
    outputs:
      - result:
          json:
            path: "/tmp/hello_world.json"

schrockn

07/31/2019, 5:38 PM

in the config editor

schrockn

07/31/2019, 5:39 PM

if you type it is fun because typeaheads

schrockn

07/31/2019, 5:39 PM

then you’ll see the materialization here. you can click to copy-paste the link. and then less it in a shell to see output

Justin

07/31/2019, 5:40 PM

Sweet - ok thanks. What does “in the config editor” mean specifically?

schrockn

07/31/2019, 5:40 PM

execute tab