I'm trying out dagster for the first time today, and just going through the tutorials. When I run a pipeline like in the

hello_dag.py

example, how do I see the output from the solids? For example, I'm expecting

solid_two

to output

foo foo

. But, I don't see that in the output.

/project/src # dagster pipeline execute -f hello_dag.py -n hello_dag_pipeline
2019-07-24 21:01:09 - dagster - INFO - 
        orig_message = "Solid 'solid_one' emitted output 'result'"
      log_message_id = "f2bb6fbc-8eb6-49bf-8893-7164481139b6"
       log_timestamp = "2019-07-24T21:01:09.704841"
              run_id = "b49ee030-d8ee-46eb-97b8-9c1aa176a41a"
            pipeline = "hello_dag_pipeline"
execution_epoch_time = 1564002069.697249
            step_key = "solid_one.compute"
               solid = "solid_one"
    solid_definition = "solid_one"
2019-07-24 21:01:09 - dagster - INFO - 
        orig_message = "Solid 'solid_two' emitted output 'result'"
      log_message_id = "77559bfb-7390-4831-8ba0-ec152dfd7323"
       log_timestamp = "2019-07-24T21:01:09.706621"
              run_id = "b49ee030-d8ee-46eb-97b8-9c1aa176a41a"
            pipeline = "hello_dag_pipeline"
execution_epoch_time = 1564002069.697249
            step_key = "solid_two.compute"
               solid = "solid_two"
    solid_definition = "solid_two"

@King Chung Huang hey King! So right now we actually made the decision to not print output values in the logs by default. We actually used to do this and blindly used

repr

to report values, but that actually was prohibitively slow for libraries such as Pandas. We’re also concerned about users leaking potentially sensitive information in logs. One can access the output values from the return value of execute_pipeline if you are using the python API.

We are releasing an improved log viewer soon which will render values for primitives types in dagit.

Ah, ok!

You can always emit your own log messages of course, but I understand that is tedious to do for every output.

Is there a way to capture the outputs to a file, when running via the CLI?

Like an inverse of the environment file?

currently you can configure a pipeline run to persist all outputs to the runs folder

Cool, I'll look for that. Thanks!

so you would add the following to your config file:

storage:
   filesystem:

and then it will persist in /path/to/your/tmp/dagster/runs/<run_id>

however fair warning, this is meant to be consumed by the system and so all interemediate values are pickled by default

@King Chung Huang another feature which may help you out is the ability to materialize values via config. So take the example of the “reusing_solids_pipeline” in the our intro tutorials run dagit in

dagster/examples/dagster_examples/intro_tutorial

to load it

^-- this is what that looks like

k

with this config file this actually outputs the output of “final” to the random temp file i specified

so that is another method of materialized and tucking away values as they flow through the pipeline

K. I've only just realized that all the examples are in the repo!

I've been copying and pasting all the code from the tutorial pages. 😄

ah we should probably make that more clear then

I see now that it's stated on the Hello, World page (https://dagster.readthedocs.io/en/stable/sections/learn/tutorial/hello_world.html). Perhaps a note on the Tutorial page would help (https://dagster.readthedocs.io/en/stable/sections/learn/tutorial/index.html).

Can I access the context from a pipeline function, in order to log some output? For example, I have a pipeline with a solid just to log

tokens

right now. Can I log the tokens directly from the pipeline?

@solid
def print_tokens(context, tokens):
    <http://context.log.info|context.log.info>(str(tokens))

    return tokens


@pipeline
def test_tokenize():
    tokens = tokenize()
    print_tokens(tokens)

@pipeline

decorated functions are only evaluated at init time and just used to determine the dependency graph of the pipeline.

only the body of

@solid

/

@lambda_solid

functions are evaluated during execution, so if you want do do some logging it will have to happen there

does that make sense?

Ah, ok. Makes sense. Thanks!

I'm trying to learn how to create a

composite_solid

that uses another solid and sets a config value in it. In this code snippet, the

prefix_value

solid prefixes the input value with the

prefix

from the config. Then, the

prefix_id

composite solid uses

prefix_value

but sets the prefix to

_id_

.

@solid(
    config={
        'prefix': Field(str, default_value='_')
    }
)
def prefix_value(context, v):
    return f'{context.solid_config["prefix"]}{v}'


@composite_solid(
    config_fn=lambda _, cfg: {
        'prefix_value': {'config': {'prefix': '_id_'}},
    },
    config={},
)
def prefix_id(id):
    return prefix_value(id)

I'm getting

dagster.check.CheckError: Invariant failed. Description: Cannot specify empty config for ConfigMapping

. But, I'm not sure how to solve it. Is this a correct way for a solid to use another solid with a custom config?

Sashank Thupukari just published a new version: 0.5.4.