installing

0.6.0

seems to work fine using pip but getting some interesting errors when trying to install via pipenv:

is this structure a generally recommended pattern for building dagster resources? https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-snowflake/dagster_snowflake

does anyone have a simple working example of how to use pytest to test a solid?

specifically on windows 10

I have a

solids.py

in the root folder, then a `tests\test_solids.py

Here's an error I am seeing:

@staticmethod
    def build(pipeline, environment_dict=None, run_config=None):
        check.inst_param(pipeline, 'pipeline', PipelineDefinition)
        check.opt_dict_param(environment_dict, 'environment')
        run_config = check.opt_inst_param(run_config, 'run_config', RunConfig, default=RunConfig())

        mode = run_config.mode or pipeline.get_default_mode_name()
        environment_type = create_environment_type(pipeline, mode)

        result = evaluate_config(environment_type, environment_dict, pipeline, run_config)

        if not result.success:
>           raise DagsterInvalidConfigError(pipeline, result.errors, environment_dict)
E           dagster.core.errors.DagsterInvalidConfigError: Pipeline "ephemeral_generate_date_suffix_solid_pipeline" config errors:
E               Error 1: Missing required field "solids" at document config root. Available Fields: "['execution', 'expectations', 'loggers', 'resources', 'solids', 'storage']".

..\..\.venv\lib\site-packages\dagster\core\system_config\objects.py:114: DagsterInvalidConfigError

using dagster 0.6.0

nevermind, looks like args with defaults still need to have their values specified.

Sashank Thupukari just published a new version: 0.6.1.

I have a scenario where I would need a solid that acts as a webhook listener. Do y'all have any examples of existing solids that function in a similar way? Or is there a more appropriate path y'all would recommend?

curious what the intention of

is_secret

is for a Field: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/types/field.py#L43

trying to understand an error I'm seeing re: input typing (in thread)

Hi! I'm thinking through a production dagster use case and am unclear on the following: let's say I'm running dagster on a remote VM/container and want to use the dagit UI to trigger and monitor pipeline executions from my local machine's browser. Is there a recommended way to go about that?

hey there! I'm looking at adopting Dagster to describe and run our data pipelines at work. one question I have is around intermediate and final output files. I've seen the

Materialization

API and understand that persisting intermediates is up to the pipeline code. Is the

FileCache

API the recommended way to manage file-based intermediates? What about final outputs that I want to tie to the particular job that created them?

Hello everyone! Just announced 0.6.0 on Twitter: https://twitter.com/schrockn/status/1182397096281853952 Would be very grateful for likes and RTs! 🙏🏻🙏🏻🙏🏻🙏🏻🙏🏻🙏🏻

Here is a medium article with more detail: https://medium.com/dagster-io/dagster-0-6-0-impossible-princess-898b459375e0#729b-42f680ef3a3a

On Windows 10 dagster/dagit 0.6.1, in the hello_dag example, the dagit pipeline execution finishes after only starting and finishing _solid_one_. This message is displayed in the logs:

An exception was thrown during execution that is likely a framework error, rather than an error in user code.
Original error message: OSError: [WinError 6] The handle is invalid

Stack Trace:
  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\site-packages\dagster_graphql\implementation\pipeline_execution_manager.py", line 312, in _in_mp_process
    instance,

  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\site-packages\dagster\core\execution\api.py", line 141, in execute_run_iterator
    step_keys_to_execute=step_keys_to_execute,

  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\site-packages\dagster\core\execution\api.py", line 101, in _pipeline_execution_iterator
    step_keys_to_execute=step_keys_to_execute,

  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\site-packages\dagster\core\engine\engine_inprocess.py", line 120, in execute
    yield step_event

  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\contextlib.py", line 119, in __exit__
    next(self.gen)

  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\site-packages\dagster\core\execution\compute_logs.py", line 34, in mirror_step_io
    yield

  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\contextlib.py", line 119, in __exit__
    next(self.gen)

  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\site-packages\dagster\core\execution\compute_logs.py", line 44, in mirror_stream
    yield

  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\contextlib.py", line 119, in __exit__
    next(self.gen)

  File "c:\users\tnaugle\appdata\local\continuum\anaconda3\envs\dagster_new\lib\site-packages\dagster\core\execution\compute_logs.py", line 68, in redirect_stream
    from_stream.flush()

prha just published a new version: 0.6.2.

Hi everyone 👋 I stumbled across Dagster the other day and it looks great! I'm considering adopting it for a project, but I have one key question... Has anyone tried running a Dagster pipeline on Windows Server? (I don't want to use Windows Server but unfortunately it's a hard requirement 😢)

Hi all! I'm currently studying Dagster and trying to use it for tidying up ML training. I got kind of stuck with the following problem. A complex solid ("train") in a pipeline needs to be run several times with somewhat different input, but the number of these runs is determined by an integer value in a global config (ML folds). I see only three solutions: • generate the pipeline on the fly based on an original config: more code = more bugs, difficult to use tools like Dagit, seems like an abuse of the framework; • iterate over inputs in a single complex solid "train": looks good in Dagit, but limited parallelism; • just generate the pipeline with a fixed number of parallel "train" solids (i.e. a fixed maximum number of runs), then use optional outputs: seems sort of ugly and with artificial constants. How could this problem be solved in Dagster in an idiomatic way?

I have a somewhat similar question: We are currently working on a classification pipeline in dagster. We have solids for reading in different inputs, let‘s say input A, B and C. Also we have different solids for training different classifiers, let‘s say classifiers X and Y. Currently we are building separate pipelines for training classifier Y with input A and B, for training X with B and C and so on. So every pipeline uses different solids. We are currently debating if it would be preferable to build one giant pipeline, where you can control the used input and classifiers through the config file. What would be the best way to approach this? Pipeline construction or changes via the config file are not possible AFAIK. So we can not have f.e. a composite solid that chooses between solids A, B and C for input generation based on a configuration file, right? Should we always have all solids connected and control their behavior via output parameters? Or outsource all functionality like reading inputs and initializing classifiers away from solids and only make the right function calls inside the solids based on the configuration? Or is the single pipeline approach the superior approach?

[1/5] Two questions regarding testing solids: Let’s say we have a simple test setup for a solid using `execute_solid`:

import unittest
  
from dagster import execute_solid, lambda_solid 

@lambda_solid
def add_one(x):
    return x + 1
 
class AddOneTest(unittest.TestCase):
    def test_for_zero(self):
        result = execute_solid(add_one, input_values={"x": 0})
        self.assertEqual(1, result.output_value())

When I run the test with

unittest

on the command line, the standard output of the test is quite extensive: (continue to read in the thread…)

Second question is about testing for errors. Let’s assume i have a simple solid that is supposed to fail for a certain input. I use this input in the test and want to test if the correct error is raised (in this case

TypeError

). However, since dagster catches that error and turns it into another type of error, I have to do something like this:

import unittest

from dagster import DagsterExecutionStepExecutionError, execute_solid, lambda_solid

@lambda_solid
def add_one(x):
    return x + 1
 
class AddOneTest(unittest.TestCase):
    def test_fail_for_string(self):
        with self.assertRaises(DagsterExecutionStepExecutionError):
            execute_solid(add_one, input_values={"x": "s"})

Any good ideas on how I can test for the underlying

TypeError

instead of the dagster error easily?

I'm running into an issue using

dagstermill

when accessing solid configs from inside the notebook. The notebook has a

dict

config, declared like:

config_field=Field(
        Dict({
            'areas': Field(List[str], is_optional=True, default_value=["a", "b", "c"]),
            'export_file': Field(Path)
        })
    )

When I access the config value in the notebook during a job run with

context.solid_config['export_file']

, I get a

KeyError

. What options do I have to debug this? Can I step through the generated notebook in the

/tmp

location on my own to reproduce it? Or is there a way to debug a pipeline job while it's running?

asingh16 just published a new version: 0.6.3.

changelog for 0.6.3: https://github.com/dagster-io/dagster/blob/master/CHANGES.md#063

I'm new, so forgive me. Scattered thoughts below. Are pipelines orchestrated entirely within a compute? And thus able to share the magic

context

resource ? Contrasting to airflow where you have a worker that runs a single task with zero context between other tasks (besides the shoe-horned xcom) If two tasks need to operate on different platforms: pyspark in one, general redshift sql in another and finishing with a python function. how does dagster handle this?

Hi Guys, I am also new to dagster and more responsible for CI/CD pipelines like Jenkins for Data Analytics and Engineering issues. Is there a good example on how to integrate, build and deploy Dagster Projects thru a Jenkins Pipeline?

Including Docker and maybe Kubernetes, if available