Hi, guys! Is there common machine learning workflow in dagster (i mean going from training a model to using the model for predict)? Like dagster best practice? I have a pipeline where I load, transform data, train model. After that I would like to start using this pipeline with this trained model in prod. What's the best way to do that? Or I shouldn't use dagster for pipeline serving at all?

Hi, why doesn't

dagster pipeline execute

return with non-zero exit code when the pipeline fails?

<!channel> Hi all! We are going to be releasing dagster 0.8.0 on Thursday, which includes a bunch of pretty deep architectural changes to help set the system up for future growth. We were thinking of doing a little webinar next week to present some of the new user-facing features and talk about some of the changes we've made to internals and where we see those leading. We were thinking about next Tuesday morning, June 16th, at 9am Pacific Time (UTC-7) but would like to make sure that time works for those who would be interested. I know we have a lot of users in CET and that 6pm might be a bit of a stretch 🙂 🕘 🕕

where i’ve to set -y while setting scheduler? Getting error

Error: Invalid value for '--repository-yaml' / '-y': Path 'None' does not exist.

Scheduler defintion

def imdex_schedules():
    print("Scheduler function")
    return [
        ScheduleDefinition(
            name='DC_Log_Cleaner',
            cron_schedule='2 * * * *',
            pipeline_name='my_pipe',
            environment_dict=basic_config,
        )
    ]

Hi, We want to run a

bash_command_solid

inside a

composit_solid

- and need to set a dependancy between the output of the bash_command and another solid in the

pipeline

. We are not able to get an output from the bash_command_solid. What would be the correct approach for such task? While researching this issue - I'm trying to understand the bash_script_solid test example - can you get the return from the bash script? what does the following line do in the pipeline ?

a()

@sephi I struggled a bit with that as well and ended up getting what I wanted from using the

execute

function in the utils module of dagster_bash. You can see what I did here https://github.com/mitodl/ol-data-pipelines/blob/master/ol_data_pipelines/edx/solids.py#L476-L480

Thx @Tobias Macey - However in our case we need to run an entire script (and not just a bash command) - can the

execute

take a script? i'll try and check it out

Hi I'm new to dagster. I'm trying to set up a schedule with several solids that depend on the date. In the scheduling tutorial the date is passed through the config to every single solid. Is this how it's meant to be done?

Hi guys, small feature request: would be great to be able to filter runs by execution date I'm still rocking 0.7.7, so apologies if this is already out.

Are there any primitives in dagster I can use to respond to celery worker failures and restarts?

Question about getting dagstermill up and running

Hello Guys, I am having an issue running solidSubset with "Launch Execution". Details Posted in Comment Sections. My Goal is to run solidSubset on a GKE Launch-Execution. But it is failing when I pass inputs values for non-first-solid.

Hi, new to dagster and looking to use it as our computation pipeline library. Currently (pre-dagster) we store intermediate results in file and in the next pipeline step load those in again. This is very similar to Dagster's SerializationStrategy, nice! Our next need is: is it possible to re-execute parts of the pipeline based on the serialized data after restarting dagit, or is this possible with the CLI tools? I can re-execute single steps in dagit, but when I restart dagit, the previous runs are not shown anymore and I have to re-run the entire pipeline before I can select parts.

Hi, I am trying to execute hello world from tutorial and when I run it like this: dagster pipeline execute -f ./hello_dagster.py

I get error: Error: Invalid set of CLI arguments for loading repository/pipeline. See --help for details.

Any ideas?

(dagster version 0.7.16)

Hello 😄 I was wondering if there has been any discussions or plans around tools that enable memory inspection of solids? Or is something like this outside the scope of dagster and dagit development? I really like the runtime details I get when running my pipeline with dagit, and I feel like details on memory usage would be helpful when debugging. For now, I'm using a combination of

tracemalloc

and

memory_profiler

in my solids to get a better understanding of their memory consumption

Hi I am new to Dagster, I've run through the tutorials, which were great, but I'm wondering if it is possible to set Dagster up in such a manner that every pipeline executed via dagit/dagster is automatically added to a repository stored in a postgresdb??

Max Gasner just published a new version: 0.8.0.

Hi all, we are thrilled to release dagster 0.8.0 "In The Zone", which marks a big step forwards for the system. Please see

CHANGES.md

for a full list of changes, and the

080_MIGRATION.md

migration guide for details on updating 0.7.x code to work with 0.8.0. We'll be hosting a webinar on Tuesday morning (9am Pacific/noon Eastern/6pm CET) to present some of these changes and hopefully invite some discussion with the community -- DM me if you'd like to be added to the calendar invite. I'll post summaries of the major changes below for comments and discussion in threads here.

Workspace, host and user process separation, and repository definition Dagit and other tools no longer load a single repository containing user definitions such as pipelines into the same process as the framework code. Instead, they load a "workspace" that can contain multiple repositories sourced from a variety of different external locations (e.g., Python modules and Python virtualenvs, with containers and source control repositories soon to come). The repositories in a workspace are loaded into their own "user" processes distinct from the "host" framework process. Dagit and other tools now communicate with user code over an IPC mechanism. This architectural change has a couple of advantages: - Dagit no longer needs to be restarted when there is an update to user code. - Users can use repositories to organize their pipelines, but still work on all of their repositories using a single running Dagit. - The Dagit process can now run in a separate Python environment from user code so pipeline dependencies do not need to be installed into the Dagit environment. - Each repository can be sourced from a separate Python virtualenv, so teams can manage their dependencies (or even their own Python versions) separately. We have introduced a new file format,

workspace.yaml

, in order to support this new architecture. The workspace yaml encodes what repositories to load and their location, and supersedes the

repository.yaml

file and associated machinery. As a consequence, Dagster internals are now stricter about how pipelines are loaded. If you have written scripts or tests in which a pipeline is defined and then passed across a process boundary (e.g., using the

multiprocess_executor

or dagstermill), you may now need to wrap the pipeline in the

reconstructable

utility function for it to be reconstructed across the process boundary. In addition, rather than instantiate the

RepositoryDefinition

class directly, users should now prefer the

@repository

decorator. As part of this change, the

@scheduler

and

@repository_partitions

decorators have been removed, and their functionality subsumed under

@repository

.

Dagit organization The Dagit interface has changed substantially and is now oriented around pipelines. Within the context of each pipeline in an environment, the previous "Pipelines" and "Solids" tabs have been collapsed into the "Definition" tab; a new "Overview" tab provides summary information about the pipeline, its schedules, its assets, and recent runs; the previous "Playground" tab has been moved within the context of an individual pipeline. Related runs (e.g., runs created by re-executing subsets of previous runs) are now grouped together in the Playground for easy reference. Dagit also now includes more advanced support for display of scheduled runs that may not have executed ("schedule ticks"), as well as longitudinal views over scheduled runs, and asset-oriented views of historical pipeline runs.

Assets Assets are named materializations that can be generated by your pipeline solids, which support specialized views in Dagit. For example, if we represent a database table with an asset key, we can now index all of the pipelines and pipeline runs that materialize that table, and view them in a single place. To use the asset system, you must enable an asset-aware storage such as Postgres.

Run launchers The distinction between "starting" and "launching" a run has been effaced. All pipeline runs instigated through Dagit now make use of the

RunLauncher

configured on the Dagster instance, if one is configured. Additionally, run launchers can now support termination of previously launched runs. If you have written your own run launcher, you may want to update it to support termination. Note also that as of 0.7.9, the semantics of

RunLauncher.launch_run

have changed; this method now takes the

run_id

of an existing run and should no longer attempt to create the run in the instance.

Flexible reexecution Pipeline re-execution from Dagit is now fully flexible. You may re-execute arbitrary subsets of a pipeline's execution steps, and the re-execution now appears in the interface as a child run of the original execution.

Support for historical runs Snapshots of pipelines and other Dagster objects are now persisted along with pipeline runs, so that historial runs can be loaded for review with the correct execution plans even when pipeline code has changed. This prepares the system to be able to diff pipeline runs and other objects against each other.

Step launchers and expanded support for PySpark on EMR and Databricks We've introduced a new

StepLauncher

abstraction that uses the resource system to allow individual execution steps to be run in separate processes (and thus on separate execution substrates). This has made extensive improvements to our PySpark support possible, including the option to execute individual PySpark steps on EMR using the

EmrPySparkStepLauncher

and on Databricks using the

DatabricksPySparkStepLauncher

The

emr_pyspark

example demonstrates how to use a step launcher.

Clearer names What was previously known as the environment dictionary is now called the

run_config

, and the previous

environment_dict

argument to APIs such as

execute_pipeline

is now deprecated. We renamed this argument to focus attention on the configuration of the run being launched or executed, rather than on an ambiguous "environment". We've also renamed the

config

argument to all use definitions to be

config_schema

, which should reduce ambiguity between the configuration schema and the value being passed in some particular case. We've also consolidated and improved documentation of the valid types for a config schema.

Lakehouse We're pleased to introduce Lakehouse, an experimental, alternative programming model for data applications, built on top of Dagster core. Lakehouse allows developers to define data applications in terms of data assets, such as database tables or ML models, rather than in terms of the computations that produce those assets. The

simple_lakehouse

example gives a taste of what it's like to program in Lakehouse. We'd love feedback on whether this model is helpful!