What's the proper way to pass a list to a solid in a preset? I have solid

datasplit_y

with an argument

id_cols

that takes a list of strings. All of the below have failed: 1

datasplit_y:
    inputs:
      id_cols: 
        - user_id
        - program_day

2

datasplit_y:
    inputs:
      id_cols:
        value:
          - user_id
          - program_day

3

datasplit_y:
    inputs:
      id_cols: [user_id, program_day]

4

datasplit_y:
    inputs:
      id_cols: ['user_id', 'program_day']

5

datasplit_y:
    inputs:
      id_cols: 
        value: ['user_id', 'program_day']

6

datasplit_y:
    inputs:
      id_cols: 
        value: ['user_id', 'program_day']

7

datasplit_y:
    inputs:
      id_cols: 
        value: "['user_id', 'program_day']"

8

datasplit_y:
    inputs:
      id_cols: 
        value: "[user_id, program_day]"

Clearly I'm throwing hail-marys at this point. Any advice?

Is there any way to currently trigger a partition backfill but insure that partitions run in some specified ordering? At the moment it seems like partitions will just run/finish in whatever order they decide to start.

Currently having a rough time figuring out why none of my schedules are running atm. Inside my test dagster setup I've made four hourly_schedule's, one for each 15 minute mark of the hour, and none of them seem to be running. Dagit isn't showing any errors for their configuration, and running them in the playground seems to work? Any tips/tricks on what I may have done wrong here? I made a cron schedule as well (every 5 minutes), and that's not running either.

I'm trying to figure out how should I define a config_schema of this shape:

categories:
  home:
  - {source: foo, mult: 1}
      
  business:
  - {source: bar, mult: 2}
  - {source: baz, mult: 3}

  ...

Here I have a dictionary with different keys (which I want to use as meaningfull values - home, business, etc) and an array of structured values (each one is {source: str, mult: int}) I see 3 ways now, all of them are quite cumbersome: • just use a Permissive for all the config - I would loose validation at the last leaf level, that's pitty • change the shape of config to have

categories:
  - name: home
    sources:
    - {source: foo, mult: 1}
      
  - name: business
    sources:
    - {source: bar, mult: 2}
    - {source: baz, mult: 3}

  ...

looks ugly adding unnecessary 'name' and 'sources' • create my own DagsterType for the whole config - looks a little like overkill here. So am I missing something? Is there a better way or what should be more "dagsterish"? D

Post about how we are monitoring

Dagster assets

(numeric values). https://dev.to/sephib/simple-pipeline-monitoring-dashboard-386p

Hi everyone; first up thanks for all the great work on dagster, really enjoying playing around with it so far. I've created a pipeline with multiple solids that can all be ran in parallel (potentially) but which all require a GPU. I can see a way to talk about resources in a distributed setting in dask: https://docs.dagster.io/deploying/dask is there a way to talk about resources for local execution? specifically for this pipeline to run each solid that needs access to the GPU needs to wait for the other to finish. I can clearly get the behaviour I want by ... putting explicit dependencies between each solid in the dag, i.e. turning

(A -> D, B -> D, C -> D)

into

(A -> B, B -> C, C -> D)

but that seems weird.

Question - is there a way for an on prem dagster to pass information to an off prem dagster ?

ie: if I wanted to do data profiling on a upstream on-prem pipeline and compare to a downstream cloud native pipeline.

Hi all! Do you think that assets a good way to materialize a daily report (currently text and tabular data)? Or should we handle that independently saving the data to s3 and storing the metadata in our warehouse?

Hi, where can I read more about how solids send data to each other depending on runtime engine used? We finished data importer for our use case lately which we consider implementing in Dagster but I am concerned about performance as data is huge. Also is there any information on how to control batches of data and error handling. For example if I have one file which is batched as 100 batches each having 3 steps, how can I control what happens if batch 56 fails at step 2. We currently implemented this in code and DB state and we have complete flexibility on what to do (cancel only the failed batch but accept import, cancel import, retry batch, accept import but log the batch failure and revert it's changes, etc.). We could easily convert this code to Dagster as is currently with DB state of import progress, however I have a feeling that state we maintain about batches, stages, imports could come as freebies if we implemented it in Dagster in a stateless way but then again I am afraid of losing flexibility on error handling I mentioned above. Any advice? Thanks.

Daggit question: in the schedules menu times are all UTC, but in runs times are shown in whatever timezone someones computer is in. Would there be a way to align these?

npm is down 😵 https://status.npmjs.org/ sorry for requesting to delete node_modules from dagster 😅

interested in the lifecycle hooks that are mentioned in https://github.com/dagster-io/dagster/issues/2520 -> is there a rough target date for 0.9.0 release?

It will be released today!

In a few hours

n00b question: I'm looking at moving from a weirdo homegrown tool to Dagster and trying to understand launchers vs executors. The docs explain them well but I'm still not sure why I want a launcher. We are planning to use the celery executor to get some durability in case the executor itself crashes or is lost. With that executor, do I still need a launcher?

It seems like for the celery executor case, the launcher would just be a no-op since all it's going to do is push a message into rabbit

Sashank Thupukari just published a new version: 0.9.0.

0.9.0 Release -- “Laundry Service” To the Dagsterati: As you just saw, we released 0.9.0, code-named Laundry Service. This release builds on the substantial architectural changes in 0.8.0, and is mainly about maturing our system around the dimensions of infrastructure, deployment, configuration management. When viewed in totality, we believe these features improve Dagster’s suitability as an internal platform, with well-defined boundaries and interfaces between teams managing infrastructure and teams building pipelines.

Independent Tool and Repository Processes and Deployment Many of you work on internal data platforms that serve multiple teams. This release provides an oft-requested capability: deploying tools and user code independently. This was accomplished by making repositories (or a set of repositories) host-able as a gRPC server. We now have a formalized interface for both querying a repository’s metadata and invoking the pipelines within it. These can deployed separately from your Dagit server. * Dagit can now be a fixed image which does not need to be redeployed when pipelines or solids change. * Team-specific repositories can be deployed independent of each other, each with their own python environments. * Repository environments no longer require the Python GraphQL stack. We believe this is a much more stable, reliable architecture for data platform teams serving multiple internal teams.

Kubernetes Improvements We added a number of features and improvements to our Kubernetes-based deployment stack. * Fully implemented user code separation described above. See https://docs.dagster.io/deploying/k8s_part2 for more details * Run termination * Can specify resource requirements for run coordinator Kubernetes pod on

@pipeline

* Support customized Postgres versions in Helm * Helm support for liveness checks for Celery works and flower in Celery on Kubernetes

Configured Users have reported that they often have large, repeated chunks of configuration across their pipelines that they never want to change at runtime. Based on this feedback, we’ve added a new ability to created pre-configured resources (as well as other definition types) that fully or partially apply configuration to pre-existing reusable resources. We imagine infrastructure teams building pre-configured resources that specify exactly the configuration they expose to leaf teams.

from some_aws_library import s3_session

# interface to s3_session:
# @resource(config_schema={'region': str, 'use_unsigned_session': bool})
# def s3_session(_init_context):

# bake all configuration with code:
east_unsigned_s3_session = s3_session.configured(
    {'region': 'us-east-1', 'use_unsigned_session': False}
)    
    
# or allow your users only to configure the region only
@configured(s3_session, config_schema={'region': str})
def unsigned_s3_session(config):
    return {'region': config['region'], 'use_unsigned_session': False}

Configured also be applied to solid definitions. This allows one to apply configuration mapping without having to introduce a

composite_solid

layer:

@solid(config_schema={'age': int, 'name': str})
def introduce(context):
    return "{name} is {age} years old".format(**context.solid_config)

introduce_aj = configured(introduce, name="introduce_aj")({'age': 20, 'name': "AJ"})

@pipeline
def return_int_pipeline():
    introduce_aj()

Hooks We have added hooks, currently marked as experimental, that enables you to execute cross-cutting success and failure handling policies on pipelines. This will enable you to, for example, send a slack message to a specific channel when any solid fails in a particular pipeline. You can also apply these on a per-solid basis. We imagine infrastructure teams providing common hooks for teams to apply common alerting policies to their pipelines. Example:

@failure_hook(required_resource_keys={'slack'})
def slack_on_failure(context):
    message = 'Solid {} failed'.format(context.solid.name)
    context.resources.slack.send_message(message)

@slack_on_failure
@pipeline
def a_pipeline():
    b(a()) # will fire if either a or b fail

@pipeline
def selective_notif().
    b.with_hooks({slack_on_failure})(a()) # only fires when b fails

Scheduler Improvements There are a number of improvements design to improve operations and deployment of the scheduler. This addresses many of the issues surface by platform teams responsible for deploying the scheduler and managing concerns that cut across user repositories: * `K8sScheduler`: a new scheduler for Kubernetes deployments that builds on top of Kubernetes

CronJob

objects. This decouples the scheduler from Dagit, enabling you to run jobs on a schedule without requiring cron on the Dagit node and enabling you to spin up multiple Dagit pods. * An instance level view of all the schedules that have been reconciled with the Scheduler. * The ability to turn on and off any schedules within your currently loaded workspace from one page. * View the state of and turn off all schedules, regardless of if there are in defined in the currently loaded workspace.

Thanks to the entire team and all of our contributors! We are always looking for constructive feedback. Our goal is to make you all successful. Feedback is critical for fulfill that goal, so keep it coming!

Hi people. Do you know if when using persistence storage for outputs it's possible to have some things not be saved forever? In a certain node I have a model that outputs a bunch of things, mostly are used only for subsequent calculations in the pipeline. On top of that some are heavy and we wouldn't like our storage to explode. Is there a way to handle this without deleting the runs?

Johann Miller just published a new version: 0.9.1.

hi i am using sphinx for auto documentation… my solid function not captured by sphinx… do you have any idea?

@solid(required_resource_keys={'vars'})
def clean_config_json(context, config_json = None, status = None):
    """Clean the dataset configuration"""
    pass