Max Gasner just published a new version: 0.6.5.

🦃

set the channel topic: 🎅🎅🎅🎅🎅 General discussion and holiday cheer! 🎅🎅🎅🎅🎅

Apologies if I missed it in the docs, is there a notion of a timeout/heartbeat for a Solid? ie, if we have long running tasks that would need to keep sending a heartbeat/alive, and if that stops, reschedule the Task as failed

Are there best practices for managing credentials within a dagster pipeline (especially when using a scheduler)? I know Airflow has some support for this built in to the UI that essentially centralizes the credentials in a DB. didn't know if this is addressed and I missed it or if there are better ways?

Is there a place where logs are being stored?

ya - depending on how you configure your “instance” https://dagster.readthedocs.io/en/0.6.5/sections/deploying/instance.html

Ah! Is there a default storage if I don't configure it? Does it make a random folder in temp?

ya if

$DAGSTER_HOME

is not set we use a temp folder that should get cleaned up on exit

Ok great thanks!

Let us know if you have any questions about setting up your

DAGSTER_HOME

(I'm new, so I had to do it for the first time last week, so I wanna make it as seamless as possible since it's still fresh in my head)

Max Gasner just published a new version: 0.6.6.

👋🏻 Dagster community

As a consultancy we do a lot of data warehouse projects on GCP. Airflow we find is not very “analyst friendly” and hard to run locally (for example won’t run on a win pc). Hence our interest in Dagster.

What are the guidelines for running on GCP? Thank you!

Hiho 👋 🙂

Dagster and Alluxio are a natural fit. Are there any plan to offer Alluxio on GCP prior to Elemental cloud?

Hi All, I have two hopefully quick questions. 1. Is there a way to leverage parallel processing in Dagster ? 2. What's the recommended way to daemonize dagit to act as a constant portal for all pipelines ? (see comments on thread for more detail on question 1)

HI Team I am trying to use Yaml file and define a common workflow format and then try to translate it into dagster format . I am first reading and loading yaml then converting it into json format. Can anybody from the team please confirm if this would be the right approach to further use it to translate it into corresponsing dagster workflow format. Basically I want tasks and everything to be defined in yaml format and dagster to represent it. I would appreciate any help regarding this or suggestions to translate yaml to dagster . TIA

this is what my sample Yaml looks like I want to write python script to translate it into Dagster workflow

cwlVersion: v1.0
class: Workflow
inputs:
  Input1:
    type: File?
    extract: string
  Input2:
    type: File?
    extract: string
outputs:
  task3_output:
    type: File
    outputSource: task3/outfile
steps:
 task1:
   in:
     files:
       source: Input1
   out: [task1out]
 task2:
   in:
     files:
       source: Input2
   out: [task2out]
 task3:
    in:
      files:
        source: [task1out, task2out]
        linkMerge: merge_flattened
    out: [outfile]

Hi Dagster folks! I’m running into an issue where a pipeline fails in dagit but works when run with

execute_pipeline

. I posted about it on StackOverflow, was wondering if anyone can help. https://stackoverflow.com/questions/59379523/dagster-pipeline-executes-successfully-when-run-with-execute-pipeline-but-not

hello! i'm trying to test out dagster on a GCP AI platform notebook. the notebooks come installed with python 3.5.3. following the demo here: https://medium.com/dagster-io/dagster-0-6-0-impossible-princess-898b459375e0 when I run

dagit -f hello_world.py -n define_hello_world_repo

I get a

RecursionError: maximum recursion depth exceeded in __instancecheck__

error, in this file: /usr/lib/python3.5/typing.py was wondering if you think this is a compatibility issue with python 3.5.3. the docs say dagster is tested against 3.5.7

Is there a way for the input to a

solid

to have a default value? e.g.

@solid
def some_solid(context, x, y=1.0):
    do_something(x, y)

Related question, is there a way to set the value for a solid in config so that aliases of the solid inherit that value?

Hi all. Can somebody tell me is there some way to implement pipeline similar to map-reduce?

We're evaluating building our own ML file/data versioning/caching solution or using an open source solution. Anyone have experience with DVC? https://dvc.org/ they use Git for data version control, which is as exciting as it is terrifying. Any other suggestions? DeltaLake looks positioned for huge enterprise, not ML startups.

Dagster should be tracked here: https://github.com/EthicalML/awesome-production-machine-learning

🎅 Merry Christmas to all from the team at Element behind Dagster! 🎅

hey guys. been poking around the docs for a few days, and definitely looks pretty cool

i'm finding myself bewildered about how to build what i think should be a pretty basic workflow, though