https://dagster.io/ logo
Join Slack
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Powered by Linen
announcements
  • f

    freiksenet

    09/13/2018, 12:58 PM
    when you just pass them to pipeline
  • s

    schrockn

    09/13/2018, 12:58 PM
    yu
  • s

    schrockn

    09/13/2018, 12:58 PM
    yup
  • s

    schrockn

    09/13/2018, 12:59 PM
    you’re thinking very clearly about this
  • s

    schrockn

    09/13/2018, 12:59 PM
    also if i’m understanding everything you’re saying, none of this requires API changes before we cut 0.2.0
  • s

    schrockn

    09/13/2018, 1:00 PM
    *none of your proposals
  • f

    freiksenet

    09/13/2018, 1:00 PM
    yeah
  • f

    freiksenet

    09/13/2018, 1:00 PM
    absolutely
  • f

    freiksenet

    09/13/2018, 1:00 PM
    I can work on that in parallel
  • f

    freiksenet

    09/13/2018, 1:00 PM
    and it should be fully backwards compatible
  • f

    freiksenet

    09/13/2018, 1:00 PM
    maybe smth for 0.2.1 or 0.3.0
  • s

    schrockn

    09/13/2018, 1:02 PM
    yeah i’d love that
  • s

    schrockn

    09/13/2018, 1:02 PM
    thing i’d be particular interested at this point
  • s

    schrockn

    09/13/2018, 1:03 PM
    1) what precisely does the backwards-compatible optional alias/pipeline node API look like 2) what is the concrete API proposal for higher order solids
  • f

    freiksenet

    09/13/2018, 1:05 PM
    yep
  • f

    freiksenet

    09/13/2018, 1:05 PM
    I think compute nodes made the alias thing pretty easy
  • f

    freiksenet

    09/13/2018, 1:06 PM
    because nodes are separate from solid definitions
  • s

    schrockn

    09/13/2018, 1:06 PM
    in terms of implementation absolute;y
  • s

    schrockn

    09/13/2018, 1:06 PM
    i love those goddamn compute nodes
  • f

    freiksenet

    09/13/2018, 1:06 PM
    maybe in future we'll have solids that generate multiple compute nodes
  • s

    schrockn

    09/13/2018, 1:06 PM
    yes
  • s

    schrockn

    09/13/2018, 1:06 PM
    the other thing
  • s

    schrockn

    09/13/2018, 1:07 PM
    is that if your head if in the space
  • s

    schrockn

    09/13/2018, 1:07 PM
    *is already in the space
  • s

    schrockn

    09/13/2018, 1:07 PM
    the other thing the compute nodes need to be able to do is spawn parallel subdags
  • s

    schrockn

    09/13/2018, 1:07 PM
    so you have a solid that emits a stream of outputs
  • s

    schrockn

    09/13/2018, 1:07 PM
    with the same name
  • s

    schrockn

    09/13/2018, 1:07 PM
    want to be able to launch N children
  • s

    schrockn

    09/13/2018, 1:07 PM
    1 for each input
  • s

    schrockn

    09/13/2018, 1:07 PM
    imagine a solid that partitions a file into a bunch of files
Title
s

schrockn

09/13/2018, 1:07 PM
imagine a solid that partitions a file into a bunch of files
View count: 1