https://dagster.io/ logo
Join the conversationJoin Slack
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Powered by Linen
announcements
  • s

    schrockn

    11/28/2018, 11:46 PM
    lol how did i miss this
  • b

    Ben Gotow

    11/28/2018, 11:46 PM
    yeah my favorite part is that it’s so complicated they made a whole visualization tool where you can watch the candy move around
  • f

    freiksenet

    11/29/2018, 8:08 AM
    lol, is there really? i'm not surprised to be honest
  • b

    Ben Gotow

    11/29/2018, 7:11 PM
    yeah check this out—http://rxmarbles.com/#combineLatest you can drag the input candy around and it actually updates the output candy live
  • b

    Ben Gotow

    11/29/2018, 7:11 PM
    haha and it’s /still/ confusing…
  • b

    Ben Gotow

    11/29/2018, 11:39 PM
    hey folks—been playing around with the config editor a bunch, what do we think about auto-populating the config template a bit further? I’d really like to be able to add a solid, and upon autocompleting the solid name, get a complete solid config block I can fill in - it’s super hard to remember the names without the type explorer also on the page
  • s

    schrockn

    11/29/2018, 11:39 PM
    ya i think that would be cool
  • b

    Ben Gotow

    11/29/2018, 11:39 PM
    I want to go from
  • s

    schrockn

    11/29/2018, 11:39 PM
    so i implemented some scaffolding
  • b

    Ben Gotow

    11/29/2018, 11:39 PM
  • s

    schrockn

    11/29/2018, 11:39 PM
    in the dagster CLI
  • b

    Ben Gotow

    11/29/2018, 11:39 PM
    directly to
  • b

    Ben Gotow

    11/29/2018, 11:40 PM
  • s

    schrockn

    11/29/2018, 11:40 PM
    ya
  • s

    schrockn

    11/29/2018, 11:40 PM
    so i think this is subtle
  • s

    schrockn

    11/29/2018, 11:40 PM
    there are going to be cases of “sparse” config
  • s

    schrockn

    11/29/2018, 11:40 PM
    where we import libraries of “resources” that schematize their entire config space
  • s

    schrockn

    11/29/2018, 11:41 PM
    in which case scaffolding would not be useful
  • s

    schrockn

    11/29/2018, 11:41 PM
    and harmful
  • b

    Ben Gotow

    11/29/2018, 11:41 PM
    hmm - that’s true… I don’t think we want to autocomplete the blocks for all config, just for solids where once you specify one var you’re required to provide the entire block?
  • b

    Ben Gotow

    11/29/2018, 11:42 PM
    (or is that not always required?)
  • s

    schrockn

    11/29/2018, 11:42 PM
    (dagster) [schrockn@mbp ~/code/dagster/python_modules/dagster/dagster/dagster_examples (schrockn/execution-from-dagit)]$ dagster pipeline scaffold_config pandas_hello_world
    context:
      default:
        config:
          log_level: ''
    execution:
      serialize_intermediates: true
    expectations:
      evaluate: true
    solids:
      load_num_csv:
        config:
          path: path/to/something
  • s

    schrockn

    11/29/2018, 11:42 PM
    so this is the current result of scaffodling the config
  • s

    schrockn

    11/29/2018, 11:42 PM
    on the command line
  • s

    schrockn

    11/29/2018, 11:42 PM
    this is manageable, for now
  • s

    schrockn

    11/29/2018, 11:42 PM
    but
  • s

    schrockn

    11/29/2018, 11:43 PM
    imagine a much much more complicated config section that is system-specified
  • s

    schrockn

    11/29/2018, 11:43 PM
    or whatever
  • s

    schrockn

    11/29/2018, 11:43 PM
    like under “execution” there is only “serialize_intermediates”
  • s

    schrockn

    11/29/2018, 11:43 PM
    but i can imagine dozens of optiosn with complicated nesting as we move forward
Powered by Linen
Title
s

schrockn

11/29/2018, 11:43 PM
but i can imagine dozens of optiosn with complicated nesting as we move forward
View count: 3