https://dagster.io/ logo
Powered by
Title
p

Peter Arwanitis

04/23/2023, 5:35 PM
== MOSTLY SOLVED 🙂 == feeling very lost 😒adpanda: getting my first baby steps done with Dagster I love the concept, but the documentation fools me to provide help how to add yaml-configurations with sections like: 
ops:
  '*':
    config:
      base_url: '<http://localhost:8000>'
  monitor_web_service:
    config:
      default_max_retries: 3
which I then plan to use like this 
@op(config_schema={
    'default_max_retries': int,
    'base_url': Field(StringSource)
})
def monitor_web_service(context, job_name: str, job_id: str, max_retries: Optional[int]):
    # from dagster.yaml
    default_max_retries = context.op_config['default_max_retries']
    base_url = context.op_config['base_url']
my concrete problem is: • where to add this configuration or reference the yaml file? • in 
dagster.yaml
=> no, in 
workspace.yanl
=> I don't think so either • on commandline using 
dagster dev
there is no 
-c config.yaml
support how? Where is the documentation explaining all config options in detail (dagster / workspace / custom ..)? thank you for pointers! 🙂
s

sandy

04/24/2023, 4:36 PM
Hi Peter - do you mind linking to the documentation? This is a common misconception, and we'd like to update it to help address it. The config that gets attached to an op is run config, which means that it's typically meant to be specified at the time you're launching a run. You can specify that config in the Launchpad when launching a run, or as an argument to Dagster's APIs for launching runs. It sounds like in your case, you want it to apply to all runs? Is that right? If that's the case, would it make sense to hardcode it?
p

Peter Arwanitis

04/25/2023, 10:00 PM
hi @sandy
do you mind linking to the documentation?
that is actually one of my problems 🙂 I can see from examples, that 
dagster.yaml
, 
workspace.yaml
exists • but doc search lead to this, where I'm looking for a doc explaining the purpose and all features • same applies to 
workspace.yaml
=> purpose, supported fields? solution: • a full featured version of the system yamls might be good, with inline comments and in the mean-time I understood that there is a config, which can be added for "run time" in launchpad (or programmatically) • the "Run Configuration" doc section (link) is ok
I'm hooked by the "*GraphDSL*" chapter, as I'm want to create a graph programmatically from a yaml • having one `op-`factory creating a 
start_webservice
with parameters • with dependencies supporting fan-in and -out • the op starts a webservice and waits until completion • when it ends a depended 
op
in the graph can be started but the doc-chapter config and implementation is a custom implementation • which I'm not able to extend to the degree of creating 
op
from a factory • which I can connect (dependencies) in a graph like so the variable number of input-dependencies is killing me as 
@op(ins={"start": *In*(Nothing)})
seems to be needed, but don't support fan-in? sry, sounds very confusing me trying to abuse(?) dagster? And it is the third night me experimenting (and treating chatgpt as a source of inspiration, which fails as it is mostly stuck with old 0.12 versions) 
api:
  base_url: '<http://localhost:8000>'
  default_max_retries: 3
ops:
  - name: job1
    depends_on: []
    max_retries: 3
  - name: job2
    max_retries: 2
  - name: job3
    depends_on: [job1, job2]
    max_retries: 1
  - name: job4
    depends_on: [job3]
    max_retries: 0
hi @sandy FYI I've put the discussion, my journey of the GraphDSL exercise into a separate thread here made two steps back, and first try to create it with explicit ops
#dagster-supportJoin Slack
Powered by