https://dagster.io/ logo
Powered by
Title
p

Peter Arwanitis

04/26/2023, 11:55 AM
running this one with 
dagster dev
shows a graph (screenshot in next thread-post) • reason for all those 
Nothing
in
and 
out
is, that the op should simply start and monitor a web-service, and start dependencies when one request finished successfully my problem is how to make 
start_job2
a direct dependency for 
start_job3
and 
start_job5
. atm it creates another instance 
start_job2_2
?
Untitled.py
that's how the dag looks like
ideas or links to examples appreciated I'm on a steep learning curve here and my next step is to make this GraphDSL config-driven 🙂
oh boy .. look at that 🙂 • bit more in/out massage
running latest from launchpad throws for 
job2
a 
op "start_job2" has multiple outputs, but only one output was returned of type <class 'NoneType'>. When using multiple outputs, either yield each output, or return a tuple containing a value for each output.
latest code and now I'm stuck again, as I don't know how to fix the error from running the 
job2
1. because I want to drive this from config in a next step 2. I have to use a op-factory 3. where I don't know how to generically provide multiple result-values (tuple) 4. plan is to return nothing at all (which means it is exactly one 
None
value)
Untitled.py
nice.. first step into programmatic using 
GraphDefinition
reusing the 
op
definitions made in last step, but replacing the 
@job
def run():
and it still renders the DAG and runs it through launchpad 🙂 next stop => op-factory
Untitled.py
and this step now replaced the explicit 
@op(out={"job0_0": Out(Nothing)})
def start_job0(context, **kwargs):
  ...
statements through a factory still works 🙂 -- so next step => run the op-factory configuration driven from yaml (stretch goal => how to create sub-graphs this way?)
Untitled.py
s

sandy

04/26/2023, 4:45 PM
hey Peter - take a look at this example of how to have multiple ops depend on the same op: https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#with-multiple-inputs
p

Peter Arwanitis

04/27/2023, 8:08 AM
hi @sandy thx, and yes it works but translating this example into a "programmatic" approach using `Nothing`s following questions came up: 1. how to translate this to 
Nothing
in and outs? 2. how does this example look like with 
@op(ins: ..., out:...)
annotations? 3. how do you translate this into a a 
dependencies
statement, which fits into 
GraphDefinition
? trying to do the first step I fail already, switching outputs to 
None
and trying to catch variable inputs 
@op
def return_one(context, **kwargs):
    <http://context.log.info|context.log.info>("return_one")
    # return 1

@op
def add_one(context, **kwargs):
    <http://context.log.info|context.log.info>("add_one")
    # return number + 1

@op
def adder(context, **kwargs):
    <http://context.log.info|context.log.info>("adder")
    # return a + b
results in 
dagster._core.errors.DagsterInvalidDefinitionError: In @graph inputs_and_outputs, received too many inputs for invocation add_one. Only 0 defined, received 1
removing the 
**kwargs
, doesn't change it
-- managed the next step to create 
node_defs
and 
dependencies
from configuration :) 
return GraphDefinition(
        name="run",
        node_defs=node_defs,
        dependencies = edges
    ).to_job()
code requires a bit of clean up but happy to share if there is interest
One issue I run into was that the job-rendering fails to layout my graph (which fooled me for a while, as I thought I have a broken implementation) 1. the launchpad > run proofed that all my nodes/edges are as configured (screenshot one) job5 for example has three dependencies 
- name: job5
    edges: [job0, job1, job2]
2. but Deployment > Overview was rendered broken (screenshot 2) 3. removing one node and two edges resulted in a proper layout (screenshot 3) Is this a known limitation or a bug? I don't expect my graph with "7 nodes" and "7 edges" being too complicated?
on a 
dagster dev
restart or reload it works now? sigh 🙂
lovely sometimes it shows the name of the input-slots too • added two more nodes
nodes:
  - name: job7
    edges: []
  - name: job0
    edges: [job7]
    retries: 3
  - name: job1
    edges: [job7]
    retries: 3
  - name: job2
    retries: 2
    edges: [job7]
  - name: job3
    edges: [job1, job2]
    retries: 1
  - name: job4
    edges: [job3]
  - name: job5
    edges: [job0, job1, job2]
  - name: job6
    edges: [job1]
  - name: job8
    edges: [job5, job6]
config for the last one
s

sandy

04/27/2023, 9:54 PM
dagster._core.errors.DagsterInvalidDefinitionError: In @graph inputs_and_outputs, received too many inputs for invocation add_one. Only 0 defined, received 1
https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#defining-nothing-dependencies explains how to set 
ins
to avoid this error
p

Peter Arwanitis

04/27/2023, 11:31 PM
thx, figured it out and get slowly used to ins, out, dependencies and others 🙂 not always sure if it is the best code, but it works and I mostly understand why
#dagster-supportJoin Slack
Powered by