19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#announcements
- s
Slackbot
10/02/2020, 8:17 PMThis message was deleted.m- 2
- 1
- m
max
10/02/2020, 8:22 PMsd- 3
- 3
- e
Eric Bellet
10/02/2020, 9:42 PMHi guys, I am discovering Dagster and I am excited about it, nice work! I am looking the doc https://docs.dagster.io/tutorial/basics_pipelines and the last lines of codes are incomplete, hopefully than can help youCopy codeassert result.success def test_complex_pipeline(): res = execute_pipeline(complex_pipeline) assert res.success assert len(res.solid_result_list) == 4 for solid_res in res.solid_result_list:sc- 3
- 2
- m
max
10/05/2020, 6:14 PMHi all! Our monthly community meeting is coming up (October 13th) and we'd like to invite interested community members to present on what they're doing with Dagster. If you have something neat in the works and you'd like to show and tell (~10 minutes), please DM me. dagstir - a
AXue
10/06/2020, 12:45 AMHi team, I’m quite a new user here. Not sure if this is the right place to ask these questions, what would be a dagster-recommended way to repeat a pipeline with different inputs, once the previous run has succeeded? Any pointers/docs/articles is appreciated! to be more specific, this is what I’m facing 😄Copy codeMessage: (DagsterInvariantViolationError) - dagster.core.errors.DagsterInvariantViolationError: Attempted to iterate over an InvokedSolidOutputHandle. This object represents the output "xxxxx" from the solid "xxxxxx". Consider yielding multiple Outputs if you seek to pass different parts of this output to different solids.m- 2
- 2
- a
Auster Cid
10/06/2020, 3:16 PMHi guys, I'm having trouble loading the partition graphs on a schedule page that has a lot of executions. It makes Dagit unresponsive until I get a timeout.sb- 3
- 14
- y
Yichen
10/07/2020, 7:57 PMHi all! Our second Dagster Community Meeting is on Tuesday, October 13th at 9 am Pacific Time (UTC-7). We’ll hear from members of the Dagster core team discussing some of Dagster’s features, and discuss deployment strategies for Dagster. If you haven't registered before, please DM me your email or fill out this form to be added to the calendar invite: https://forms.gle/K2dni4iPq48GnUEq6. If you aren’t able to make it, we’ll share a recording afterward. Here’s a preview of the agenda for the second meeting: https://docs.google.com/document/d/17wvfmyOLpADLnkoSnG9kmyDuyuvlz4suCNPKk2L2xHs/edit?usp=sharing🎉 3👋 1👍 1 - u
user
10/08/2020, 8:58 PMalangenfeld just published a new version: 0.9.14. - a
alex
10/08/2020, 9:00 PMmlb- 4
- 6
- b
Berkay Öztürk
10/10/2020, 4:32 AMHi, what are common ways to trigger pipelines externaly? I wish there was a REST API like the one in Airflow 2.d- 2
- 1
- n
Noah K
10/10/2020, 11:35 AMDoes anyone have a working example of dagster_celery working? It appears that the default worker and executor configs disagrees on the queue arguments (specifically x-max-priority) which causes celery to crash out.d- 2
- 3
- n
Noah K
10/10/2020, 12:00 PMIs the config file path for dagster_celery actually connected to anything? It doesn't seem to complain even with completely invalid YAML - n
Noah K
10/11/2020, 1:43 AMAre there any docs on how pipeline planning actually works when using @pipeline? - n
Noah K
10/11/2020, 1:43 AMSpecifically re: using remote executors, how does the code for the pipeline end up in a place the executor can run it?d- 2
- 2
- n
Noah K
10/12/2020, 1:20 AMWhat's the right way to set up a parameterized pipeline? I could set a solid_config for the underlying solids but that seems like it would get complex. Can I pass in resources at run time?s- 2
- 2
- n
Noah K
10/12/2020, 1:21 AMLike I have a pipline which is "process file" and I want to pass in path which then gets used in lots of solids. - n
Noah K
10/12/2020, 1:21 AMSo I can then launch a copy for each file in my bucket. - n
Noah K
10/12/2020, 1:35 AMComposite solid, probably? - l
Leor
10/12/2020, 1:35 AMrun_config/presets? - n
Noah K
10/12/2020, 1:37 AMI would think of a preset as for something with only a few values - n
Noah K
10/12/2020, 1:38 AMThis is an arbitrary input parameter - l
Leor
10/12/2020, 1:38 AMrun_config, then - n
Noah K
10/12/2020, 1:42 AMBut isn't that set per solid? - f
Frederic Abiven
10/12/2020, 1:42 AM - n
Noah K
10/12/2020, 1:42 AMSo either I use a composite solid, or I just set the selector to
?* - f
Frederic Abiven
10/12/2020, 1:42 AMI created a mode where I input a csv file as a resource for the pipeline - n
Noah K
10/12/2020, 1:44 AMYeah, that seems the easier way - n
Noah K
10/12/2020, 1:44 AMCopy code@resource def path(ctx): return ctx.resource_config['path'] - n
Noah K
10/12/2020, 1:45 AMSomething like that?f- 2
- 14
- n
Noah K
10/12/2020, 2:04 AMsolid_config with selector *, composite solid with internal plumbing for a single solid_config entry, or the wonky shim resource I showed above.d- 2
- 3