Hey guys, i recently upgraded to 0.14.1 and I need use a workspace.yaml for the

dagster-daemon

I am a bit confused on what include in that file. Before i had everything in the dagster.yaml file. If there is an example any where i could take a look at that would very helpful

Has anyone run into an issue (possibly with docker?) where navigating to

localhost:3000

gives an error like the following, but

0.0.0.0:3000

works fine?

@here

@here Can anyone explain the difference between schedule job and partition (with backfill jobs)?

Also the behavior for partitioned jobs is that the scheduler will run future scheduling jobs?

How should I solve this flow? I have an op (

op A

) which does an api call that creates an item. (response is urn) Then I have a sensor which receives events on the progress of calculating the item (which happens outside our control) This sensor can receive events of multiple items. Then I want to continue the job which used

op A

because there is specific data in the job later on. I want to create something like a second op which can wait/ do something on an event by the sensor when the urn matches. I don't want to use polling as this would cause extra delays or resources when you make the polling delay low.

Hello, I am currently trying to utilize dagster in scientific data preprocessing pipelines. I want to reuse the same op multiple times with different parameters each time. I apparently cannot simply pass hard coded arguments to ops within a job definition, therefore i wonder, what is the dagster way of passing different parameters to each instance of the same op?

Hi, How can I run the docker executor in my test? I tried looking into the tests of that package (https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-docker/dagster_docker_tests) but these depend on

dagster-test

which is not published to PyPi so I cannot use this ... Probably there are better ways to launch a docker image in a test. Related to this, my use case is: Our code is already separated in different docker images. How should I best call these docker images? Should I use the docker executor or directly work with the docker client (https://pypi.org/project/docker/)?

Hello, i have a dagster job that needs ephemeral storage (run on kubernetes). however, our kubernetes cluster (openshift 3) doesn't do ephemeral storage so my pod gets evicted. So i want to use a volume for my job. but a kubernetes Job does not support volume claim templates, so i can only specify a fixed volume. So ... i can not run two of these jobs simultaneously. Is there a way to tell the dagster k8s run coordinator to never start two runs of a certain job simultaneously, while still allowing all my other jobs to run concurrently ?

Happy Friday! Working on getting some graphql requests built and can't find the header specification anywhere so I was wondering if someone could point me in the right direction... here's the error that I'm getting currently: "Organization has multiple deployments, but no deployment specified" Attached are the headers I currently have

Hello! Quick question, is there any way to concatenate two DynamicOut (prior or after

collect()

call) of the same type in a job, or is it mandatory to send them into an Op to do so ?

hi - I'm trying to add slack messaging, from dagster-slack. I've been working off the old docs, but I'm not very familiar with dagster yet, so I'm hitting some issues. right now, I'm trying to pass a resource key, after requiring it:

@op(config_schema={"slack": str}, required_resource_keys=set(["slack"]))

(I'm not even sure if that is right at all) but I don't know how to actually 'provide' the key. I thought I could add the slack Oauth token in config, but that doesn't seem to be the case...

In a helm deployment of dagster, what is the appropriate way to mount a file into the gRPC server for a user code deployment? This file would be referenced by a dagster sensor

Feature suggestion to add an option to have python module name used as namespace during AssetGroup.from_package_module or from_package_name. What can I say I am a lazy programmer, lol https://github.com/dagster-io/dagster/issues/6816

Hello, I'm currently trying out partitioning but I could not find any example in the documentation on how to implement 

dagster.PartitionedConfig

 . Did I miss something?

Hello, my run record timing is off by a few hours. The run are trigger by a sensor and the sensor ticks timing is correct. So what determines the timing? My Postgres DB? The grpc that runs the run? The dragsters instance? Digit?

Hello. I want to create a sensor that receives requests from a web service, runs job and returns response to the web service with job ID. How can I do this? I need a sensor that waits for an event, not fired every x seconds.

Hi there, I have a job with many operators. Some of which are heavy and cannot run in parallel, while others are lightweight and can run independently of each other. I would like to limit only the heavy ones from running simultaneously while allowing the other to run as much as possible. How can I do so? Thanks!!!

Hi guys, new here, trying to build a POC with scheduling. I'm reviewing the docs and I'm getting hung up on what's being returned in this schedule function.

@schedule(
    cron_schedule="45 6 * * *",
    job=hello_cereal_job,
    execution_timezone="US/Central",
)
def good_morning_schedule(context):
    date = context.scheduled_execution_time.strftime("%Y-%m-%d")
    return {"ops": {"hello_cereal": {"config": {"date": date}}}}

specifically the last line here -- is this returning a schedule definition for an op in the hello_cereal_job? Would this syntax change if are were more than one op in this job? Thanks!

Hey folks! Is there any way of using a simple memory based io_manager together with multiprocessing execution? The reading and writing to disk between ops creates a huge overhead in our application

Hey everyone! First message here 👋 I'm trying to build a subgraph that would either execute an expensive op if a check passes, or skip it and return the input if the check fails (input/outputs are file paths). I've been looking at "Conditional Branching" but I'm not sure how to merge the two resulting branches (return the output of the op that was executed. I've also been looking to ExpectationResult which sounded close to what I was doing, but I can't figure out how to access the result at runtime. (I've been looking in the expensive op context)

I noticed recently that my Dagster jobs have stopped parallelising ops() when using the MultiProcess executor. I'm using v0.14.1 I configure the executor to autodetect the number of CPUs using 0 as per the docs

execution:
  multiprocess:
    config:
      max_concurrent: 0

Is this a regression? Or intentional change of behaviour?

Hi team, I am trying to execute dagit in my local machine but I am not sure why dagit is looking in a different path of I have in workspace.yaml. (I am using V0.14.1). I really appreciate some help

Hi Team, I just upgraded to latest version of Dagster 0.14.2. When I try to run the legacy

construct_pipeline_with_yaml

it is throwing below error.

File "C:\test\parallel-dag\venv\lib\site-packages\dagster\core\execution\api.py", line 414, in _logged_execute_pipeline
    ) = _check_execute_pipeline_args(
  File "C:\test\parallel-dag\venv\lib\site-packages\dagster\core\execution\api.py", line 910, in _check_execute_pipeline_args
    pipeline_def = pipeline.get_definition()
  File "C:\test\parallel-dag\venv\lib\site-packages\dagster\utils\__init__.py", line 222, in __hash__
    return hash(tuple(self))
TypeError: unhashable type: 'dict'

I upgraded from python 3.7.9 to 3.10 as well. Instead of reading from the yaml file I'm passing a Dict directly and everything is same as Dagster documentation for pipeline dsl.

Hey all, when organizing your jobs via repos, I notice if one job / import is broken, it causes all the jobs and such to be unusable. Is there a configuration around this / practice to where dagster resources are loaded independently from one another?

👋 is there a clean pattern for running a dynamic mapping in job, inan ordered way? basically I want something like

@op(out=DynamicOut(int), config_schema={"start": int, "stop": int)
def generate_things(context) -> int:
    for i in range(context.op_config["start"], context.op_config["stop"]):
        yield DynamicOutput(value=i, mapping_key=f"{i}")

@graph
def graph_of_complex_things(i):
    ... # do graph of things with i

@job
def dynamic_size_ordered_job():
    for i in generate_things():
        graph_of_complex_things(i)

I was poking around and

generate_things().map(graph_of_complex_things)

is the closest thing I can find but isn't ordered

Hi Dagster team! I have a question about logging. I have some dagster code that I am attempting to run in a kubernetes pod using the dagster cli like so

dagster job execute …

. However, the only logs I see in the kubernetes pod is

2022-03-01 03:07:12 +0000 - dagster - DEBUG - run_microdump - 900c13db-7ef0-4f9a-aa86-ca258e2e2ea9 - 7 - RUN_START - Started execution of run for "run_microdump".
2022-03-01 03:07:12 +0000 - dagster - DEBUG - run_microdump - 900c13db-7ef0-4f9a-aa86-ca258e2e2ea9 - 7 - ENGINE_EVENT - Executing steps using multiprocess executor: parent process (pid: 7)
2022-03-01 03:07:12 +0000 - dagster - DEBUG - run_microdump - 900c13db-7ef0-4f9a-aa86-ca258e2e2ea9 - 7 - get_full_copy_tables - ENGINE_EVENT - Launching subprocess for get_full_copy_tables
2022-03-01 03:07:13 +0000 - dagster - DEBUG - run_microdump - 900c13db-7ef0-4f9a-aa86-ca258e2e2ea9 - 10 - get_full_copy_tables - ENGINE_EVENT - Starting initialization of resources [devdb_tables_fetcher, io_manager].
2022-03-01 03:07:13 +0000 - dagster - DEBUG - run_microdump - 900c13db-7ef0-4f9a-aa86-ca258e2e2ea9 - 10 - get_full_copy_tables - ENGINE_EVENT - Finished initialization of resources [devdb_tables_fetcher, io_manager].
2022-03-01 03:07:13 +0000 - dagster - DEBUG - run_microdump - 900c13db-7ef0-4f9a-aa86-ca258e2e2ea9 - 7 - ENGINE_EVENT - Multiprocess executor: parent process exiting after 1.16s (pid: 7)
2022-03-01 03:07:13 +0000 - dagster - DEBUG - run_microdump - 900c13db-7ef0-4f9a-aa86-ca258e2e2ea9 - 7 - RUN_SUCCESS - Finished execution of run for "run_microdump".

and I don’t see logs from my python code itself. I’ve tried logging with both

<http://context.log.info|context.log.info>

and

get_dagster_logger

and both don’t seem to show up. I was wondering if there are any other configurations I should be checking in this case?

Hey guys! We run dagster on k8, just upgraded it last week to v0.13.18, looks great! We’ve ran into a strange issue this morning though that the UI gets stuck for some pages with

upstream request timeout

For example the /runs /status amd /workspace pages worked every now and then after refreshing the browser multiple times but in general they just show a timeout message like

upstream connect error or disconnect/reset before headers. reset reason: connection failure

. Don’t see any strange error logs in kube for either our dagit or our daemon pods. The issue seems intermittent , for a few minutes everything works as normal and then the timeouts return. Any ideas how I can fix this?

https://docs.dagster.io/deployment/run-coordinator writes: For example, you can add the following to your dagster.yaml:

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 25
    tag_concurrency_limits:
      - key: "database"
        value: "redshift"
        limit: 4
      - key: "dagster/backfill"
        limit: 10

but this is a global configuration. Assuming I have different teams with separate pipelines and in particular repositories / workspaces teams might contribute unique new resources in their use cases. For example one team might have a https://github.com/dagster-io/dagster/blob/master/examples/hacker_news_assets/hacker_news_assets/resources/hn_resource.py HN resource and another team a resource for other APIs. How can I merge the configuration from the resources of the different teams to configure the run coordinator? (i.e. if some resources are available only in specific teams) Is it possible to somehow import the resources of teams (and not hardcode it at the startup of dagster in the centralized yml configuration)?

Hi all. I have revised my code to convert a partition schedule job into a scheduled job. On the dagit console, I am able to see that there is a scheduled next time which is my expected run time. But strangely nothing gets run at that time. Can you please look into my code? Also, attaching DAGIT console screenshot.