Anyone know why the parent process is exiting? No other info provided

Hi all, I need to return a specific object that cannot be pickled from an Op for other Ops to use later down the job, we do not have control over the object itself (it's from an external library), and must be the same object for the other ops to use, what would be the best way to solve this?

I have integrated dagster_pandas and defined a DataFrame type. Is there any way to get richer error reporting, e.g. which row & column has the issue? All I see is this:

2021-11-18 11:00:12 - dagster - DEBUG - load_uniswap_trades - 19832c79-3c58-4909-ac08-fb6f22dbbe24 - 47605 - load_uniswap_v2_trades - STEP_OUTPUT - Yielded output "result" of type "UniswapV2TradeDataFrame". Warning! Type check failed.

Can you specify intraday dependencies on the partitioned jobs by date stuff?

Like this node depends on 40 days of some other node ?

Hi friends! When developing pipeline on a local machine how do you guys recommend working on two branches at once. For example, let’s say on branch one, we make some edits. Then we kick off the job which takes an hour to run. While that job is running, I want to work on something else in the same dagster repo. So I make branch 2 off master. But this’ll mess up the run happening based off branch one. I figured one way to do it is to spin up multiple instances of dagit pointed to multiple dagster home and two git repos checked out. But idk if there’s a more first class way?

Is there a way to use

Selector

in conjunction with other fields? Like say I wanted to accept:

date: YYYY-MM-DD
jar: some_string

and

date: YYYY-MM-DD
jar_path: some dict

as both valid configs, but I only want

jar

or

jar_path

specified, not both. And I would want to require one of them The example in the docs doesn’t use

Selector

with other fields

Mocking

OpDefinition

I'm writing a test for a

failure_hook

, which relies on

context.op.op_name

. So when I'm testing the hook I need to build a

HookContext

with an

op

. Can anyone suggest a good way to mock an

OpDefinition

that I can pass to

build_hook_context

?

Adding Additional Context to Log Messages I plan to have several instances of

dagit

running, each running jobs from several (remote) gRPC repos. All run workers will submit structured log messages to Google Cloud Logging, where I will monitor (dashboards) and alert for all jobs. I need a way to distinguish between log messages on the basis of the

dagit

instance that ran the job, i.e. I'd like to inject same fairly static info into each log message like a human-friendly instance ID. I created a

logging.Formatter

which structures and partially serializes the

logging.LogRecord

and also created a

logging.Handler

which wraps a

google.cloud.logging.Client

. I configure the logging in

dagster.yaml

like so

python_logs:
  dagster_handler_config:
    handlers:
      myHandler:
        class: my_logging.google_cloud_logging.Handler
        level: DEBUG
        formatter: myFormatter
    formatters:
      myFormatter:
        class: my_logging.google_cloud_logging.Formatter

My thought was to extend my formatter to take additional "static" values which could be injected into the log record and this will work find for now. But I'm keen to hear your thoughts on how to achieve this more gracefully. Perhaps somehow using [logging.LoggerAdapter](https://docs.python.org/3/library/logging.html#loggeradapter-objects).

Hello guys , kinda stuck trying out dagtser first time. I am trying to pass a resource to a job, so it can be accessible to the ops but I'm getting

dagster.core.errors.DagsterInvalidConfigError: Error in config for job
    Error 1: Missing required config entry "resources" at the root.

I already have config_schema specified on the resource, Do I need to specify another config for the job/graph or what am i missing ? here is a stackoverflow link with the code

<https://stackoverflow.com/q/70024709/6473175>

hi guys, just want to check on what is the best practice on using sensors. here is my use case... 1. call and api to generate a report 2. poll another status api with reportId returned from the first api... waiting for success status 3. call third api to get report data my idea initially was to have the following: 1. Op to call api to generate report and output reportId 2. sensor to receive api and start polling and then generate a runRequest for next Op 3. Op receive runRequest and download report however from what im seeing from the doc, its not possible to use a sensor within a job... they are independent of any jobs.. which means i cant recreate the sensor dynamically with the specific reportId on runtime... any idea how i can solve this issue?

Hi all, i'm facing an issue that when I run a job of dagster lots of

/usr/bin/python3 -c from multiprocessing.spawn

multiprocessing.semaphore_tracker

processes start running and system got overload and with only 8 concurrent jobs my 16 cores/64 ram systemd overlods, in fact i'm not importing

multiprocessing

this library in my code please help on it.

I see in the docs that in the newer versions of Dagster, you can specify config at the job and graph level instead of just at the op level. Does anyone have an example of what this looks like?

Hey all! Just getting started with Dagster, and I'd like to have 1 Workspace with 3 Repos (one for each of our regions). I get how to do that part, but is there a way I can have a shared folder for

ops

etc? Each region uses common systems we're pulling data from but they're on different instances with their own configs. I'd like to avoid redundancy as much as possible, so does anyone have advice on the best way to approach this?

Am I going crazy or there was a way to override run config on the CLI without a YAML file? I have a single config “overwrite” for a single solid and it seems a bit overkill to have to declare a whole new run config dict just to change that. Ideally I could set it to

False

in the pipeline declaration and just add

-c {"solids": {"my_solid": {"config": True}}}

when calling

dagster pipeline execute

(as you can see I didn’t move to 0.13 yet)…

Hi everyone, when we deploy a new sensor for the first time, it is automatically in the off state and we need to manually go into dagit and turn on the sensor. Is there a way when deploying a new sensor to ensure it is deployed and active? Or will we have to manually turn the sensor on whenever a new one is deployed?

Hi! I have the following problem — I have two graphs that fetch data from a few data sources for many graphs downstream. One job takes a few hours, but can start at 3AM. The second one is quick but can only start after 7AM. I would like to combine all graphs into a big daily super-graph, but i can't come up with an elegant solution of starting parts of this super-graph at different times. Any ideas? I considered just sleeping until 7AM in the second job, or requesting a restart after

7AM - time()

seconds. What bothers me with these is the unnecessary coupling of getting the data and scheduling.

Being able to run from checkpoints / data stored in s3 via the io_manager is extremely helpful for local dev work. But I often want to break a solid / op down into smaller chunks which then means I can't run from the previous failure. Anybody got a hacky solution to that?

Hey all, looking for a way to be able to pass **kwargs to a op. I see the documentation references passing kwargs for Resources. Maybe I missed it somewhere ?

🔍 📉 Inactivity Detection It's easy to know when a job ran and had an error - error logging, [job failure sensor](https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#job-failure-sensor). But it's not so easy to know when a job that should have run, didn't. This is a problem that I've experienced in the past and the only good way I've found to solve it is to detect for some signal dropping below some threshold and alert when that happens - inactivity detection, a sort of "dead man's switch". I've achieved this in the past by creating metrics based on log messages. If the metric falls to zero, then inactivity can be assumed. But what if the schedule for that job is sparse? Like once a day. And the job is mission critical... 💥 Maybe it's jobs watching the jobs? But what if those jobs fail? I would be very interested to hear your ideas on how to solve this one.

May have found a run status sensor buglet I have a run_status_sensor that is correctly triggered on completion.:

@run_status_sensor(pipeline_run_status=PipelineRunStatus.SUCCESS)
def sensor_on_run_success(context: RunStatusSensorContext):
    print(f"Sensing Success of pipeline: {context.pipeline_run.pipeline_name}")

The logs however show it as “failure”:

2021-11-22 12:44:38 - SensorDaemon - INFO - Completed a reaction request for run fa8e1c61-71f6-40f1-a932-e95ef472c802: Sensor "sensor_on_run_success" processed failure of run fa8e1c61-71f6-40f1-a932-e95ef472c802.

So the sensor works correctly but the log is wrong.

hi all, i have changed my compute logs configurations to:

compute_logs:
  module: dagster.core.storage.noop_compute_log_manager
  class: NoOpComputeLogManager

but still, they are saved to storage directory in ~/dagster_home/…… same happened with “S3ComputeLogManager” my dagster.yaml is:

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 200
    tag_concurrency_limits:
      - key: "type"
        value:
          applyLimitPerUniqueValue: True
        limit: 16
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username: 
      password: 
      hostname: "127.0.0.1"
      db_name:  "dagster"
      port: 5432
event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      username: 
      password: 
      db_name: dagster
      hostname: 127.0.0.1
      port: 5432
schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      username: 
      password: 
      db_name: dagster
      hostname: 127.0.0.1
      port: 5432
#compute_logs:
#  module: dagster_aws.s3.compute_log_manager
#  class: S3ComputeLogManager
#  config:
#    bucket: "dagster-compute-logs"
#    prefix: "dagster-test-"
compute_logs:
  module: dagster.core.storage.noop_compute_log_manager
  class: NoOpComputeLogManager

schedule creates containers, how to prevent them from being created or deleted automatically after completion?

How can I re-use an old sqlite intermediate storage when developing locally via dagit? See below each new sqlite db for each time I've booted up dagit.

I found a bug in the EMR runner in

dagster-aws

with how tags are set on the cluster - https://github.com/dagster-io/dagster/pull/5755 should fix it

Hello, I have setup dagit (on ECS service) behind an AWS ALB with OpenID authentication. This uses a cookie managed by the ALB and I currently have a 5min session expiry. This mostly working, but after 5 minutes dagit is making heartbeat type async graphql calls which fail unless I refresh the page. Going directly to the graphql endpoint without first being authenticated seems to work - it may be related to CORS? In any case, I am new to using ALB with OpenID and appreciate if anyone has tips on the issue (I could set the session expiry to a much higher time, but prefer not to).

Hey y'all - I've set up my

dagster.yaml

(attached) to include specifications for the run_launcher, run_storage, event_log_storage, schedule_storage, and compute_logs... but I still end up with a local folder called

storage

containing a folder named with the run ID in turn containing folders named for each op. Is there a config I'm missing that affects this?

Is it possible to do Linear Dependencies without return?

from dagster import job, op


@op
def return_one(context):
    pass


@op
def add_one(context):
    pass


@job
def linear():
    add_one(add_one(add_one(return_one())))

Hello Everyone, Can someone help me to setup the Dagster in my local environment ?

Hi all, I want to know that what should be happened with the running jobs if I restart the daemon? Currently when I restart the dagster some time jobs autometically restart some time got cancelled then I need to restart them manually, so can anyone please tell me what should be the expected behaviour?