Hi Everyone. I wanted to know about stdout/err redirection. Was expecting that simple print("test") will appear in the web-ui stdout section but it does not. Is there any kind of decorators/redirections to be used for this? Thank you 😄

Hello! help me please? how to change Serving on http://127.0.0.1:3000 in process 27713 to Serving on http://hostname:3000 in process 27713 (dagit -f hello_world.py )

Hi everyone, I've been trying to make use of Passing Configuration to Multiple Solids in a Pipeline section of https://docs.dagster.io/concepts/configuration/config-schema to try to pass a token as a resource to two separate solids that will need the token. Following the tutorial, I've added

@solid(required_resource_keys={"value"})

to both solids and use them within the solids by

context.resources.value

. I have also added into the pipeline decorator

mode_defs=[ModeDefinition(resource_defs={"value": make_values_resource()})]

and included

run_config={"resources": {"value": {"config": token}}}

in

execute_pipeline

where

token

is my token for both solids in string. Running it with dagit gives me this error

/home/dazza/anaconda3/envs/versedai/lib/python3.8/site-packages/dagster/core/workspace/context.py:504: UserWarning: Error loading repository location gather_pipeline.py:dagster.core.errors.DagsterInvalidDefinitionError: Resource key "value" is required by solid def query_NLP_data, but is not provided by mode "default". In mode "default", provide a resource for key "value", or change "value" to one of the provided resources keys: ['io_manager', 'values'].

What have I done wrong here?

Hi team, I need your help, please. I am very confused with the scheduler and the partitions. I configured my monthly scheduler following the docs but when I look for available partitions I can't see the las two months. How can I update the partition in Dagit? @chris @daniel excuse my dummy question.

Is there a bug with dagstermill? I am unable to get a very simple example to output from a notebook: https://github.com/jeremyadamsfisher/dagstermill-min-failing-example

Hi team, our user code deployment is constantly crashing due to OOM error. The k8s user code pod memory limit is set to 1 GB and the its just running 3 sensors which is calling some external grpc APIs. The memory is constantly increasing and crashing periodically. When I disable all the sensors the memory stays constant. Any recommended way to debug this?

hello! 3 questions please 1 I cant find workspace.yaml to connect my gitlab repository ( my folder cd /home/myuser/.local/bin) https://docs.dagster.io/concepts/repositories-workspaces/repositories 2 I cant start as background (i tryed dagster-daemon run -h 0.0.0.0 -p 3000 ) 2021-09-10 123602 - SchedulerDaemon - INFO - Not checking for any runs since no schedules have been started. 2021-09-10 123602 - BackfillDaemon - INFO - No backfill jobs requested. 3. debian or centos preefer?

Hi, new to Dagster and seem to be struggling with some kind of heisenbug on

0.12.9

does anyone have any hints/tips/corroborative tales? • I have some solid factories that are invoked at the module level, defining some solids (that happen to yield dynamic outputs ) • typically on the first run through, everything works as expected • if I start a brand new run for the same pipeline, or attempt a rerun, or try either of those having fully restarted dagit + dagster-daemon, I get one of the two errors below • I deleted the 'history' and 'storage' (sqlite) directories in the development dagster-home dir, and this problem seems to reset (so I can have another go) nope this was just random luck occasionally pre-run (i.e. run fails at check time)

dagster/check/__init__.py", line 170

dagster.check.CheckError: Invariant failed. Description: pipeline_X has no solid named factory_solid_Y.

occasionally mid-run (i.e. runtime failure)

dagster/core/execution/plan/plan.py", line 684

dagster.core.errors.DagsterSubprocessError: During multiprocess execution errors occurred in child processes:
In process 637582: dagster.core.errors.DagsterExecutionStepNotFoundError: Can not build subset plan from unknown step: factory_solid_Y

this is in a development environment, so using QueuedRunCoordinator, SqliteRunStorage, multiprocess execution repeatedly trying to 'launch execution' or 'rexecute all' from dagit can get me through the first error. might then catch the second error, or might succeed entirely (or get caught on a downstream failure, which is fine) I have a feeling there's something weird happening with the solids being called once on module import. Seems like valid Python to me. Not yet sure how much time & effort it's going to take to go back to basics for a minimal reproduction... by way of the simplest example I can imagine (solid_factory could look like the one in the docs, or even simpler)

@solid
def regular_solid():
    # works as expected
    pass

factory_solid_Y = solid_factory(some_closure_args)  # flaky

Hello team, I am trying to run a calendar in the pipeline to run it every day.

@daily_schedule(

pipeline_name="topic_models_pipeline",

start_date=datetime.datetime(2021, 1, 1),

execution_time=datetime.time(11, 47),

execution_timezone="US/Eastern",

)

def topics_schedule(date):

return {

"solids": {

"load_data": {

"inputs": {"date": {"value": date.strftime("%Y-%m-%d")}}

}

}

}

However, when I run it, it asks me that it must be running the dagster daemon, so I proceed to activate the following command

dagster-daemon run

However, nothing happens, dagit does not seem to detect that it is running, from the demon terminal I get the following output:

2021-09-10 08:46:18 - dagster-daemon - INFO - instance is configured with the following daemons: ['BackfillDaemon', 'SchedulerDaemon', 'SensorDaemon']

2021-09-10 08:46:18 - SensorDaemon - INFO - Not checking for any runs since no sensors have been started.

2021-09-10 08:46:18 - SchedulerDaemon - INFO - Not checking for any runs since no schedules have been started.

2021-09-10 08:46:18 - BackfillDaemon - INFO - No backfill jobs requested.

2021-09-10 08:46:48 - BackfillDaemon - INFO - No backfill jobs requested.

2021-09-10 08:46:48 - SchedulerDaemon - INFO - Not checking for any runs since no schedules have been started.

2021-09-10 08:47:18 - SensorDaemon - INFO - Not checking for any runs since no sensors have been started.

2021-09-10 08:47:18 - BackfillDaemon - INFO - No backfill jobs requested.

2021-09-10 08:47:18 - SchedulerDaemon - INFO - Not checking for any runs since no schedules have been started.

2021-09-10 08:47:48 - BackfillDaemon - INFO - No backfill jobs requested.

2021-09-10 08:47:48 - SchedulerDaemon - INFO - Not checking for any runs since no schedules have been started.

It is important to note that both processes run in a

pipenv environment

, that is, in both terminals, I must start the environment and then run the daemon and in the other start the dagit.

Hey Team, I ran into a problem, as soon as the pipeline start running (through the scheduler) the following error popup:

dagster.core.errors.DagsterInvalidConfigError: Error in config for pipeline topic_models_pipeline

Error 1: Missing required config entry "resources" at the root. Sample config for missing entry: {'resources': {'slack': {'config': {'token': '...'}}}}

But I actually do have those configuration parameters and I can run them through the dagit playground interface clicking in the Launch button. When running the schedule, does this process look for this env variables someplace else?

Quick question: Can you yield multiple runs from a scheduler like you can with a sensor?

does the python logging config (https://docs.dagster.io/concepts/logging/python-logging) apply to dagit / dagster-daemon? or is it strictly for user code?

Hi team, any examples on how to use context managers with resource functions for cleanup? Can I just yield from the resource function?

Hi Everyone, How can I have two pipeline related solid configs in single config.yaml file? eg:

pipeline1_solid:
 config:
	param1:temp
	param2: temp2
pipeline2_solid:
 config:
	param1:temp
	param2: temp2

Basically I want to have individual pipeline solid configuration in single YAML config file.

Good day! Does it exist best practices for organizing Dagster project with pipelines, schedules, repositories and e.t.c? I have found some project with structure like:

__init__.py
repositories.py
solids
    solids.py
pipelines
    pipelines.py
schedules
    schedules.py

Is there any other possibilities?

Hello! Is there any way to set the configuration per workerqueue in the deployment of dagster with celery instead of setting it on the

configSource?

Hi! I am using dbt 0.20.2 along with dagster 0.12.10 and have configured a "dbt run" and "dbt test" as mentioned here: https://docs.dagster.io/integrations/dbt#invoking-multiple-dbt-commands-in-the-same-pipeline . The pipeline runs fine but no asset is getting materialized in dagster. Is there an additional step required to enable asset materialization?

Hey team, I am running into the following error when scheduling ~400 pipelines in Dagster at once (with a max queue size of 20 runs in parallel). I am using v0.11.13. Any ideas?

PIPELINE_FAILURE
Caught an error for run a564083e-8f75-45a9-b9a2-63e5382d5e52 while removing it from the queue. Marking the run as failed and dropping it from the queue: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1631520273.921130387","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3008,"referenced_errors":[{"created":"@1631520273.921127399","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"
>

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/run_coordinator/queued_run_coordinator_daemon.py", line 154, in run_iteration
    self._dequeue_run(instance, run, workspace)
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/run_coordinator/queued_run_coordinator_daemon.py", line 206, in _dequeue_run
    location = workspace.get_location(repository_location_origin)
  File "/usr/local/lib/python3.7/site-packages/dagster/cli/workspace/dynamic_workspace.py", line 36, in get_location
    location = existing_location if existing_location else origin.create_location()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/origin.py", line 255, in create_location
    return GrpcServerRepositoryLocation(self)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 478, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File "/usr/local/lib/python3.7/site-packages/dagster/api/list_repositories.py", line 13, in sync_list_repositories_grpc
    api_client.list_repositories(), (ListRepositoriesResponse, SerializableErrorInfo)
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 143, in list_repositories
    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 89, in _query
    response = getattr(stub, method)(request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)

Hi Team, following this doc https://docs.dagster.io/concepts/solids-pipelines/pipelines#pipeline-dsl I could able to run the pipeline from dagit UI like dagit -f pipe.py from the terminal, but not able to generate the pipeline dynamically into another py file. Is it possible or what I need to do to get the static pipeline generated from the yaml programmatically?

Hi Dagster team, I’m curious about workarounds for this github issue, where I want to use dagster to run several solids within the same pipeline using different docker images. For background, we use

CeleryK8sRunLauncher

which allows us to have resource/compute separation of solids within a pipeline, but seems like dagster doesn’t allow us to have package / environment separation natively.

Hi guys, I think I may be missing something re:

tag_concurrency_limits

. I am trying to control concurrency of runs via tags. I have setup a test locally w/ the docker run launcher where I have been manually triggering the below pipeline at approx the same time via Dagit. While I expect them to run one a time, each run immediately starts. Do I have this test setup correctly?

dagster.yaml

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    tag_concurrency_limits:
      - key: test
        value: value
        limit: 1

repo.py

@solid
def sleep():
    time.sleep(60)

@pipeline(tags={'test': 'value'})
def sleep_graph():
    sleep()

@repository
def repository() -> List[Any]:
    return [
        sleep_graph,
    ]

I’m trying to implement a custom run launcher and I’m running into this error when trying to load my run launcher:

dagster.check.ParameterCheckError: Param "run_launcher" is not a RunLauncher. Got None which is type <class 'NoneType'>.

I’m having a hard time figuring out what I’m doing wrong

hey guys, it looks like below code is blocking — is there a way to make it async, which is to return the RUN_ID early before the whole run is finished successfully?

new_run_id: str = client.submit_pipeline_execution(
        'hello_graph',
        run_config={},
        mode="default",
    )

i think there definitely is a way to do that asynchronously, because the UI is doing that: trigger a pipeline run, navigate to the page of that run with the RUN ID, and then stream the run events

Hi everyone, Need help regarding the location of dagster_home folder. We have used kubernetes for dagster deployment and mounted efs volume to our pod. We are trying to move dagster_home from container to efs volume but it keeps giving graphql error.

hi y'all. I'm super new to Dagster and trying to get my mind around it. I'm trying to run examples from the Dagster repo locally but can't seem to find the right command. Any help would be appreciated.

Can an

asset_sensor

access the asset metadata? I thought perhaps the

asset_event

input would provide an avenue for that, but I think it is of type

EventLogEntry

and that doesn’t have the asset’s metadata accessible as far as I can tell

I think the document here https://docs.dagster.io/concepts/solids-pipelines/pipelines#pipeline-dsl is wrong. How can we pass same solid to itself as input while the actual add_one needs a input. Can someone please explain? Even if I run it using dagit, it shows error for A