Hi team, I would like to ask for urgent help, please. Suddenly gRPC doesn't work . I am getting this error and the app is blocked. (I am using windows; I think Dagster hates Windows OS. I have never got these RPC issues using Linux)

Hey guys ! I found out that if a Solid from one Pipeline, executes_pipeline another Pipeline, Dagit’s Interface does not render the page to the second pipeline in the UI, not just that, it doesn’t even show Debug logs of the other pipelines, only the first one. Did someone else had this issue ? example :

@pipeline
def pipe1:
   run_pipe2()


@pipeline
def pipe2:
   # printing debug messages

@solid
def run_pipe2():
   execute_pipeline(pipe2)

so pipe1 runs pipe2, but in the UI you won’t see the pipe2 running or printing logs .

Move my thread from random to dagster-support since random seems to be much cooler than this one. 🙂 Hi, I am new to dagster. I currently use scala to create spark jobs and then use airflow to glue them into a dag. These spark jobs run on either EMR or K8S clusters. Some advantages of dagster over airflow are `solid events`and typed interface between tasks in a dag. In my use case, can my spark jobs still leverage these nice features? Thanks for any hints.

@rex we’re having an issue very similar to this: https://github.com/dagster-io/dagster/issues/4487 We have a solid that uses the Python

multiprocessing

library to spawn child processes. Is there something we’re missing? We thought we could use any valid Python in our solids.

How do I use Dagit as a service to show logs? So I have a Flask Server which takes some data from user, creates a pipeline using that data and executes the pipeline using Python API(execute_pipeline()). What I want is to show the Runs tab with logs as is shown in Dagit to the user. I'm new to Dagster(well to Software Engineering itself), so any help will be appreciated:-)

I think I've maxed out the memory on the VM I'm running dagster on to the point that I can't ssh into the server (will run the pipeline container with a limit on its memory consumption in future). Is there a way to kill all running pipelines via API? Looking through docs and failing to find one.

Is there an easy way to map over dynamic inputs when a solid has multiple inputs? Suppose I have a solid which looks like

solid(a, b)

and I have dynamic lists

A

and

B

. Ideally I'd want to do something like

zip(A,B).map(lambda a,b: solid(a,b))

When is planned release date for

0.13

? I am planning to migrate to the new system but I would like to know the timeline

Hi everyone, Is there a way to run schedule as soon as the pipeline gets created?

Can we combine pipelines to another pipeline. Like instead of using solids use 2 pipelines under one pipeline?

Hello Everyone! How to pass resources from YAML file to sensors?

Can some one shed some light here https://docs.dagster.io/concepts/solids-pipelines/pipelines#pipeline-dsl I'm very much interested to try the construct pipeline dependencies from yaml, but the documentation is not clear on how to use it.

Hi all 👋, I am trying out a following approach for installing dagster, I have two namespaces: - dagster-infra - dagster-user I installed dagster daemon & dagster in the 

dagster-infra

 namespace and I installed user deployment & configured job namespace in the 

dagster-user

 namespace. Along with this, I populated Postgresql credentials secret in both the namespace as required. my pipeline example reads environment variable which comes from the secret. for example name of k8s secret my pipeline example needs is 

topsecret

 which is present in the 

dagster-user

 namespace & also updated in dagster-instance configmap as 

env_secrets

 in the dagster-user namespace itself. now with this setup ideally my job pod should get the secret injected but it is not. though it works when I update the dagster-instance configmap from the 

dagster-infra

 namespace which is consumed by daemon with the same 

env_secrets

after doing that I've noticed these secrets gets appended into user deployments as well as jobs. based on this, I've the following questions: • Why does the daemon need to know about secrets? • Why does user deployment gets the secrets injected? • is there any other way to tell daemon that my pipeline needs the given secret?

Hello! Is there a way to pass a pandas dataframe as input to a dagster pipeline? Specifically, I have an Azure function that triggers a pipeline when a storage blob is uploaded, and I’m wondering if there’s a way to “feed” that the storage blob data directly to the pipeline.

Is there a way to implement some sort of control flow for a dag? For instance I might want to include great expectations in a solid, but merely receive a notification when the condition fails so I can investigate, instead of failing the entire run

Hi, is there a way to pass values into to pipeline (from Sensor/Schedules) more easily? The

make_values_resource

feels awkward https://docs.dagster.io/concepts/configuration/config-schema#passing-configuration-to-multiple-solids-in-a-pipeline

Due to our cooperate environment, I can not deploy dagster in docker/k8s, I would like to deploy those on per host basis and operate those on the files of those hosts. is from I've seem run launcher run the pipeline in the repository, this doesn't exactly fit my need, is there a way to have a repository but launch runs on different machine?

Hi team I am getting troubles using composite_solids, I need to pass several dataframes to the downstream solid but I am getting different errors depending of the strategy. I am not sure how to do that and what the message errors mean. @prha Hi, I was wondering if you could give some insights, please. I was looking in the docs without success 🙏

Is it currently possible for

run_status_sensor

to run on the completion of multiple pipelines? From what I have tested, passing multiple pipelines to the

pipeline_selection

parameter will make the sensor function run if any of the selected pipelines completes.

Hit this error

Operation name: JobMetadataQuery

Message: (psycopg2.errors.QueryCanceled) canceling statement due to statement timeout

[SQL: SELECT event_logs.id, event_logs.event 
FROM event_logs 
WHERE event_logs.run_id = %(run_id_1)s ORDER BY event_logs.id ASC
 LIMIT ALL OFFSET %(param_1)s]
[parameters: {'run_id_1': 'ec1ead9e-874a-4de0-b0a8-da5f0b544890', 'param_1': 0}]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

Path: ["pipelineRunsOrError","results",0,"assets"]

Locations: [{"line":60,"column":3}]

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/executor.py", line 452, in resolve_or_error
    return executor.execute(resolve_fn, source, info, **args)
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/executors/sync.py", line 16, in execute
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/schema/pipelines/pipeline.py", line 270, in resolve_assets
    return get_assets_for_run_id(graphene_info, self.run_id)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/fetch_assets.py", line 59, in get_assets_for_run_id
    records = graphene_info.context.instance.all_logs(run_id)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 1013, in all_logs
    return self._event_storage.get_logs_for_run(run_id, of_type=of_type)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/storage/event_log/sql_event_log.py", line 234, in get_logs_for_run
    events_by_id = self.get_logs_for_run_by_log_id(run_id, cursor, of_type)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/storage/event_log/sql_event_log.py", line 201, in get_logs_for_run_by_log_id
    results = conn.execute(query).fetchall()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1263, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 324, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1462, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1815, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1996, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1772, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)

But found the screenshot below when I actually looked through the logs and think the error presented to the Dagit UI was just 'directly caused' by this error.

Seems like a silly question, but how do I specify the

serviceAccountName

for the job pods that get run via the K8sRunLauncher? I’ve tried a number of combinations in the

tags

portion of my pipeline/solids to no avail.

Can I have an example that what's a

RunLauncher

and what's

executioner

?

can we call solid in sensor definition?

Pretty sure you can't...?

I all, i'm new to dagster and i'm trying to use dagster to lunch a job on databricks. I got the extension for databricks but i have no ideia how to use it to create a solid to create a run on databricks from my jar file

should i create a DatabricksJobRunner and then use it to submit a run?

or should i use create_databricks_job_solid

Hi all, I was hoping to get some help with deploying on EC2. I am following the docker-compose example but would like to separate dagit and the user-code on two instances. I believe this should amount to doing a bunch of the docker-compose networking manually. I have run all the services execpt the example pipeline on one instance. Exposing port 80 and 5432 makes the UI and postgres available. On the other instance I have run the pipeline, updating the environment variables and

dagster.yaml

to point to the original instance for the db.

workspace.yaml

on the first instance points to the second on port 4000, which is exposed. Unfortunatly I am still getting inactive grpc channel on the dagit status.

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses" debug_error_string = "{"created":"@1629281721.636797067","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3008,"referenced_errors":[{"created":"@1629281721.636770438","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}" >

Any idea what I'm missing?

Hi team, Is there any way to run pipelines parallel in DAGIT UI using graphQL client. I have a function in which I am creating the graphQL client and submiting pipeline execution. This function is called using multiprocessing. I can see the multiprocessing part working fine but pipeline submit is not happening parallel, it is waiting for previous job to be completed before submitting the new pipeline execution. here is the function: def load_url(p_in_pipeline_name): client = DagsterGraphQLClient('localhost', port_number=5049) try: run_config_info = get_run_config(p_in_pipeline_name=p_in_pipeline_name) #callfunction to get run_config for pipelines new_run_id: str = client.submit_pipeline_execution( p_in_pipeline_name, repository_name='datamax_etl_repository', run_config=run_config_info, mode="default",) status = client.get_run_status(new_run_id) except DagsterGraphQLClientError as exc: print('graphql error.......', l_error) Here is the multiprocesssing function: with concurrent.futures.ProcessPoolExecutor() as executor: futures = [executor.submit(load_url, p_in_pipeline_name=l_pipeline) for i in range(4)] for future in concurrent.futures.as_completed(futures): print(future.result())

Hi team. I am seeing something very weird running backfills. It is normal to have success execution with a critical error? I am running it in red hat REHL 7 server. @daniel hi, I am getting troubles with grpc in linux as well but this error is different. It is looking a path of local windows computer in the linux server how is that posible? (The big deal is this is a hidden error because dagit said the execution is success)

FileNotFoundError: [WinError 2] The system cannot find the file specified

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\host_representation\grpc_server_registry.py", line 177, in _get_grpc_endpoint

server_process = GrpcServerProcess(

File "C:\Users\****\Anaconda3\envs\borrar\lib\site-packages\dagster\grpc\server.py", line 1060, in __init__

self.server_process, self.port = open_server_process_on_dynamic_port(

File "C:\Users\****\Anaconda3\envs\borrar\lib\site-packages\dagster\grpc\server.py", line 1011, in open_server_process_on_dynamic_port

server_process = open_server_process(

File "C:\Users\****\Anaconda3\envs\borrar\lib\site-packages\dagster\grpc\server.py", line 978, in open_server_process

server_process = open_ipc_subprocess(subprocess_args)

File "C:\Users\*****\Anaconda3\envs\borrar\lib\site-packages\dagster\serdes\ipc.py", line 195, in open_ipc_subprocess

return subprocess.Popen(parts, creationflags=creationflags, **kwargs)

File "C:\Users\*****\Anaconda3\envs\borrar\lib\subprocess.py", line 858, in __init__

self._execute_child(args, executable, preexec_fn, close_fds,

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\subprocess.py", line 1311, in _execute_child

hp, ht, pid, tid = _winapi.CreateProcess(executable, args,