Hi Everyone, How do I run mysql queries in solid? How do i establish mysql connection? should I have to use pymysql library or any other library?

Hi, with Solids. Do we often import a whole script and run it within a solid ? I'm trying to understand how we avoid a really big script with lots of solid definitions?

Hi, I have a bit of a convoluted question. I have a pipeline whose end point is a png image. To produce the image I wrote an IOManager whose handle_output method creates and saves that image from the data object producted by the terminal node of my dag. I want the image file name to be of the form "image_{param1}_{param2}.png" where param1 and param2 are the parameterisations to my pipeline that went into producing that data. However, I'm having a hard time getting param1 and param2 into the IOManager. I was hoping that by putting them in the metadata property of the Output object, they would be visible, but that doesn't appear to be the case. I could, I suppose, stick them in the resources, but I'm trying to avoid that. I'm wondering what's the proper paradigm for this.

Hello! I would like to know if there is a way of re-executing a pipeline that failed on dagit with a different configuration within the same tab (and run_id)

Hello All, We have a use-case to create a pipeline dynamically from UI or API reference without the restart of dagster server? Is this possible? (newbie to dagster). Like a button on dagit UI , we pass the required configuration via

API

and create pipeline on spot.

Hello! Need help confirming if gRPC related error is due to company's network as I am ignorant on gRPC stuff. I am usually on VPN due to WFH, but as a check decided to come into the "office" as coworker working on-site is having this gRPC error.

Hey all, another newbie question here.... from what I can tell, any code that is being used to generate pipelines for

dagster

would need to have

dagster

installed. For instance, I have a bunch of classes written to access

rclone

endpoints as part of a larger

docker app

. I assume, to write pipelines for that framework, I would have to have all of

dagster

included in the list of dependencies when building that product?

not sure how big the

dagster

install is, and this seems obvious, but I assume this would bloat any tooling we are writing. it also makes that code directly dependent on

dagster

Hey y'all. Seeing something fucky with sensors. I have a sensor function (which returns a generator) decorated with

@sensor

, and I have the same function (copy/paste) passed into an explicit

SensorDefinition

call as the

evaluation_fn

. But the decorated function is working, whereas the

SensorDefinition

instance I'm constructing is not doing anything. Specifically, it sets up and runs as a sensor, but it's not invoking the

evaluation_fn

as far as I can tell. Any idea what I need to do before calling

SensorDefinition

if I'm not able to use

@sensor

? Would be super helpful 🙂

Hey all. I am working through setting up a parametric pipeline, where the user can pass a dictionary, typically of

str

, but it seems that just passing a dict to a pipeline is way out of line with the core logic of `dagster`s resources` ... seems there has to be a shorter path than something like this?:

@solid(
    config_schema={'endpoint' :str, 'collection': str, 'asset_type': str, 'asset_name':str},
    required_resource_keys={'endpoint', 'collection', 'asset_type', 'asset_name'}
    )
def get_path(context) -> str:

    endpoint = context.resources.endpoint
    collection = context.resources.collection
    asset_type = context.resources.asset_type
    asset_name = context.resources.asset_name

    path = f'{endpoint}/{collection}/{asset_type}/{asset_name}'
    return path

@resource(config_schema=str)
def endpoint(init_context):
    return init_context.resource_config

@resource(config_schema=str)
def collection(init_context):
    return init_context.resource_config

@resource(config_schema=str)
def asset_type(init_context):
    return init_context.resource_config

@resource(config_schema=str)
def asset_name(init_context):
    return init_context.resource_config

@pipeline(
    mode_defs=[ModeDefinition(
        resource_defs=
            {
                'endpoint': endpoint,
                'collection': collection,
                'asset_type': asset_type,
                'asset_name': asset_name,

            }
        )
    ]
)
def create():
    path = get_path()
....

ideally,

dagster

will be running as a background service, where end users are simply passing arguments to some kind of

cli

. Are there methods or practices within

dagster

that lower the front-end complexity such that we don't have to map every argument so explicitly? This is especially necessary for our tooling since the methods will always required user input for every run and will not likely be tied to any scheduler

Hi there. We're having issues with our pipelines: as it seems, pipelines randomly fail during execution, and it just freezes the whole thing, all schedules stop working and, although we've set up hooks for Discord notifications, we get nothing when this happens, and we're forced to restart our containers. The error it's showing is the following:

Caught an error for run 49895d5a-ad57-494a-a310-ce6115cdc193 while removing it from the queue. Marking the run as failed and dropping it from the queue: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1623854325.910854270","description":"Error received from peer ipv4:172.19.0.2:4000","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

Stack Trace:
  File "/usr/local/lib/python3.8/site-packages/dagster/daemon/run_coordinator/queued_run_coordinator_daemon.py", line 154, in run_iteration
    self._dequeue_run(instance, run, workspace)
  File "/usr/local/lib/python3.8/site-packages/dagster/daemon/run_coordinator/queued_run_coordinator_daemon.py", line 206, in _dequeue_run
    location = workspace.get_location(repository_location_origin)
  File "/usr/local/lib/python3.8/site-packages/dagster/cli/workspace/dynamic_workspace.py", line 36, in get_location
    location = existing_location if existing_location else origin.create_location()
  File "/usr/local/lib/python3.8/site-packages/dagster/core/host_representation/origin.py", line 251, in create_location
    return GrpcServerRepositoryLocation(self)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/host_representation/repository_location.py", line 478, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File "/usr/local/lib/python3.8/site-packages/dagster/api/list_repositories.py", line 13, in sync_list_repositories_grpc
    api_client.list_repositories(), (ListRepositoriesResponse, SerializableErrorInfo)
  File "/usr/local/lib/python3.8/site-packages/dagster/grpc/client.py", line 143, in list_repositories
    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)
  File "/usr/local/lib/python3.8/site-packages/dagster/grpc/client.py", line 89, in _query
    response = getattr(stub, method)(request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)

If you need further information about our setup, I'll be more than happy to share them. Thanks in advance!

If I have a solid that calls

<http://context.log.info|context.log.info>("...")

then the message shows up in the pipeline run log in Dagit. However the solid is also making calls into legacy python code that uses the standard

logging

module, and those logs don’t show up. Is there a way to make those logs visible in Dagit (without having to modify the legacy code to use

context.log

)?

Hi team, I am trying to debug an issue with my sensors and I added some

print

statement with in the sensors. But I am not able to find them in the

dagster-daemon

logs. Is there any other place that I need to look into to find the

stdout

logs from sensors?

Hi all, how do we use part of our Solid Input definition into our Output definition? The output definition has no context to draw from what I understand?

Hi all! I was wondering if Dagster supports dynamic discovery of user deployments in Kubernetes, e.g. based on annotations on a pod or service exposing the gRPC API? I've read through the docs on how to separate the Dagster infrastructure and user code, but that approach currently requires specifying the hosts and ports of the user code deployments as part of the Dagster deployment (see https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#separately-deploying-dagster-infrastructure-and-user-code)

Hi all, I have received this error message below. Do you know what this is and where to find the meaning of the different error codes? Thanks

[DagsterApiServer] Pipeline execution process for xxx unexpectedly exited with exit code -11

Hello, everyone. This is my first question here, so I'd like to say thanks for developing this great tool. I'd like to know if there is any way to access the metadata produced by dagster (and displayed as the output). The regular dagster output goes like this: 2021-06-17 151230 - dagster - DEBUG - my_pipeline- 47989433-702c-4246-9c8d-ab4c8bab4be6 - 13936 - merge_transformations - LOADED_INPUT - Loaded input "clean_daag_df" using input manager "io_manager", from output "result" of step "clean_dzag" [...] 2021-06-17 151230 - dagster - DEBUG - my_pipeline - 47989433-702c-4246-9c8d-ab4c8bab4be6 - 13936 - merge_transformations - STEP_SUCCESS - Finished execution of step "merge_transformations" in 98ms. I was wondering, if I can get this information, specially start and finish time (but also pipeline run id and solid execution id, if possible), in order to transfer it to a file or to a database. Thanks in advance.

Has anyone run into an issue where a pipeline and its solids are showing up in regular Dagit but not in read-only Dagit? It is happening on my end and I haven’t found any obvious reasons for this behavior yet. This is on

0.11.11

.

Hi, I am trying to define a resource for a container registry URL (we have a dev and prod registry). I see that there is a

hardcoded_resource

, which looks to be exactly what I want, but from the docs, it looks like it is meant to be used for mocks in testing. Is there a recommended approach for this?

Hi, I'm hitting GRPC size limits when executing pipelines from a sensor. The sensor collect ~4k records of 500bytes/ea from which it yields a run. Going too much over 4k gives the error. Is there anyway to use IOManager to workaround this issue? eg materialise assets using an IOManager in the sensor and have the first solid in the pipeline pick up from that rather than sending all the data through grpc? cheers

Hi there, I am yielding 2 objects from solids as the dagster 'Output' class. But in my custom IO_Manager, it reads a single 'obj'. Does yielding 2 Outputs come back as a collection? If so, do we access it by name, or by index? Thanks

Hi all - I haven’t changed any code and am deploying with Docker, but I’m seeing

ModuleNotFoundError: No module named 'dagster.core.definitions.event_metadata'

on today’s releases - is anyone else having the same issue?

Hi, is there a way to set configs via .yaml files to `RunRequest`’s

run_config

and Dagit’s playground or the way to go is using

Presets

?

Hi everyone! I have a question. How does Dagster help improve Data quality ?

Hey all, I know it's the weekend, so no need to respond atm. I was just poking around with the

docker_deploy

example from the

dagster

repo and was wondering about how one might send commands to a containerized application running on

dagster

? As in, I will have users that want to run a pipeline from their local terminal with some arguments:

{'my_arg': 5}

to say a service called

my_service

, which is containerized and available to

dagit

... are there any docs on how to forward those instructions? The assumption here would be that they are not connected to / have access to the server side

dagit

, ie - some basic

cli

or other

client

for input

Hey all... running into issues with getting a simple toy docker deployment to work... I am starting with the

docker_deploy

example from the latest release, but even though the

docker-compose up

command seems to launch everything, my new pipeline doesn't seem to show up in the available repos:

My repo and pipeline definitions in

repo.py

seem to be fine, as

dagster

doesn't complain about them when running

docker-compose -f docker-compose.yml up

, but I must be missing something about how the mapping between the

workspace.yaml

and where

dagster

finds the docker images / repos? Currently, the

repo.py

for

beekeeper_pipelines

is copied over to

/opt/dagster/app

in it's container, and the

workdir

is set appropiately Also, I'm assuming that adding new containers to the running

dagit

/

dagster daemon

is as simple as starting the

new_service

container with the proper args and joining the proper docker network? ( without having to add the new service to this example compose file, rebuild and up? )

- keep running instance of dagster/dagit
- pull new_repo container
- start new_repo container on the dagster docker network
- add new_repo container to workspace.yaml
- refresh repos within dagit

??

Basic Question: Is User code deployment a deployment abstraction for a repository? And consequently, can use code deployment have only one repository