I’m having trouble passing some information between solids.

solid_A

fetches some data from an API, and uses an IO Manager that persists the data to a JSON file in GCS. The path at which to store the file is specified as config for IO Manager as described in the docs here.

solid_B

then loads that file from GCS to BigQuery, but I’m not sure how to pass the GCS URI between the two solids. I can’t have

solid_B

use the same IO Manager and have the IO Manager’s

load_input

handle locating the file because I don’t want to “load” it to perform computation. Is there a way for

solid_A

to access that IO manager config to return it as on output? Also, is there a better way I could model this series of tasks?

Hi, When will the version in conda-forge be updated to 0.10.9? currently it is 0.10.8 https://anaconda.org/conda-forge/dagster/files

Hi, first of all congratulations to that amazing framework! While building up an environment based on the deploy_docker example, I ran into the following problem: The pipeline being launched in a separate container via DockerRunLauncher needs access to volumes to manipulate data being present on the host machine. While the container running the gRPC-Server (using the same image as the worker-container) has access to the volumes (specified in the docker-compose.yaml), the launched worker-container does not. Is there a way to enable/specify volumes for the pipeline? What would be a possible solution for that problem? Thanks, Daniel

Trying to build a docker deployment using the examples, and I am getting the following error: Building dagster_daemon. Attempting this on a new AWS Workspace Linux desktop. Step 1/6 : FROM python:3.7 ---> ac9dead5ba6f Step 2/6 : RUN pip3 install dagster dagster-postgres dagit ---> Running in 8aff09a1c834 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f2f0f8dbf50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/dagster/

Good day everyone, and thank you for Dagster. I am very new to it, and would appreciate a bit of help, please. I'm working through the tutorial and the examples, but was wondering if there is a good reference DAG that can how me how to iterate through an arbitrary list APIs and download some data, either in serial, or better yet, concurrently. So it might be a case of calling the same API ,but with different parameters. That would then be encapsulated in a solid, right? And you would use something like requets.io to make the calls? Thank you in advance for any advice. I am going to look at more examples to hopefully understand if this is the correct approach.

Hi, I'm working on what I think should be a simple variation of the parameterized solids tutorial but I've hit a wall. I keep getting an error in the dagit UI that says

Missing required config entry "solids" at root level

. My execution block looks like this:

if __name__ == "__main__":
    run_config = {
        "solids": {"load_data": {"config": {"file_path": "data/day_1_input.txt"}}}
    }
    result = execute_pipeline(day_1_pipeline, run_config=run_config)

Any idea what could be going on?

Hey all! Would anyone be able to point me to some documentation that describes the different modes sensors can have? I've scoured the dagster documentation with no luck, all I can see is that "default" mode is the default mode.

Hey team, I'm about to deploy my dagster app to a k8s cluster. I need to use a K8s persistent volume to store my ML model and call it within a solid. Where should I place the volumeClaim spec? I don't see in the helm template a place for this (maybe it is, but I can't finde it).

New blog post from esteemed community member @dhume:-) https://dagster.io/blog/shared-spaces-at-drizly Please feel free to share!

Also feel free to like/RT https://twitter.com/schrockn/status/1371517778360823815

I'm working on reimplementing a daily uploads folder within a pipeline and figured I should write it as a FileManager. Curious if anyone has gone down that road already? Specifically I create a directory on local disk named from the current date (e.g. 20210315) and then as the last stage of my pipeline I write the contents of the folder to S3. The data that goes into the folder is a few different types of thing. Some of it is CSV files, there are also some output files from mongodump, and there will also be some .tar.gz files. Any thoughts on best practice given the mechanisms that Dagster has available?

Hi @schrockn thank you for this amazing product. I listened to the interview that you had with @Tobias Macey on his Python podcast and remember that you asked for as much feedback as possible. We are just getting started but my first thoughts are that it would be nice if there was an even easier production deployment path, such as using the various cloud market places. In our case that would be GCP, so I believe a GCP Marketplace deployment option would really be of benefit. I also believe that if we can have more reference architectures and real-world examples in the vein of the the airline demo, that will really help everyone as far mapping their existing frames of references to Dagster goes. Thanks again, and please keep up the good work.

Have to say: being able to perform unit tests on operators (or solids in Dagster's case) feels like uncharted waters for me. Love it!

Is there a possibility to create multiple list of same type of schedulers based on the users from the db, like creating dynamic schedulers on the runtime?

Is there a good pattern for testing in isolation that a resource can be initialized? i.e. "when given some default context, it successfully produces an instance of the resource"?

Q: Does Dagster has any feature or interface that allows one to push events or info up to Dagster dashboard from inside a runner/executor? (Somewhat along Prefect's Artifact API of sorts.)

Hey all, not to pile on the questions, but can anyone also tell me if there's a way to truly delete a sensor from dagster? I think I broke this one sensor that absolutely will not trigger because of some dagster configurations and want dagster to recognize it as a brand new sensor that has never ran before. Any help is greatly appreciated

Hi Is there programmatic way to get artifacts created by pipeline? My use case is to create report in the last solid and send it by mail. Thank you.

Hey everyone, looking at using dagster to execute some workflows. One particular use case we have involves taking a list of about 100,000 urls and getting the page

<title>

for each url then doing some further (fairly simple) processing for each one. The 100,000 number is the initial backlog and once that is complete about 1000 more would need to be processed each day. I'm not sure what the best way to structure this is. From reading the docs it seems like there are a few options: 1. Use partitions and have 1 partition for each url (not sure if partitions can scale out this much) 2. Have one pipeline that takes a single domain and does the processing required for it, then a second pipeline that takes the list of urls and executes the first pipeline for each url 3. Do it all within one pipeline and use a dynamic graph to fan out once the list of urls is fetched (not sure if dynamic graphs can handle fan out to 100,000, I'm guessing not? 4. Do it all within one pipeline and handle the processing of all urls inside a single solid. It would be a pity to do this as you lose a lot of the great observability that dagster seems to offer. Any help or thoughts much appreciated :)

I am in the process of getting started with Dagster locally. However, always see the warning of: The dagster instance configuration file (dagster.yaml) is not present. DAGSTER_HOME=/path/to/dagster_home dagit -f pipelines/hello_dagster.py is also not fixing it. How can I configure a minimal dagster setup correctly?

Furthermore, I want to better understand how to make dagit scale (beyond a single main data engineering team). How can I connect data dependencies (I guess this is called workspace)? I know that some smaller organizations seem to prefer to write everything into a single big DAG which certainly has some great benefits with regards to observability. How could I reconstruct such a E2E full graph from projects originating from different workspaces/teams?

Suppose I have around 5000 users, each user, for example, has a specific task to run at a specific time . And for that reason a scheduler needs to be created. Now to create a schedule there are two approaches, 1. Make schedulers of each user for the same task. But this isn’t working as each scheduler’s run takes a lot of time that this doesn’t seems feasible. 2. Make a single scheduler that request multiple runs for the same task with user’s config. It works great but it stops sending pipeline tasks to queue after some time. And the error is related to grpc which returns a. grpc_status:14 b. failed to connect to all addresses c. Failed to pick subchannel d. Failed to fetch execution plan for <scheduler> e. StatusCode.UNAVAILABLE What would be a better approach to handle such task?

Hello, I think I have a use case for partition sets, but I want to make sure I'm grokking your design intent for this feature. We have a bunch of simple jobs we run in a different tool currently that perform a daily truncate / reload of data from Salesforce objects into their respective Snowflake table. After reading your docs, I'm thinking I could adapt partition sets to this problem by writing a single pipeline to handle the truncate / reload functionality, and a partition function which returns the list of Salesforce objects to load. Does this sound like a valid way to use partition sets, or is there a better way I'm missing?

running into what appears to be a bug trying to test preconfiguration of a resource using

StringSource

fields that draw from an environment variable. reproduction code in the thread.

How are the diagrams in the RFC discussions created (e.g., in https://github.com/dagster-io/dagster/discussions/2902)? I’m writing up a proposal for discussion and would like to generate similar diagrams!

prha just published a new version: 0.11.0. See https://github.com/dagster-io/dagster/releases/tag/0.11.0 for the release notes.

Map reduce all the things !

new docs site 👌 1000

Can anyone share a pattern they use for including stack traces in failure hooks? I currently use a

failure_hook

on all my pipelines that looks like this:

@failure_hook(required_resource_keys={"slack"})
def slack_message_on_failure(context: HookContext):
    message = f"*ALERT*: Dagster failed while running pipeline `{context.pipeline.pipeline_name}` at solid `{context.solid.name}`.\nCheck out the logs at <https://dagster.ioby.network/instance/runs/{context.run_id}>"
    context.resources.slack.chat_postMessage(channel="product-alerts", text=message, icon_emoji=':scream_cat:', username='Uh-oh Cat')