Hey guys got 2 questions while trying to use dagster to make our python file (which has a lot of if/else conditionals and repeated python functions) more readable and easy to use: • are aliases the only way to reuse a solid multiple times in the same pipeline? • Are conditional outputs the only way to mimic if/else python conditionals?

Hello, maybe there is a visual interface for creating flow by analogy with streamsets?

Helllo, i am currently working with dagster to parse a really long excel file, my startegy involves read a line and execute a pipeline with the data from that line, seems a not so good way to do things. I would like to run it just once, can someone give me a opinion on how to accomplish that. Thanks in advance.

One of my dagster jobs failed and the logs in the job pod give me a traceback error. I'd like to see the same traceback in the Dagit UI but I can't seem to find it (I have Debug filter enabled). I only see that my job failed and a message that logs are being retrieved. Is it possible to see the pod logs in the dagit ui?

Has anyone here worked with running Singer taps/targets via Dagster? I'm wondering if it's worth trying to contribute support for Dagster as an executor to the Meltano project, or if I should skip on it and just write a plugin for Dagster to handle it directly.

Hi everyone

I'm new with dagster (it's awesome) and I have a question about executors/execution

I have a simple pipeline but it took 1 minute between each solid

I am testing Dagster with this example as it is similar to my use case. I can run the pipeline successfully by manually executing with dagit ui both locally and in our cloud environment. I then tried adding a

dagster.*hourly_schedule*

to the pipeline but am getting the error below on each scheduled run. (FileNotFoundError) - FileNotFoundError: [Errno 2] No such file or directory: 'sources.json'. Any advice would be greatly appreciated. This is the first time I am trying to schedule a job and just trying to figure it out. Thanks

Hello, would it be possible to schedule a job to run every 20 seconds?

Hi I am trying to deploy dagster or kubernetes. When I launch execution I got this error. What I am missing?

Hello, I am using User Code Deployments (gRPC) (https://dev.to/michielhub/dagster-with-user-code-deployments-grpc-2c16) Do I understand correctly that if I want to add new repository and/or new pipelines to repository then each time I have to reinstall Dagster (helm uninstall dagster ..... & helm install dagster...) ? Or they should refresh automatically? Because UI does not show new repos/pipelines after I have added new images to docker registry, but it shows them after I have reinstalled dagster. Thanks in advance

Hi, I am new to Dagster and I have a short Q, what's the back-end that stores each Node information on the DAG (i.e., a node/solid status, metadata about the solid, run status, code that the solid runs, etc.) - I am looking to add some custom information to the nodes/solids and was wondering what'd be the best way to do something like this.

Hi, I have some existing workflows written using luigi that I'd like to migrate to dagster. luigi uses the concept of tasks that the user subclasses and then a workflow is a hierarchical, object oriented, dependency representation. what examples may I look at for dagster to get going at migrating the work flows please?

Johann Miller just published a new version: 0.9.16.

Hi, What are Your experience about how "big" servers (to create kubernetes cluster) we should need for dagster? Which component needs what min/optional/max resources (CPU, RAM)? I understand that this could depend from operations inside pipelines. We want to achieve: -get data from source (postgresql); -do data tranformations with pandas/numpy ... Cleaning Filtering Joining Sorting Splitting Deduplication Summarization; -save to db;

Hi, I got this error: ImportError: cannot import name 'schedules' from 'dagster'

@schedule(name='heatbeat', pipeline_name='heat_beat_pipeline', cron_schedule='* * * * *') def get_schedules(): return []

Do you have any document on how to use @schedule?

Or example?

You can see how I'm using it in my code here https://github.com/mitodl/ol-data-pipelines/blob/main/src/ol_data_pipelines/edx/schedule.py

Thanks, but I want to schecule every minute.

this exmaple is daily

@He Tommjust confirming you saw my comment above, let me know if changing the import doesn't help

@daniel Here is my code, and i got some error: https://dpaste.com/8ERKLY57U

@daniel thanks a lot!