Hi, have been running a dagster data pipeline for a few days. The compute logs are being stored in the local folder. The size is continuously growing and disk space is becoming a problem. Is there any way of cleaning old data automatically or put size limit on the folder?

I could use some help here with Dagster on K8s. The Dagit Web server is running fine The user code is provided in a seperate container - which runs on its own. It fails when we try to launch a run from the user code and not much help is coming either from

kubectl describe pods

or datadog

I have an op that saves a csv somewhere and which I call several types in the same graph. I’d like the filenames to depend on aliases. I’m I’m giving the ops different aliases in the graph definition. That all seems to work ok (in dagit where it shows my graph all of the nodes have the correct name). But somehow I can’t seem to access the alias from inside the op. I’m trying to access it via

context.alias

which throws this error:

AttributeError: 'SolidExecutionContext' object has no attribute 'alias'

what’s weird is that I have a test for this op (via pytest) and there everything works. I can debug into the op and access

context.alias

without problems. any ideas where this discrepancy comes from and how to access

alias

inside an op in real runs?

How to set the correct time zone? I am running the script at 04/02/2022 08:00 PM. In my panel in "Runs" , the launch time is 05/02/2022 01:00 AM. Because of this problem, when I do datetime.now() in the script, I get 04/02/2022, and not the desired 05/02/2022

So what is the "correct" way to retry ops on exception? I see two options 1. raising

RetryRequested

. This option is documented however in dagit we don't get the correct stack trace. I see the

RetryRequested

exception, not the original one. 2. passing

retry_policy

param to

@op

decorator. I see the correct stack trace, also I see a warning that this is an experimental param. What's the correct way of retrying ops?

What's the best way to maintain a consistent run datetime across ops? I have ops writing to a hive-style S3 path and want to make sure they all agree on a single date that the run was started at. Easiest would be putting the date in the job config at job creation time, but I'm wondering if there's a more native way

Hi Team, I am unable to open the document for dagster_prometheus.resources. Link : "https://docs.dagster.io/sections/api/apidocs/libraries/dagster-prometheus#dagster_prometheus.resources.PrometheusResource" Could you please help us for the same ?

Hi there, we are dyvenia, we provide consulting and development services for data engineering/science for more traditional big manufacturing companies

I am considering dagster as a replacement of Prefect due to Prefect's unclear pricing strategy (they seem to want to double their price every 6 months)

but I cannot see any pricing information for dagster, is this something we can discuss over a call?

Hello, I have a question regarding AssetMaterialization, we are currently Materializing a sql table with some metadata like the number of rows. We would like to read this metadata to ensure that everytime before we write, the number of rows we are going to write is not smaller than the one currently stored in the last Materlization metdata. Is that possible? Is there a better approach? Thanks

When I start my computer from sleeping, my docker containers are stopped because failed heartbeats in dagster daemon. Not very convenient. Is there a way to avoid this? I suspect that in the background dagster daemon compares current time with last successful healthcheck time. Like validating successful heartbeats in a separate loop

Hey, is it possible to print my own logs when running my repository function? I.e. at init time? I’d like to display e.g. the env I’m running in. The dagster logger only works within jobs it seems 🤔 i also tried

logger = logging.getLogger("dagster")

to no avail

Hi all, is there anyway of password protecting the Dagit UI? we have an instance that is displaying API keys for Cloudwatch in the workspace and i'd like to secure the interface a bit more if possible..

hello everyone once more 😃 I've implemented the logger with

@logger

decorator and with some default config items (look on screenshot) and want to understand, is it possible to set here, as the default value for the "name" parameter, the name of the Job currently running, or its ID for example??? just instead of the simple string "dagster" 😃

Hi all. I am currently migrating to op, job, graph. In a pipeline with solids, I only needed to set environment variables once using os.environ[x]=[y] in a solid and they were accessed from following solids as well. However, the environment variables set up in an op disappears once it's out of the op and other ops don't have access to them within the same graph (job). Am I missing something or is this a bug? Thank you very much for your help in advance.

Howdy - This post isn't a problem I need help with, mostly just want to discuss the idea. Does anyone currently maintain a Dagster instance using PIpenv instead of pip+venv? I'm using the latter, but I'm intrigued by some of the features of Pipenv. The main hurdle I'm facing is that it seems far simpler to get a docker deployment going using

pip install -r requirements.txt

and then running the Dagster services than it would be when working with Pipenv. Are there any other pros/cons I should consider here? Does anyone think Pipenv is worth the fuss? Anyone else think it's not worth the hassle and pip+venv works fine? Thanks in advance for the discussion 🙂

Hello, I just started working with Dagster and I would like to explore the possibility of getting the input load from a http or grpc endpoint? Most of the examples in the tutorial involved loading files from CSV files.

I don't want to muddy up the discussion in that ☝️ post, so I'll ask a tangential question here instead 🙂 Looking at this WIP example shared in that thread, why would one want to use the

out

parameter for the op config instead of just a return value for the function? Is it because it has multiple return values and one is not always guaranteed? Are there other reasons one might use that pattern?

Hi all, I have an op that extracts large amount of data from an external data source by running a sql query . I tried solving this by creating multiple partition queries and yielding the each query result as a dynamic output. We want to run these queries in parallel to reduce the execution time. we have tried with multiprocess execution, but still the queries are executing in a sequential manner. is there a way to run these queries in parallel using dagster?

Hi Team. Can you please provide an example to use Subscription from GraphQL in a python code ?

Hi everyone! Could anyone provide a working example of Dagster operations parallelized with dask?

I’m trying to create a number of jobs dynamically. The jobs themselves are essentially the same (load data from source, store in datalake), but the source tables and destination files differ. When I build this using ops they tend to interfere with each other (sometimes data is unavailable for one of the sources), so it seems the job abstraction makes the most sense (they are independent, after all). So, my real question is, what’s the best way to structure this kind of pattern? I’m currently using a job factory that returns an op with the source and destination set, but this gets abstract and unreadable quite quickly. Is there a nicer way to go about this?

Hi Guys! Can I control the multiprocessor executor count on individual pipelines or using tags? We have different memory requirements on pipelines and some fail with ChildProcessCrashed and others need a high amount of concurrent processes.

Hi Dagsters! I'm using

celery_executor

and need to limit the parallelism on specific ops (by tag for example) within a job. In the docs I found only a solution for

multiprocess_executor

and for

celery_k8s_job_executor.

but we want to keep using the

celery_executor

. Is there any solution/ ticket for future? thanks!

Hi all, my team has the need for a pipeline where each step requires it's own python env (there are dependency conflicts so they can't all be combined in the same env). My thought is to encapsulate each step as a Dagster Job, but I can't quite figure out how this would work if we wanted to launch the pipeline from Dagit. Is there a way to chain Jobs together so that they can be visualized and the entire workflow can be launched from Dagit? Another idea would be to nest the individual graphs/jobs into a larger graph/job, but it's not clear to me how the ops in each graph would get executed in the appropriate python env. Is this currently possible with Dagster? I'm running 0.13.18 in a K8s deployment. Thanks in advance for helping me make sense of this!

Is there a pattern to make backfills more efficient? When scheduling a single job a 5min/run overhead (waiting for databricks to spin up a cluster, waiting for the job container to spin up, etc) is no big deal, but when scheduling hundreds (daily runs over a few years) or thousands (hourly runs over a few years) this adds up quickly.

hi, i'm trying to evaluate dagster for my purpose (some periodic batch data processing). i'm following basic examples with schedule (hello_cereal_job). so i gather i've to run dagit along with dagster daemon which is fine. the problem i'm having is that i have to manually enable each schedule on startup via web GUI of dagit. is there a way how to create already enabled schedule ? i've been to api docs but it did not help much ;/ thanks for any ideas

one more question 😉 .. is there any snippet to see how a job/op can run in context of existing flask application but sill in thread of scheduler daemon ? eg. op/job would be able to use app ctx in terms of it's config, sqlalchemy session, etc. ?

I tried implementing a shorter partition scheduler than the shortest currently available which is hourly. It looks like this is not a trivial thing to do. Is there a shortest path to adding a new per minute partition scheduler (or even sub minute)?