Will there be a way in the future to simply choose a job that you want alerting for rather than having to tag the job first and then choose the tag? Not a big deal but would be good.

How is usage calculated? Is there some kind of node multiplication to the minutes if multiple nodes are used in a run?

if I have set up an alert policy for slack in the cloud UI do I also have to make an alert policies yaml? I recently set up a tag on a job and then made an alert policy for that tag to notify a slack channel for both failure/success, but when I ran the job in launchpad it didn’t alert the channel on completion. Is this because I ran the job this way rather than it being scheduled?

Hi folks. We’re working on a new way to deploy code to Dagster Cloud Serverless that makes updates much faster - it doesn't need to build a new Docker image on each commit, and it can deploy new Python code changes much faster on our servers. In our tests, a code update with no dependency changes went from taking over 3 minutes to 40 seconds on average. If you want to try this out early, here is how you can enable fast deploys: 1. Ensure you are using our new GitHub workflows. These are defined in two files:

.github/workflows/deploy.yml

and `.github.workflows/branch_deployments.yml`: a. Open

deploy.yml

, copy and save the

DAGSTER_CLOUD_URL

environment variable. You will need this later. b. Replace both yml files in your repo with the updated files from here: https://github.com/dagster-io/dagster-cloud-serverless-quickstart/tree/main/.github/workflows c. Replace the

DAGSTER_CLOUD_URL

environment variable with the saved value. 2. Ensure you are using version 1.0.17 of dagster-cloud. Your setup.py or requirements.txt should not be pinned to an earlier version. 3. Add

ENABLE_FAST_DEPLOYS: 'true'

to the top level

env

section in both yml files. To test this out: 1. Make a change to your code to trigger the updated workflows and ensure the code update works. Any subsequent code changes should get deployed fast - in about 30-40 seconds. 2. If you need to disable this feature, just remove the

ENABLE_FAST_DEPLOYS

line. To use fast deploys with your secrets, you must use the new secrets UI. This is now available under Environment variables in the Deployment tab. We’d be happy to hear your feedback if you try this out!

Hi, I’m cleaning up some junk assets on our account created by devs while testing things. When trying to wipe/delete a specific parent Asset group from the Assets page, it doesn’t seem to be doing anything. When I follow through the asset key structure to the actual Asset details, a GraphQL error occurs with the following message:

Operation name: AssetViewDefinitionQuery

Message: Internal Server Error (Trace ID: 3366240499784658422)

Path: ["assetOrError","definition","configField"]

Locations: [{"line":120,"column":3}]

Is there anyway this can be fixed?

How can I update my trial account to paid? The Usage/Billing buttons are grayed out under the Cloud settings page.

Hello, don’t know where to put this but I think there’s something wrong with how the backfill function is handle for Snowflake. I’ve configure a daily job, and when I materialize it using the UI I choose a date. Ex :

2022-11-19

This will run the backfill for that specific day. The problem I have is that dagster do the following things; 1. Delete the specific range rows from the table. 2. Create a temporary stage 3. PUT the asset into that stage 4. COPY the content of that stage to the table. My problem is that the

Delete

statement use the

Between

clause like this

DELETE FROM DAGSTER_DB.MY_SCHEMA.MY_TABLE WHERE date BETWEEN '2022-11-19 00:00:00' AND '2022-11-20 00:00:00'

But since the

Between

clause is inclusive both ways in Snowflake it translate to

DELETE FROM DAGSTER_DB.MY_SCHEMA.MY_TABLE WHERE date >= '2022-11-19 00:00:00' AND date <= '2022-11-20 00:00:00'

Then when I load data dagster always delete the next partition…. For example if I have some data for

2022-11-20

all data for that day will be deleted when I run the partition for

2022-11-19

Is there anything I can do to avoid that? Am I missing something obvious? Thanks

hey friends <!here> - (sorry to proactively blast this in two channels) this slipped out quietly last week, but we recently pushed out a new Env Vars UI on Dagster that lets you natively manage secrets. this new feature is accessible from the Deployments tab (see image) Check out the blog here: https://dagster.io/blog/dagster-managing-secrets to learn more about how to use it and how its implemented, I’d love to hear your feedback on it.

Hello 👋, we are using Dagster Serverless, and we are having difficulties with the bandwidth limitation (Maximum of 100 GB of bandwidth per day). We need to build a daily ETL process that involves downloading and uploading about 70GB of data each day. The problem is that we will also have to backfill more than 2 years of historical data (about 60TB in total). Is there a chance we could increase the bandwidth limit in order to perform the backfill using Dagster Serverless?

👋 Hello all, catching up a bit on run isolation. It looks like a really cool feature, but I have a few questions: 1. Will there be reduced cost in the future for using non-isolated runs outside of the faster startup time? Seems like going from 4cpu, 16gb ram -> .25cpu, 1gb ram should be cheaper, but, perhaps I’m misunderstanding how this works together 🤔 a. EG: Non-isolated runs should be much cheaper for short running jobs, where lots of the cost will be from the ~3 minute startup time. However, many of our jobs are in the 30+ minutes to 2 hours range. Currently, we don’t have much a reason to want to switch over since the dagster cost is a small portion of the overall job cost, and we have still other jobs are running and taking advantage of the increased compute resources of isolated runs (even if this isn’t ‘best practice’). 2. Do we have access to said PEX files? Something I’m looking into is creating a PexStepLauncher and a great starting point could be to just deploy said pex objects to an EC2/ECS instance. This is almost like a “psuedo” hybrid deployment :blob_thinking: Thanks!

Hi, I tried running a job with multiprocess executor having 10 ops which are supposed to start in parallel. When I run it locally all the 10 ops are starting in parallel but when I run the same job on the dagster cloud it is starting only 2 ops in parallel and once the 2 ops is successful then it starts another 2 ops. Could you please help how do I configure the dagster cloud so all the ops starts in parallel?

hello, i noticed something odd this morning - I started two runs that materialize separate dbt assets from the same project at around the same time (this was manually, so there was a few seconds difference between them). One of them went off without a hitch (lets call this run X) but the other one (run Y) yielded warnings of the form

Materializing unexpected asset key: AssetKey([a,b])

where

a,b

is an asset from run X! Also, while asset materialization events for run X were successfully logged, there were no such events for run Y even though the logs indicate that the dbt job was successful. I notice that the run has

isolation:disabled

so i suspect this has something to do with it - any advice on what happened and how to prevent it from happening in the future?

Hi all :dagster:, question on Branch Deployments - is there any costs incurred for having a bunch of old branch deployments? (assuming none have scheduled jobs running) Would be good to know so we can enforce some good hygiene. Thanks

👋 I've been doing some deployment testing and I seem to have messed up my ability to deploy with the standard deploy.yml. My build gets stuck on the

Dagster Serverless Docker Deploy

job. on layer #10 of the dockerfile I just spin indefinitely on

Preparing metadata (setup.py): started

The problem may have been when I removed the github code location and re-added it? Any insight into what steps I can take to resolve from here?

Or for visibility into this 🐛 here

Hello! We are using github actions to deploy and want to upgrade from python 3.8 to 3.10 but with the deploy.yml that was provided, the only option to select Python version seems to be with the fast deploy option, which does not work in our case (some issue with PEX). Any other way to select python version correctly, ideally by using slim-3.10 instead of slim-3.8?

Has the range of IPs changed for Dagster Cloud Serverless? We are having issues reaching some resources for which we had whitelisted a range of IPs given a few months ago, could they have changed?

Hello everyone, I’m trying to configure an hourly schedule that execute with 1 day delay. Is there an easy way to do that? I Found some parameters from

build_schedule_from_partition_job

but they doesn’t seem to work for hourly partition. https://docs.dagster.io/concepts/partitions-schedules-sensors/schedules

Is there anyway to view memory usage information from the runner? We’re getting a few process run OOM while running with isolated executions but are quite surprised by this (FWIW: These processes use around ~5 GB of RAM when running on our local machines, and we don’t expect them to get close to the 16GB limit, yet still running OOM). Outside of that, is there anyone rationale of what could be going on here? I understand the dagster runner probably has some overhead, but even if we only were able to use 2/3 of the memory that should be plenty).

Hello everyone, I’m trying to configure 2 assets that have the same MultiPartitionKey. The first asset work, but on the second one I got this error.

ValueError: Tried to get asset partitions for an output that correponds to a partitioned asset that is not partitioned with a TimeWindowPartitionsDefinition.

Anyone know what I should do?

Hi, where do I find the IP range for dagster cloud serverless? https://docs.dagster.io/dagster-cloud/deployment/serverless I am unable to find it here Thank you

Hey I’m trying to configure a

multi_asset

with partitions and I can’t find where I could configure the partitions for

outs

. I thought that they would take the

partition_def

from the

multi_asset

but it seems like they are not. I’m always receiving the following error.

dagster._check.CheckError: Failure condition: The output has no asset partitions

Here a snipper of the configuration of my

multi_asset

@multi_asset(
    partitions_def=hourly_partitions,
    op_tags={"kind": "s3"},
    required_resource_keys={"aws_secrets"},
    retry_policy=default_retry_policy,
    outs={
        "found": AssetOut(
            is_required=False,
            metadata={"priority": "high"},
            io_manager_key="io_manager",
        ),
        "not_found": AssetOut(
            is_required=False,
            metadata={"priority": "low"},
            io_manager_key="io_manager",
        ),
    },
)
def do_things(context):
    json_response = {'what_ever':3}
    if "tokens_found" in context.selected_output_names:
        yield Output(value=json_response, output_name="found")
    if "tokens_not_found" in context.selected_output_names:
        yield Output(value=json_response, output_name="not_found")

Is there a way I can add a

partition_Def

in the

Output()

?

in case you all are curious about some of the recent work to speed up serverless deploys: https://dagster.io/blog/fast-deploys-with-pex-and-docker

URGENT: Still waiting for compute resources to spin up We’re using dagster-cloud serverless and have been getting the above error message for the last hour with a job that runs every 15 minutes. The jobs retry for about about 3 minutes and eventually fail.

I’ve been having a lot of issues with the dagster agent over the past two days. Seems a bit off and on. Lots of

dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC

with either

Error Code: Unreachable

or

Error Code: Unavailable

. Is there some service issue going? I’ll post more of the errors in the thread.

How can I signup for release candidate notifications? Version upgraded on our production instance and it broke some things.

Can you point to a branch deployment with the dagster graphql client?

Unable to terminate job. I have a job that got stuck in a “starting” state for the last 33 hours (for a job that is supposed to run every 3 hours). I’ve tried terminating the job, but it persists in the “starting” state. I want to do the “force terminate” but there’s this warning about computational resources. How do I ensure that the resources are cleaned up after doing this? Also, how can I prevent this stuck state in the future or at least get alerted if a job is taking much longer than expected?

Hello 👋, I've been experiencing some issues with failed runs before they start. In the logs I see

Agent xxxxxxxx detected that run worker failed.

Over the past 7 days, we have encountered three instances of such errors, which account for approximately 0.1% of the total runs. I was wondering if anyone has investigated the root causes behind these worker failures. Additionally, I wanted to suggest the possibility of implementing an automatic process wherein the agent can initiate the launch of a new worker when such failures occur. This could help mitigate the impact of these issues and ensure smoother operations. Thanks!

Fixed IP addresses