dagster #dagster-support

Pablo Beltran

11/07/2022, 10:30 PM

Hey there! We are trying to set up an entrypoint for our workspace image but I think that the entrypoint is being overwritten/not run is this expected and if so is there a way around this?

Selene Hines

11/08/2022, 12:24 AM

Hi - is there an easy way to convert a job definition + config into a run request? Or is there a clean way to have a sensor trigger a follow on job (that doesn't necessarily have the exact same config params as the job/op that triggered it)?

saravan kumar

11/08/2022, 2:25 AM

Anyway to pass a simple string to an ops? I have an ops which queries database , but the query is parameterized.. the parameter is different during the execution of the job, so it has to be just a string whose value can be changed ..It cant be output of another ops

Ray Hilton

11/08/2022, 4:09 AM

General Dagster use-case question. We're processing large (multiple TB) of data across many machines using k8s/ec2 and tools like dask. We'd love to be able to model this in dagster for a) visibility and b) asset provenance/tracking. How would one model this? it seems like Dagster is geared towards doing all the computation within the python/dagster process

Joshua Smart-Olufemi

11/08/2022, 12:05 PM

Hi. When setting up my

dagster.yaml

, in the Compute Log Storage and the Local Artifact Storage sections, for the

base_dir

, do I use a

or a

for my path directory? My computer automatically uses the backward slash when copying paths but the dagster docs show a forward slash when indicating

base_dir

🤖 1

D 1

Robert Lawson

11/08/2022, 12:44 PM

I have dagster-dbt running well, but I can't seem to tell the best way to run schema and data tests after materialization. This seems like it should be part of the asset itself but I don't see how to do that in the docs. (https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.load_assets_from_dbt_project etc.) - is the only way to have a separate op for testing DBT assets?

dagster bot responded by community 1

Samuel Stütz

11/08/2022, 1:10 PM

Has anyone come across an Error like this before?

Copy code

An exception was thrown during execution that is likely a framework error, rather than an error in user code.
dagster._check.CheckError: Invariant failed. Description: all_assets_materialize has no op named account10__loadtest1__asset1.

We tried some loadtest of dagit. And generated all this generic assets in a factory. Locally this execution works but on k8s deployed it fails these are the asset ops

Copy code

@asset(key_prefix=context.asset_key_prefix(), name=asset_name, ins={"upstream": AssetIn(key=upstream_key)}, group_name=asset_group_name, partitions_def=daily_partitions_definition)
        def anAsset(upstream):
            time.sleep(asset_materialization_sleep_sec)
            return 42
        assets += [anAsset]

dagster v 1.0.16

Abhishek Sawant

11/08/2022, 2:34 PM

Hello, I'm trying to run redshift query from op >job >repository from the example. but i'm getting below error:

Copy code

<path>/lib/python3.9/site-packages/dagster/_core/workspace/context.py:538: UserWarning: Error loading repository location dagster_redshift.repository:AssertionError
....
dagster_redshift/op/copy.py", line 76, in <module>
    assert example_redshift_op(context) == [(1,)]

The query is running but it is giving above error, below is the sample code (same from the example)

Copy code

from dagster import build_op_context, op
from dagster_aws.redshift import redshift_resource

@op(required_resource_keys={'redshift'})
def example_redshift_op(context):
    return context.resources.redshift.execute_query('SELECT 1', fetch_results=True)

redshift_configured = redshift_resource.configured({
    'host': '<http://my-redshift-cluster.us-east-1.redshift.amazonaws.com|my-redshift-cluster.us-east-1.redshift.amazonaws.com>',
    'port': 5439,
    'user': '<usr_id>',
    'password': '<pwd>',
    'database': 'dev',
})
context = build_op_context(resources={'redshift': redshift_configured})
assert example_redshift_op(context) == [(1,)]

Zachary Bluhm

11/08/2022, 3:08 PM

Hi all, not sure of this is intended But it appears that cancelling a backfill does not terminate queued runs. Is this intended behavior? There also doesn't appear to be a way to cancel all queued runs by tag in the UI. I currently have 2800 queued runs and can only cancel one page at a time

dagster bot responded by community 1

Jesper Bagge

11/08/2022, 3:37 PM

Hi all! I’m looking for pointers on how to specify a default executor to be the multi_process one when deploying on K8s. I’ve followed the guide to deploy via Helm but since I’m running most jobs by sensor I will end up with thousands of dead pods every day I’m using the standard K8s executor. How do I specify that I want all jobs to run in the same pod?

Zachary Bluhm

11/08/2022, 3:50 PM

Is there a way to launch a job that materializes assets over multiple partitions?

Taylor

11/08/2022, 7:25 PM

Hey All - I was reviewing the Dagster Deployment Guides looking at the Running Dagster as a Service on a bare vm/ec2 instance. Are there any recommendations or guide for sizing the vm?

🌈 1

nickvazz

11/08/2022, 7:49 PM

Not sure if this is intentional or not but looks like you cannot specify different output locations for dagstermill notebooks offending line: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/asset_factory.py#L218 error it throws: https://github.com/dagster-io/dagster/blob/339d14cadc77cc31bf291ad44eedb863a7cf5c2[…]/python_modules/dagster/dagster/_core/definitions/assets_job.py

dagit -f ./dagstermill_output_notebook_io_manager_bug.py

dagstermill_output_notebook_io_manager_bug.py

Mike Atlas

11/08/2022, 9:27 PM

Is it possible to linearize backfill run requests? I have a daily partitioned job to go back through the prior year's worth of data, and would like to limit the number of parallel runs. Alternatively I could probably just make a sensor job with a cursor that "ticks" on a daily partition and set it back to start a year prior.

🌈 1

dagster bot responded by community 1

nickvazz

11/09/2022, 1:23 AM

should

dagster._check.CheckError: Invariant failed. Description: Can't supply a ConfigMapping for 'config' when 'partitions_def' is supplied.

happen with an asset job containing a dagstermill notebook asset? https://github.com/dagster-io/dagster/issues/10041

❤️ 1

Son Giang

11/09/2022, 8:51 AM

Hi team, what is the difference between

celery_executor

and

celery_k8s_job_executor

Marco Jacopo Ferrarotti

11/09/2022, 11:15 AM

Hi, I'm testing dagster in a mixed cloud/hpc environment and I would like to write a custom run launcher to launch jobs on the hpc partition which is governed by a workload manager (PBS).. Is there any kind of documentation and/or blog post explaining how to implement a custom run launcher?

Harry James

11/09/2022, 11:29 AM

Hi all! I'm testing a large single pipeline which can be conceptually split into parts A and B with A passing a single value to B. I would like to test B without computing A (i.e. passing a fixed value). Is this possible and if so how would I go about it? Here is a code snippet to try and demonstrate what I'm after

Copy code

@asset
def asset_1():
    return 5


@asset
def asset_2(asset_1):
    return asset_1 + 2


result = materialize_to_memory(assets=[asset_2, SourceAsset(key="asset_1", value=5)])
print(result.output_for_node("asset_2"))

dagster bot responded by community 1

Deepa Vasant

11/09/2022, 1:54 PM

Hi All, can we specify dagster_home as some s3 bucket?

Jose Estudillo

11/09/2022, 2:05 PM

Hi everyone, I just upgraded to dagster=1.0.16 that has a transient dependency on

grpcio<1.48.1,>=1.32.0

. The thing is dagit hangs as soon as I start it, if I update

grpcio==1.50.0

, it works normally, but this dependency is out of the defined range, any thoughts? Thanks in advance.

Zachary Bluhm

11/09/2022, 2:43 PM

Hey yall - quick noob question If I have a "graph" back asset, such as the one created by the DBT integration, is there a way to declaratively set an upstream dependency to this graph? I essentially want to do something like:

some_conditional_op

dbt_graph

Do I need to just set

some_conditional_op

as an explicit upstream in each assetdefinition I create as part of the DBT load assets function?

geoHeil

11/09/2022, 2:48 PM

https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#when-all-partitions-have-new-materializations for > Notice the

context.advance_cursor

call marks specific materializations as consumed. In future context calls, these materializations will not be returned. <<< What does this mean if not all assets are partitioned in the same way and one i.e. daily and another one i.e. hourly are to be merged. For the first hour of the day - it should work fine. But if the daily asset is consumed (at least once) it is no longer available (if I understand this correctly. How would such a join of assets with non-equal partitions work where one asset needs to be joined for a whole period of time.

Copy code

multi_asset_sensor

lists:

Mike Atlas

11/09/2022, 3:43 PM

I'm seeing this error for the first op in my graph; running on ECS.

op
finished without success or failure event. Downstream steps will not execute.

Dependencies for step
next_op
were not executed: [...]. Not executing.

Process for run exited (pid: 1).

The logs show an error like:

Copy code

Exception while writing logger call to event log: (mysql.connector.errors.ProgrammingError) 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'PIPELINE_FAILURE', '2022-11-09 15:42:15.864446', NULL, NULL, NULL)]\\n[parameter' at line 1
Exception while writing logger call to event log: (mysql.connector.errors.ProgrammingError) 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'PIPELINE_FAILURE', '2022-11-09 15:42:15.864446', NULL, NULL, NULL)]\\n[parameter' at line 1

I'm running Dagster daemon 1.0.6 - is there a bug fix or something for this in a newer version?

🤖 1

Charles Lariviere

11/09/2022, 3:59 PM

Hey 👋 I'm curious if there's a good way to record the asset materialization time as metadata, with the caveat of including both the op and io components as a single value? We can currently decorate both the asset and IO Manager's

handle_output

function to log each as separate metadata, but we would prefer to ideally record it as a single value. I see that Dagit is able to surface this using the

STEP_INPUT

and `STEP_SUCCESS`/`STEP_FAILURE` events, but I wonder whether that's available within the context of the asset definition?

Szymon Zaborowski

11/09/2022, 5:17 PM

Hi, is it considered good practice to have an asset that doesn't downstream anything? Last op in our pipeline was doing some model updates - it could return boolean value for success but that doesn't represent any persistent data that need to be materialized, maybe it's better to keep few original Ops in one asset instead? for example (sorry for pseudo notation 🙂 ) having:

{asset1: {key: our_model, ops: [train_model, deploy_model(optional)]}}

instead of

{asset1: {key: our_model, ops: train_model}, asset2: {key: deployed_status, ops: deploy_model}}

where asset1 downstreams to asset2 Thanks for reply and have a nice day 🙂

Chris Dong

11/09/2022, 5:38 PM

Noob question, can we delete dagster backfill jobs using the CLI?

Joshua Smart-Olufemi

11/09/2022, 8:30 PM

Copy code

I TRIED RUNNING THE "DEFINE AN ASSET" TUTORIAL AFTER SETTING UP MY DAGSTER IN AN EBVIRONEMNT IN POWERSHELL AND GOT THIS ERROR WHEN I RAN dagit -f cereal.py                                                                                                                                                                                                                                                                                                                                                                                          Stack Trace:  File "C:\Users\josh\Desktop\toggle assignment\assignment\venv\lib\site-packages\dagster\_grpc\server.py", line 230, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "C:\Users\josh\Desktop\toggle assignment\assignment\venv\lib\site-packages\dagster\_grpc\server.py", line 104, in __init__
    loadable_targets = get_loadable_targets(  File "C:\Users\josh\Desktop\toggle assignment\assignment\venv\lib\site-packages\dagster\_grpc\utils.py", line 33, in get_loadable_targets     
    else loadable_targets_from_python_file(python_file, working_directory)
  File "C:\Users\josh\Desktop\toggle assignment\assignment\venv\lib\site-packages\dagster\_core\workspace\autodiscovery.py", line 27, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file, working_directory)
  File "C:\Users\josh\Desktop\toggle assignment\assignment\venv\lib\site-packages\dagster\_core\code_pointer.py", line 75, in load_python_file  
    os.stat(python_file)

  warnings.warn(
2022-11-09 21:23:40 +0100 - dagit - INFO - Serving dagit on <http://127.0.0.1:3000> in process 15224

dagster bot responded by community 1

Zach P

11/09/2022, 9:04 PM

I recently had a job fail due to timeout issues (Set it too low! oops!), however, the underlying task actually succeded! Is there a way to override the status of the job to “success”?

Aaron Hoffer

11/09/2022, 9:22 PM

Does

max_resume_run_attempts

retry jobs that fail to start when

run_monitoring

is in enabled? I noticed that I’ll occasionally get that if a kubernetes pod isn’t created within the specified timeout, but it doesn’t get retried, just fails.

Zach

11/10/2022, 2:06 AM

do all the assets in a given repository that share the same resource key have to have the same resource object (by identity)?

dagster bot responded by community 1