Hi! How can I record materialization metadata (https://docs.dagster.io/concepts/assets/software-defined-assets#recording-materialization-metadata) while using assets without return values (https://docs.dagster.io/tutorial/assets/non-argument-deps) ?

Hey yall, we are looking to leverage something similar to: https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#monitoring-daily-assets-to-materia[…]ly-asset-using-a-multi_asset_sensor in order to handle hourly -> daily dependencies I'm curious if there are any performance considerations here? This example assumes a single upstream asset and a single downstream asset. In a real world implementation this would probably grow to be NxM (N = upstreams, M = downstreams). Do we actually need to check each downstream asset if they all exist as part of the same job?

Hey folks, I am running into an issue with dagster-postgres on azure, we are using workload identities to authenticate and so the PGPASSWORD is a token that needs to be refreshed every ~60 minutes. Our current solution is to refresh the k8s secret and restart the deployment. We looked at the dagster-postgres code and it seems difficult to supply a refreshed token because it's given the full connection string. The other option would be to maintain a custom PostgresStorageClass but ideally we would like to maybe find a solution/create a PR that works in both scenarios (static and dynamic pgpasswords). Any tips?

Got an

ExperimentalWarning

when adding an executor to my code location's

Definitions

-- is there a more recommended way of defining an executor for assets?

Is there a way to get resources by name from an

OpExecutionContext

? i.e. instead of

context.resources.foo

, can I use something like

context.resources.get("foo")

?

last question of the day (hopefully): I'm assuming no, but is there a way to use

AssetSelection

with wildcards or regex? I'm trying to do something like

AssetSelection.groups("foo") & AssetSelection.keys("bar/*")

Hi - I am running into an unexpected error using dagster-serverless:

Multiprocess executor: child process for step historical_fire_perimeters_geojson unexpectedly exited with code -9
dagster._core.executor.child_process_executor.ChildProcessCrashException

Hi guys, I have a question related to verifying p execution config.

Hi guys, I have a question related to verifying Op execution config that I haven't been able to answer via the documentation or src. My setup is approximately:

@asset
def bar(context):
  # materialize an asset ...

@asset(config_schema={"bar":str})
def foo(context) -> Output[str]:
  return Output(context.op_config['bar'])

exec_config = { "execution": { "config": { "start_method": {"forkserver": {} } } } }

job = define_asset_job(
  "foo_job",
  selection=[foo],
  config=ConfigMapping(config_fn=lambda run_config: {**run_config, **exec_config}),
  executor_def=multiprocess_executor
)

@asset_sensor(name="foo_sensor", job=job, asset_key="bar")
def foo_sensor(context, event):
  yield RunRequest(run_config={ "ops": { "foo": { "config": { "bar": "baz" } } } } )

defs = Definitions(sensors=[foo_sensor], assets=[foo])

My job successfully enqueues and executes

foo

but I am unsure whether

exec_config

is concatenated and read. Clicking "view tags and config" in Dagit's Run detail view only shows the op config. Can anyone confirm whether my approach is either correct or incorrect?

Hey @Filipe Almeida it comes down to whether you want to be able to execute them independently. If you'd like to be able to execute the test without executing the run, they should be separate ops.

I have a question regarding trying to define an asset that is downstream of multiple fanning ops. I've been wrestling with this for days and I never seem to get anywhere.

dagster._core.errors.DagsterInvalidDefinitionError: op 'collect_foo_values' cannot be downstream of more than one dynamic output. It is downstream of both "get_foo_by_bar:foo_link" and "yield_bar_links:bar_link"

How are you supposed to resolve assets downstream from multiple fanning processes?

I have a question regarding the Docker Executor. Currently I have Dagster deployed with docker on a single beef server. I also have 2 code locations running in the same server. I’m using the Docker Executor to launch jobs. I would like to move the Dagster deployment to K8s. I would also move the code locations there. However, I still want to run the jobs on the beefy server (as docker containers). Would setting the DOCKER_HOST environment variable in the code location container be enough to achieve this behavior?

I’m new to Dagster, and this question is on how I should structure my code I’m extraction data from an API with more that 300 endpoints The for extract, transform (convert from xml to a table), load (csv an an S3 bucket) The way I understand Dagster is that each extracted endpoint, converted xml and csv in S3 is an

asset

. How do I avoid writing +900 assets?

Hi All! Another noob question here, I'm setting up a Dagster project on top of an existing solution using AirByte + dbt. For some reason, my AirByte tables are not linking as a source to my dbt models. See screenshots. What I tried/checked: • I checked this documentation https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/part-one#step-3-configure-dbt-model-data-sources • I checked all the source tables listed in a yml under

sources:

. ◦ Does it need to be on a separate file? Or can it be mixed on another file with

models

? (I tried to separate one model to another file, run build for the dbt model & reload dagster, but it didn't work) ◦ All the AirByte tables sources are coming from a dbt package (and not the main dbt project). I don't think it's the issue, as I tried to move one of the models to the outer project. I'd appreciate any help.

Hello, I just updated to version 1.1.10 in order to solve the

build_asset_reconciliation_sensor

bug related to Source Asset. I have now the following error during sensor tick:

dagster._core.errors.DagsterUserCodeUnreachableError: The sensor tick timed out due to taking longer than 60 seconds to execute the sensor function. One way to avoid this error is to break up the sensor work into chunks, using cursors to let subsequent sensor calls pick up where the previous call left off.
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/sensor.py", line 491, in _process_tick_generator
    sensor_debug_crash_flags,
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/sensor.py", line 558, in _evaluate_sensor
    instigator_data.cursor if instigator_data else None,
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/repository_location.py", line 830, in get_external_sensor_execution_data
    cursor,
  File "/usr/local/lib/python3.7/site-packages/dagster/_api/snapshot_sensor.py", line 72, in sync_get_external_sensor_execution_data_grpc
    timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 403, in external_sensor_execution
    custom_timeout_message=custom_timeout_message,
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 184, in _streaming_query
    e, timeout=timeout, custom_timeout_message=custom_timeout_message
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 137, in _raise_grpc_exception
    ) from e
The above exception was caused by the following exception:
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1673597250.934518131","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81,"grpc_status":4}"
>
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 180, in _streaming_query
    method, request=request_type(**kwargs), timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 168, in _get_streaming_response
    yield from getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in _next
    raise self

I built the sensor using following code:

from dagster import AssetSelection, build_asset_reconciliation_sensor

freshness_sla_sensor = build_asset_reconciliation_sensor(
    name="freshness_sla_sensor", asset_selection=AssetSelection.all()
)

My assets are mainly loaded from a dbt project which has around 30 different models and sources Is there any way to know the root cause of the sensor tick timeout ?

How does one get access to Vercel and Buildkite? My PRs have failing checks but I lack the necessary permissions to see what the issue is and have a green PR 😞

Are assets run within a job not logged/materialized in the Asset page? e.g. I have an asset-definition file

from dagster import asset 

@asset
def asset1():
    return 100

and another job-file

from dagster import job
from asset_file import asset1

@job
def sample_job():
    q = asset1()

Running the "sample_job" fails to materialize an entry in the Asset-page on dagster UI

hey all 🙂 Is there any way to render dagit into a static site?

This message was deleted.

Hi there! We would like to do a POC in our company, but I'm not use our use case is covered. We need to create a pipeline that processes sets of images. This pipeline has intermediate products. So our first guess was to define this intermediate products as assets, with their dependencies being other intermediate products, so the DAG is generated. However, what we need is that each time our cameras collect a new set, a new pipeline starts for that set (and assets are generated for that set). But it seems we would need to define the sets before we know them.... Maybe we need asset partitions that are dynamic instead of static? We could simply use ops and jobs but that would limit the benefits of using dagster in my opinion... Or maybe I totally messed up with my proposal and there's an easy way of using dagster that can fulfill our needs 😅 Thanks for your time!

Can you bake execution configuration into a single asset? I have one that needs a little extra memory when it runs, but I'd like to keep it as part of the same job and schedule as the rest of its asset group. I can add this on the launchpad, but can I add it to the codebase somewhere?

execution:
  config:
    resources:
      limits:
        memory: 1.0Gi

Hi all! I need to use some Pyminizip, a package that requires gcc, but it is not pre-installed in the python3.8:slim version that is run by default. I tried to install gcc as a step in the .github/workflows/*.yml files (just before the step named

Build and deploy to Dagster Cloud serverless

):

- name: Install gcc
  run: sudo apt-get update && sudo apt purge gcc -y && sudo apt-get install gcc -y && sudo apt-get clean

But it seems that this is not effective and I still get this error during the next step:

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Isrc -Izlib-1.2.11 -Izlib-1.2.11/contrib/minizip -I/usr/local/include/python3.8 -c src/py_minizip.c -o build/temp.linux-x86_64-3.8/src/py_minizip.o
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1

Do you have any idea how I could install it? (Would changing the Python version allow me to use something different than Python:slim?) Thanks!

Hello, I have a question/request for

class MultiPartitionsDefinition(PartitionsDefinition)

in 🧵

Imagine it's a no, but is there anyway to configure whether or not asset names need to be unique across code locations? I have 3 locations that all use separate instances of the same system, and each code location uses its own IO Manager configuration to go to a unique bucket. Ideally, I'd be able to name all of the assets the same, but I'd be able to differentiate them by their code location/asset group

Hi everyone! My Dagster Cloud is giving me a hard time. Scheduler has stoped working👀. Just some background. Jobs can be triggered manually and they are completing successfully. Scheduler stopped working yesterday after triggering several jobs as it usually does. There is no error or something. Just this '-' in the Run part. I have not changed anything in the code. Can you please help what it can be about?

hi, is there a way to control the Job Schedule outside of the code base?

Hi team, is it possible to add a delay start time for an operation in a job?

Hi - Rookie question: I am needing to access all of my asset values from within a single op. How to I get access to the repository (which is in a different top-level file) to call

get_asset_value_loader

without encountering the circular dependencies error?

Hi, I’m having trouble using

materialize()

for testing assets. Details in 🧵

Hi team, what is the difference between

dagster project scaffold

and

dagster project scaffold-code-location

? seem that is exactly the same struct and content of the files.