after rebooting the problematic Dagit, it is all good now.

I tried to run

execute(_shell_command_ = cmd, _output_logging_ = 'STREAM', _log_ = context.log, _cwd_ = '/my/dir/path')

with dagster-shell but the temporary script directory was still set somewhere else, am I doing anything wrong there?

Hi team, we just upgraded to 0.12.3 from 0.11.x. We noticed that our partition-based daily schedule kicked off pipeline runs over the the weekend even though the partition selector we use generated partitions that have already been run, as evidenced by the partition and run_keys shown below - this wasn't happening before. Although we have

cron_schedule="0 3 * * *"

in our schedule, the weekend runs were previously inhibited, I presume due to the run_key having already been used? We can (and should), of course, change the cron_schedule to add MON-FRI, but did we just fluke the desired behaviour before 0.12.x?

Hi all, I am trying to set up dagster with traefik ingress on

<ip>/some-path

. I understand that this works in traefik by using stripPrefix, which uses X-Forwarded-Prefix header. Any idea what I have to change in dagit to make it work? Otherwise I get blank page, because the ui fetches the web assets from default path

/

Hey All. I just wanted to check something. If I want my Dagster instance to be persistent, by this I mean; If I redeploy my docker image and it re-creates my container on AWS, it should still have all history of runs ran before and so on. What would be the best way to go about this? To setup my dagster.yaml file with the Postgres/MySQL database (AWS S3 can also work here) as Run Storage, Events Logs and Schedule Storage and so?

Hi, from what I can tell, the Dagster library is type-annotated and should work with type-checking tools. But I am getting a diagnostic from the first step in the Dagster tutorial, it appears that the

@solid

decorator is not being understood. Is this a known issue, or is there somewhere that explains how to properly configure PyRight to handle Dagster? Thanks. (Note that goto-definition and other language server functionality is properly working, the dagster library is on PyRight's search path) (Note also that Pyright seems to think

solid

is a module rather than a function, that probably has something to do with it).

Hi, I am running pyspark job as a solid, by defining and configuring a spark resource with all configurations and the job is running fine as expected. However, I am not able to see the spark logs, that I would generally see if I would have run the job as a spark-submit without the use of dagster. Like for example, I do not see the spark job application ID to look up the job on the spark cluster. Please suggest a way too see all logs generated by the spark application.

👋 hi, I’m in the process of evaluating dagster for our needs and was wondering how dagster would support this use case: we have a process that syncs with a database every hour. However, if the latest sync runs for over an hour, we don’t want two syncs to be running at the same time, so we would delay until the already running sync finishes. I’m thinking one way to do this would be with a sensor but wasn’t clear if a sensor could monitor itself (basically). We could use an hourly partition to schedule but I’m worried about the overlap case

hey guys, when will 0.12.5 come out? it is really hard to put external libraries into our company’s internal repository (~ 2 weeks for Dagster related libraries), so when i request the latest Dagster, i always request as latest as possible

Hi, Is there a way to compose pipeline from multiple pipelines, similar to Composite Solids? I see

@graph

experimental API, that seems to try unifying the concept of solids, pipelines and composition, but I am not able to figure out how to use it for graph/pipeline composition.

Hello, what would you say are reasonable limits to think about task counts with dynamic orchestration? Hundreds, thousands or tens of thousands tasks? 🙂

Hello! Is there any way that dagster get notified when a job stops for any reason? I got an OOM on kubernetes so the pod died but dagster continued with its execution which made stop all the system since I set the concurrency at 1

Hi, Is there a guide/example on how to configure methods on a

ResourceDefinition.mock_resource

in tests? I'd like to be able to assert that methods are called on the resource if possible.

Hi, how can one detect if a solid is run as a part of the whole pipeline vs the solid is run as a part of a re-run request (either that single solid alone OR that solid onwards)

Also, if I use an IOManager (say S3), then can I re-run specific solids? or just a part of the pipeline? If I do that, the intermediate data will be fetched from S3 right?

Hi, We have dagster deployed on Kubernetes with EFS (https://aws.amazon.com/efs/) mounted as PersistentVolume to Kubernetes. We have used official helm for this, with some modifications for EFS PV/PVC in main dagster helm chart. We would like to use DBT as well with Dagster. Both of them, require file system path directory, in order to work. Dagster needs it in

workspace.yaml

for

python_package:

or

python_file:

DBT needs it for its SQL Models. Now, as far as, Dagster is running in

in_process_executor

mode, they (Dagster & DBT Solids) can easily access the EFS, as: 1. We have mounted EFS to main Dagster Deployment. 2. All solids will be executed in same Dagster Deployment as a system process, which will also has the access to EFS. But the question comes, when Dagster will be running in

celery_k8s_job_executor

mode, where each solid is executed in a separate ephemeral kubernetes pod.

_*In

celery_k8s_job_executor

mode, will DBT solids which will get executed in a separate ephemeral k8s pods, have access to the EFS?*_

Because, the EFS PV/PVC changes are done for main dagster deployment. I am not that much expert in k8s, but is it possible, if those ephemeral kubernetes pod uses the same main helm chart image as base image, which already has a mount with EFS, the DBTs will have access to EFS as well? It will be really great if someone can give some feedback on this. Thanks.

Hi guys, Just wanted to let you know that installing dagster-postgres on M1 macs is currently challenging with the install requirements:

psycopg2-binary<2.9

.

psycopg2-binary==2.9.1

has smoothed the installation for M1 macs.

Hi all, how do we add solid definition to this kind of solid?

update_cohort_source = dbt_rpc_run_and_wait.configured(
    {"models": ["tag:cohort"]},
    name="update_cohort_source",
)

Basically, I wanna add RetryPolicy to this solid, but I have no idea how to add that to this kind of solid. Any suggestion?

Is it reasonable to take the python package approach (as opposed to GRPC server within docker container) when running Dagit, the dagster-daemon and Postgres within docker containers configured via docker-compose.yml ?

Hi all, Running into issues with when running the dagit and dagster commands locally after clean dagit install. The traceback shows module not found errors. I ran into the exact same issues with dagit/dagster installed via both pip and poetry w/ pyenv as the python version manager. Running version 0.12.4 for all dagster packages. Any thoughts?

Running dagit...

Traceback (most recent call last):
  File "/opt/homebrew/bin/dagit", line 5, in <module>
    from dagit.cli import main
  File "/opt/homebrew/lib/python3.9/site-packages/dagit/__init__.py", line 1, in <module>
    from dagster.core.utils import check_dagster_package_version
ModuleNotFoundError: No module named 'dagster.core'

Running dagit...

Traceback (most recent call last):
  File "/opt/homebrew/bin/dagster", line 5, in <module>
    from dagster.cli import main
ModuleNotFoundError: No module named 'dagster.cli'

Hi! I'm getting this error in dagit on pipeline view. pipeline overview tab doesn't show everything. i rolled my code back to state which is working fine on another machine but the error doesn't go away.

Operation name: PipelineOverviewQuery

Message: Event logs invalid for run id 17b38fc5-db54-4033-a53e-705f4fe19596

Path: ["pipelineSnapshotOrError","runs",0,"assets"]

Locations: [{"line":19,"column":9}]

Stack Trace:
  File "/usr/local/lib/python3.8/site-packages/graphql/execution/executor.py", line 452, in resolve_or_error
    return executor.execute(resolve_fn, source, info, **args)
  File "/usr/local/lib/python3.8/site-packages/graphql/execution/executors/sync.py", line 16, in execute
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/dagster_graphql/schema/pipelines/pipeline.py", line 268, in resolve_assets
    return get_assets_for_run_id(graphene_info, self.run_id)
  File "/usr/local/lib/python3.8/site-packages/dagster_graphql/implementation/fetch_assets.py", line 62, in get_assets_for_run_id
    records = graphene_info.context.instance.all_logs(run_id)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/instance/__init__.py", line 1000, in all_logs
    return self._event_storage.get_logs_for_run(run_id, of_type=of_type)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/storage/event_log/sql_event_log.py", line 199, in get_logs_for_run
    events_by_id = self.get_logs_for_run_by_log_id(run_id, cursor, of_type)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/storage/event_log/sql_event_log.py", line 178, in get_logs_for_run_by_log_id
    raise DagsterEventLogInvalidForRun(run_id=run_id) from err

If I try to run a pipeline i'm getting this in terminal:

dagster_dagit                | Exception in thread postgres-event-watch:
dagster_dagit                | Traceback (most recent call last):
dagster_dagit                |   File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
dagster_dagit                |     self.run()
dagster_dagit                |   File "/usr/local/lib/python3.8/threading.py", line 870, in run
dagster_dagit                |     self._target(*self._args, **self._kwargs)
dagster_dagit                |   File "/usr/local/lib/python3.8/site-packages/dagster_postgres/event_log/event_log.py", line 260, in watcher_thread
dagster_dagit                |     dagster_event: EventRecord = deserialize_json_to_dagster_namedtuple(
dagster_dagit                |   File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 241, in deserialize_json_to_dagster_namedtuple
dagster_dagit                |     dagster_namedtuple = _deserialize_json_to_dagster_namedtuple(
dagster_dagit                |   File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 252, in _deserialize_json_to_dagster_namedtuple
dagster_dagit                |     return _unpack_value(seven.json.loads(json_str), whitelist_map=whitelist_map)
dagster_dagit                |   File "/usr/local/lib/python3.8/site-packages/dagster/serdes/serdes.py", line 276, in _unpack_value
dagster_dagit                |     check.invariant(
dagster_dagit                |   File "/usr/local/lib/python3.8/site-packages/dagster/check/__init__.py", line 167, in invariant
dagster_dagit                |     raise CheckError(f"Invariant failed. Description: {desc}")
dagster_dagit                | dagster.check.CheckError: Invariant failed. Description: Attempted to deserialize class "EventLogEntry" which is not in the whitelist.

can you please help, how do i fix this error?

Hi 🙂 How do you pass an

asset_key

downstream? Basically, I am pulling in a local file utilizing the

yield AssetMaterialization

@solid()
def solid__get_file(context):
    file = os.path.abspath(f"some_file.csv")
    yield AssetMaterialization(
        asset_key="file",
        metadata={
            "file": EventMetadata.path(
                file
            )
        }
    )
    yield Output(None)

How do I pass this into a solid downstream?

Hello #dagster-support, I use istio proxy sidecar containers, so the pods created by dagster job resources in my k8s cluster are failing to reach a completed state since the istio proxy sidecars continue to run even after the main containers finish executing their steps. All I need to do is add a

curl -X POST <http://localhost:15020/quitquitquit>

to the end of the jobs’ commands and it will shutdown the sidecars. How can I do that with the Helm chart though? Thank you.

Hi all, I’m learning dagster now (coming from airflow). What is the best practice for managing many pipelines/repositories? Putting all the repos in

workspace.yaml

is the only way? For example in Airflow, there is gitsync or volume mount to manage many dags but I wonder how I should go about managing many pipelines and repositories on dagster. Thanks 🙂

Hey all, old school DBA here (super modern data stack beginner). I'm stepping my way through the tutorials and have hit a road block. Trying to get the dagster-daemon to run, I'm getting an "The environment variable $DAGSTER_HOME is not set" error. I'm working on windows in an Anaconda install in a new python virtualenv created for the purpose of learning dagster. Any help getting around this is greatly appreciated. I'm not clear if the error is related to a Windows env variable or some other that I'm not familiar with. Thanks!

Hi folks! I’m helping my team at DigitalOcean evaluate Dagster for our data eng use cases, and I’m wondering if there’s someone from Dagster/Elementl I could PM briefly with a couple of questions

Hi Team, what version will officially make the switch from solid to op?

Hi, I am trying to implement my own custom IOManager. The way our pipeline is designed currently is that all solids return

Nothing

i.e. the

handle_output()

of the IOManager will always have

obj

as

None

. Thus, the dependencies are specified using

input_defs=[InputDefinition("name_of_previous_solid", Nothing)]

for the next solid. I still want the logic of

load_input

of my custom IOManager to work before we start executing this next solid. But it seems it skips it altogether. Any suggestions on what can be done?

Is there any way to get a resource's config from within a solid?