Hello! The new documentation pages don't scroll or automatically navigate to the top of the page when clicking on a link to a new page unless the page Im currently on prior to switching to new page is at the top already. I searched if there is an issue opened on this but didnt find it. Im on Chrome version 89.

I’m digging into the ‘custom dataframe type’ with

create_dagster_pandas_dataframe_type

and got my first one working 🙌🏼. Is there a built-in way for the dataframe validation to fail if there is a column that’s not defined in the ‘dataframe type’? If so, I’d love to see an example of how that’s done.

Hello, I know if I want to output say 5 rows from a pandas dataframe in dagit's log output, I could use the EventMetaData feature of dagster (per https://github.com/dagster-io/dagster/issues/3234), but I would think I should be able to just output it using the context.log.info() function. I get an error if I do:

<http://context.log.info|context.log.info>(df_final.sample(n=5).transpose())

Error message:

dagster.core.errors.DagsterExecutionStepExecutionError: Error occurred while executing solid "feature_selection":

File "D:\Python38\envs\ashli_dev\lib\site-packages\dagster\core\execution\plan\execute_plan.py", line 190, in _dagster_event_sequence_for_step

for step_event in check.generator(step_events):

File "D:\Python38\envs\ashli_dev\lib\site-packages\dagster\core\execution\plan\execute_step.py", line 311, in core_dagster_event_sequence_for_step

for user_event in check.generator(

File "D:\Python38\envs\ashli_dev\lib\site-packages\dagster\core\execution\plan\execute_step.py", line 70, in _step_output_error_checked_user_event_sequence

for user_event in user_event_sequence:

File "D:\Python38\envs\ashli_dev\lib\site-packages\dagster\core\execution\plan\execute_step.py", line 596, in _user_event_sequence_for_step_compute_fn

for event in iterate_with_context(

File "D:\Python38\envs\ashli_dev\lib\site-packages\dagster\utils\__init__.py", line 363, in iterate_with_context

return

File "D:\Python38\lib\contextlib.py", line 131, in __exit__

self.gen.throw(type, value, traceback)

File "D:\Python38\envs\ashli_dev\lib\site-packages\dagster\core\errors.py", line 197, in user_code_error_boundary

raise error_cls(

The above exception was caused by the following exception:

dagster.check.ParameterCheckError: Param "msg" is not a str. Got 681 214 212 230 521

Parch 0 0 0 0 0

Fare 76.7292 7.75 7.25 83.475 7.8958

Embarked C Q S S S

Sex male male male female male

Name Hassab, Mr. Hammad Kiernan, Mr. Philip Perkin, Mr. John Henry Harris, Mrs. Henry Birkhardt (Irene Wallach) Vovk, Mr. Janko

Age 27 NaN 22 35 22

Survived 1 0 0 1 0 which is type <class 'pandas.core.frame.DataFrame'>.

However, I get no error if I do:

<http://context.log.info|context.log.info>(f"5 random rows: \n {df_final.sample(n=5).transpose()}")

I get the output that I expect. With Python's standard lib logger, I can output a partial dataframe without using f-string or having to convert to string.

Hey! Im trying to integrate great expectations but the example is not really clear to me. Is there any other resources where I can check examples?

Has there been any thought on auto-discovery for Dagster gRPC servers through orchestrators like Docker Swarm, ECS, or Kubernetes? I’m revamping our Dagster deployment here, and it’d be nice if I could say “load from labelled Swarm services” in

workspace.yaml

instead of manually listing each one.

Rex Ledesma just published a new version: 0.11.2. See https://github.com/dagster-io/dagster/releases/tag/0.11.2 for the release notes.

Hello ! We have a pipeline that run every hour. This pipeline needs to call an API, and, to do that, need to retrieve an Access token from our Auth provider. The Access token is valid for 24 hours. So far, we retrieve a new token at every run, but, it could be great to fetch the token once a day, and reuse it, while it's valid. Is there any way to store the token using one of dagster features ? Or maybe the best solution is to deploy a "shared" Redis, and have the job retrieve the token from Redis, and, if it's not in Redis, make the request ?

hi. i've been seeing these errors in the browser console - the problem seems to go away if i switch back to 0.11.1

Hi all! Got another sensor related question for you. I'm in a situation where I need to send more than 10 run requests to a pipeline. On version 0.10.9 and my question is a two-parter 1. Is there any way to increase this 10 run request limit? 2. How does the QueuedRunCooridinatorDaemon detect that the run request limit has been reached? Ideally, I would not want the sensor to keep piling on tasks for the pipeline and DoS the daemon to death. I'd like to incorporate such functionality in my sensor where if the run limit for a pipeline has been reached, the sensor will not populate the queue

Dagster has really influenced some of the ways we think about and communicate our analytical approach to our clients . Thanks. https://cpcs.ca/three-steps-to-build-trust-in-your-data-and-analytics/

Hey, how can I specify the node to be used by the K8sRunLauncher jobs? I see that there is a nodeSelector for the CeleryK8sRunLauncher but not for K8sRunLauncher. Same question about resource limits for jobs.

Hi all, I am very new to Dagster and exploring it for our use-case. Currently going through the docs and got confused between composite solids vs pipelines. We have different sets of computations, each independent, doing the same steps but on different assets (tables) and have different upstream dependencies. I was tempted to create configurable/reusable composite solid, programmatically create new invocations of the composite solid for each set of computations with different config params. But in this case, we cannot execute each set independently as solids are not executable and we might need this capability as each have different upstream dependencies. On the other hand, if I model each set as an independent pipelines, I am not sure how to create them programmatically as pipelines does not support configs. Could someone help me design our use-case and correct me if I am missing something? I also see there is an ongoing effort to merge pipelines and composite solid into graphs and make solids executable. https://github.com/dagster-io/dagster/discussions/2902

Thoughts on running dagster as a cron replacement, like people do with airflow

my introductory take is "use dagster for anything you'd use airflow for", wondering if that's a bad understanding

Hello everyone, How can I automatically restart the scheduler if the value of cron string gets changed in the scheduler definition?

Hello Everyone, Im looking for a good documentation on distributed deployment. Best practices on efficient way to update/upload dags on a running instance. I’m planning to deploy and test dagster in a distributed fashion, and I’d be glad to know how other people had done it.

hi 👋 what is the compatibility between dagster helm chart and user deployments? e.g. can i have a deployment on 0.11 and the cluster on 0.10? trying to figure out how to deal with version drift

Hello everyone! I wonder - is dagster can be used independently? See lots of posts that It should be used with other schedulers like Airflow

if a step is re-run (due to a run failure) should the InputContext upstream_output.config for the re-run step change? I have an IO manager and the handle input depends on the configured output path of the previous step but when the step is re-run the output path is recalculated as if a the previous step was run again. Because the output path contained a timestamp the handle_input function can no longer find the file.

What is the cleanest way to execute solid each 10 seconds (in infinite loop)?)

Hello! My dagster daemon (version 0.11.2) apparently complains if I move my repository definition file to a different location (I simply just moved it into a "repositories" folder a level above my workspace.yaml file) even though I've updated my workspace.yaml to reflect the new location. In other words, my workspace.yaml went from:

load_from:

- python_file:

relative_path: sc_repositories.py

executable_path: /path_to_my/python

to:

load_from:

- python_file:

relative_path: repositories/sc_repositories.py

executable_path: /path_to_my/python

I get an error that it can't find

sc_repositories.py

file because it is looking for it in the original location. So I am baffled by this since in my workspace.yaml file, I told it that it is now in the repositories folder as indicated above. Not knowing what to do after looking at the documentation, I then deleted the tables in the Postgres database. Then after executing

dagster-daemon run

again, I no longer get the error message. So apparently the location of my repository is being hard-coded/saved or persisted in the Postgres backend, is that right? I looked at the documentation for dagster daemon, but didn't find any mention about this.

Hi-- I'm new to data engineering and I'm looking for a push in the right direction. I have been tasked with writing a system where our users (chemists) will upload a set of experiments to the cloud, where we'll run simulations and report back. I think dagster is a good fit for running the simulation/analysis pipeline. I'm thinking: I'll write a Flask app to upload the data to a GCP bucket, then use the GraphQL api to start the pipeline. The client would then long-poll the GraphQL api until the job completes. Is this roughly the right approach? Any advice?

What will be dagster pricing model in the future?

Hello! I am working with a Apache proxy infront of dagit. And for dagit it self it works great, however, the graphql playground doesn't. I can reach the page but it fails to connect to the websocket. For some reason the authorization header isn't included in the request to the graphql websocket. If I manually add the header using the http headers tab in the graphql interface, it works great. Do you think this might be a issue with how the graphql playground handles headers? I can't seem to find a setting for Apache which resolves the issue at least. And it seems to be working for all other dagit pages.

MLFlow Tracking has some nice features that seem to be lacking in Dagster (a record of the current git commit, ML metrics, etc.) They also integrate nicely with Databricks that allows for easy cluster deployment. However, they really lack in building complicated pipelines, in which Dagster excels. Is there a way to get "the best of all worlds"? That is, integrate Dagster with MLFlow and have it thus run on Databricks? Or is there a good alternative?

hi! is there a way to pass an env variable to sensors via the helm chart? I tried using

env

in my userDeployment but no luck with that (i see it in the configmap, but if i reference it from my sensor it fails)

This is probably a silly question, but hear me out … if someone wants to build a new pipeline with (already built) solids, what’s the ideal path for that? I’m thinking they would grab the repo that contains the solids, and build a pipeline locally and execute to test? and run dagit locally if they desired? In other words, there’s no way to build a pipeline in the main instance of dagit using prebuilt solids, right?

Is there a gcs equivalent to the

dagster_aws.s3.sensor.get_s3_keys

function?

Hi! Is there in dagster any possibility of terminate pipeline, if it execute more than some time? (something like timeout of pipeline execution) Thank you

Do folks here have strong opinions about

setup.py

vs

requirements.txt

for dagster(+dagit) projects? I’ve seen both used for Flask/Django applications - is there a best practice I should be aware of?