dagster #dagster-support

Xu Zhang

07/28/2021, 5:55 AM

Copy code

Application failed to start after 60 seconds, output: /export/apps/python/3.7/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/export/apps/python/3.7/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/export/apps/python/3.7/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/export/apps/python/3.7/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/export/apps/python/3.7/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)

Xu Zhang

07/28/2021, 5:55 AM

🙏

Emily

07/28/2021, 1:01 PM

wanted to give some feedback on the new dbt resource. for one, it has definitely improved our dbt pipeline and made it much cleaner overall! but one small thing that i came across i think could improve it a little more if possible. when i was initially setting it up, i wanted to set the config like this, where we can set

models

only one time. but in reality, when we try running our pipeline this way we get errors on certain steps (like

run-operation

) since macros dont take

models

as an argument. so in practice, i had to take

models

out of the config here and pass it separately in the

run

and

test

solids. it would be awesome (if possible) to be able to pass

models

in the resource config so we only have to set it once, but only have it pass into the dbt commands where its expected as an input (like

run

and

test

but not

run-operation

)

Copy code

"resources": {
        "dbt": {
            "config": {
                "target": DBT_TARGET,
                "profiles_dir": DBT_DIR,
                "project_dir": DBT_DIR,
                "models": ["*"],
                "ignore_handled_error": True,
            }
        },
    },

Deveshi

07/28/2021, 2:50 PM

Hi - just wondering what does status "Skipped" mean here in the backfill? Thanks!

William Reed

07/28/2021, 5:18 PM

Hello all, just checking out dagster for our team and wondering what the whole multi-tenancy situation looks like. Is there anything like RBAC, multi-user support, etc. or does anyone with access to the dagit instance have access to all the pipelines? Thank you!

Xu Zhang

07/28/2021, 6:03 PM

Sorry Daniel @daniel , I’m seeing a similar

[Errno 30] Read-only file system: '/export/content/lid/apps/dagster-web'

error when I re-execute a pipeline, but not when I click

Launch Execution

in playground page.

Copy code

dagster.core.errors.DagsterLaunchFailedError: Error during RPC setup for executing run: OSError: [Errno 30] Read-only file system: '/export/content/lid/apps/dagster-web'

  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e6feb50d76cd5e604738da124e6cd04deae3ce84b1e12b8646074e6845081e63/site-packages/dagster_graphql/implementation/utils.py", line 26, in _fn
    return fn(*args, **kwargs)
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e6feb50d76cd5e604738da124e6cd04deae3ce84b1e12b8646074e6845081e63/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 11, in launch_pipeline_reexecution
    return _launch_pipeline_execution(graphene_info, execution_params, is_reexecuted=True)
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e6feb50d76cd5e604738da124e6cd04deae3ce84b1e12b8646074e6845081e63/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 50, in _launch_pipeline_execution
    run = do_launch(graphene_info, execution_params, is_reexecuted)
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e6feb50d76cd5e604738da124e6cd04deae3ce84b1e12b8646074e6845081e63/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 38, in do_launch
    workspace=graphene_info.context,
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e6feb50d76cd5e604738da124e6cd04deae3ce84b1e12b8646074e6845081e63/site-packages/dagster/core/instance/__init__.py", line 1350, in submit_run
    SubmitRunContext(run, workspace=workspace)
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e6feb50d76cd5e604738da124e6cd04deae3ce84b1e12b8646074e6845081e63/site-packages/dagster/core/run_coordinator/default_run_coordinator.py", line 32, in submit_run
    self._instance.launch_run(pipeline_run.run_id, context.workspace)
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e6feb50d76cd5e604738da124e6cd04deae3ce84b1e12b8646074e6845081e63/site-packages/dagster/core/instance/__init__.py", line 1409, in launch_run
    self._run_launcher.launch_run(LaunchRunContext(pipeline_run=run, workspace=workspace))
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e6feb50d76cd5e604738da124e6cd04deae3ce84b1e12b8646074e6845081e63/site-packages/dagster/core/launcher/default_run_launcher.py", line 97, in launch_run
    res.message, serializable_error_info=res.serializable_error_info

I think it failed due to the same reason that

compute_log_manager

had the other day (and you helped me). I looked into the

default_run_coordinator

doc, but there is no base_dir etc. to set..

Xu Zhang

07/28/2021, 7:20 PM

Ok, I think i found a bug when we click the

Re-execute All

button: I have two pipelines A, B with the

same name

but at different locations. When I trigger the run for B using the Playground, the selector sent back to the Dagster is B, which is expected; however, when I click the

Re-execute All

button to re-execute the B pipeline, the selector sent back to Dagster is actually A.

Xu Zhang

07/28/2021, 7:33 PM

Xu Zhang

07/28/2021, 7:38 PM

i’ve created issue: https://github.com/dagster-io/dagster/issues/4415

Arun Kumar

07/29/2021, 2:32 AM

Hi team, whenever there is backfill triggered on an upstream pipeline, our sensors will automatically trigger all the downstream pipelines based on the assets produced by the upstream pipelines. But currently these downstream pipelines runs are triggered as normal runs, Is there any way by which I can trigger them as backfill runs? How would I identify if the assets are produced by backfill runs or normal runs?

takan

07/29/2021, 9:24 AM

Hi all, I’m editing the default user-code-example to use GCS instead of S3 and I have created my own image, set up dagster and user-deployment charts separately, and now from the playground UI on dagit, I ran it with “Launch execution” but the daster-run pod fails with ErrImagePull and I think it’s because I use GCS private repository. How can I fix it? I tried inserting

Copy code

imagePullSecrets: 
  - name: gcr-secret

under

dagster-user-deployments

on values.yaml for dagster chart and on values.yaml for dagster-user-deployment chart but no luck so far. Also, if I need to put my gcr-secret in values.yaml for dagster chart, why do I need it there? Because I separated my deployment of user-code with dagster-user-deployment chart, I thought dagster chart is not responsible handling user-code-deployments anymore. Please help me out to understand it better 🙂

pdpark

07/29/2021, 6:37 PM

Calling a function in a repo definition that appears to be called multiple times?

Copy code

@repository
def prod_repo():
    run_config = get_params()
    return [
        my_graph.to_job(
            config={
                "solids": {
                    "do_stuff": {
                        "config": {
                            "a_param": run_config.get("a_param")
                        }
                    }
                }
            }
        )

Oyin

07/29/2021, 6:39 PM

I have a question with regards to conditionally running a solid. If for instance, I am doing a type check and the check fails. How do I write a conditional solid in such a way that it only executes if a type failure occurs? Maybe send a message to slack or an email to the stakeholder?

Navneet Sajwan

07/29/2021, 7:24 PM

Hey everyone, I am new to dagster, and still exploring dagster ecosystem. I want to create multiple repositories. But I don't want my older repositories to reload , when I add a new repository to workspace.yaml to build the new repository. Is there a way around it?

Rob Meng

07/29/2021, 9:07 PM

Hi folks, Has anyone looked into auth on Dagster? I guess authN is easy with a proxy in front. What about AuthZ, RBAC, and impersonation? I'd pay to have it 🙂

Mikhail Knyazev

07/29/2021, 9:17 PM

Hey, does

dagster-daemon

need to access repo files to run sensors? I run repos in docker containers via gRPC, and it seems

dagster-daemon

doesn't pick up any defined sensors...

Jean-Pierre M

07/29/2021, 9:54 PM

Hi all. I have a pipeline where my first solid reads an S3 bucket and yields a DynamicOutput for each file it finds. I then map over the dynamic output with a composite solid. The composite solid is itself composed of three solids, let's say A, B, C. The issue I'm having is when I run the pipeline, with the default or multiprocess executor, the pipeline is running all the A solids for all the dynamic outputs first, then all the B solids, then all the C solids. Is there a way to change the execution orders so that the pipeline runs A then B then C for the first dynamic output before moving on to the next dynamic output. E.g. My pipeline execution waterfall looks like this (for 3 files):

Copy code

read_bucket
  composite_solid.A[file_1]
    composite_solid.A[file_2]
      composite_solid.A[file_3]
        composite_solid.B[file_1]
          composite_solid.B[file_2]
            composite_solid.B[file_3]
              composite_solid.C[file_1]
                composite_solid.C[file_2]
                  composite_solid.C[file_3]

But I would like for it to be:

Copy code

read_bucket
  composite_solid.A[file_1]
    composite_solid.B[file_1]
      composite_solid.C[file1]
        composite_solid.A[file_2]
          composite_solid.B[file_2]
            composite_solid.C[file2]
              composite_solid.A[file_3]
                composite_solid.B[file_3]
                  composite_solid.C[file3]

I have tried playing around with the

dagster/priority

tag in the solid definitions but it doesn't seem to do anything in this case. Thanks in advance!

Arun Kumar

07/29/2021, 11:34 PM

Hi team, a basic question. We have some

print

statements in the repo.py file, how I don't see those in the Kubernetes user deployment pod logs. Everything looks fine in the my local docker environment. Any thoughts on why this could happen?

Gabe Calvo

07/30/2021, 12:45 PM

Hi all, I think I might have found a bug, but I might as well be doing things wrong. I'm trying to use DynamicOutputs with dagster-azure's adls2 io manager and I get:

Copy code

An unexpected exception was thrown. Please file an issue.
ValueError: too many values to unpack (expected 3)

  File "c:\users\gcalvo\appdata\local\pypoetry\cache\virtualenvs\9lnrhjfr-py3.9\lib\site-packages\dagster\core\execution\plan\execute_plan.py", line 193, in _dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "c:\users\gcalvo\appdata\local\pypoetry\cache\virtualenvs\9lnrhjfr-py3.9\lib\site-packages\dagster\core\execution\plan\execute_step.py", line 326, in core_dagster_event_sequence_for_step
    for evt in _type_check_and_store_output(step_context, user_event, input_lineage):
  File "c:\users\gcalvo\appdata\local\pypoetry\cache\virtualenvs\9lnrhjfr-py3.9\lib\site-packages\dagster\core\execution\plan\execute_step.py", line 380, in _type_check_and_store_output
    for evt in _store_output(step_context, step_output_handle, output, input_lineage):
  File "c:\users\gcalvo\appdata\local\pypoetry\cache\virtualenvs\9lnrhjfr-py3.9\lib\site-packages\dagster\core\execution\plan\execute_step.py", line 488, in _store_output
    handle_output_res = output_manager.handle_output(output_context, output.value)
  File "c:\users\gcalvo\appdata\local\pypoetry\cache\virtualenvs\9lnrhjfr-py3.9\lib\site-packages\dagster_azure\adls2\io_manager.py", line 73, in handle_output
    key = self._get_path(context)
  File "c:\users\gcalvo\appdata\local\pypoetry\cache\virtualenvs\9lnrhjfr-py3.9\lib\site-packages\dagster_azure\adls2\io_manager.py", line 23, in _get_path
    run_id, step_key, output_name = context.get_run_scoped_output_identifier()

The code runs fine if I test it locally (in-memory), but returns this error when using the adls2 io manager. Am I doing something wrong? Cheers, Gabe

Francois-DE

07/30/2021, 1:16 PM

Hey there. I just want to check, could the folder structure of a dagster project have an impact on the time it takes to spin up the gRPC server? I keep getting this error:

Copy code

Stack Trace:
  File "/media/veracrypt1/sandbox/repos/poc-datalake-pipeline/dlpoc-venv/lib/python3.7/site-packages/dagster/core/host_representation/grpc_server_registry.py", line 181, in _get_grpc_endpoint
    fixed_server_id=new_server_id,
  File "/media/veracrypt1/sandbox/repos/poc-datalake-pipeline/dlpoc-venv/lib/python3.7/site-packages/dagster/grpc/server.py", line 1080, in __init__
    fixed_server_id=fixed_server_id,
  File "/media/veracrypt1/sandbox/repos/poc-datalake-pipeline/dlpoc-venv/lib/python3.7/site-packages/dagster/grpc/server.py", line 991, in open_server_process
    wait_for_grpc_server(server_process, client, subprocess_args)
  File "/media/veracrypt1/sandbox/repos/poc-datalake-pipeline/dlpoc-venv/lib/python3.7/site-packages/dagster/grpc/server.py", line 931, in wait_for_grpc_server
    f"Timed out waiting for gRPC server to start with arguments: \"{' '.join(subprocess_args)}\""

It is not a resource issue, as I'm running this locally with an i7 10th gen and 32gig RAM.

Alexander Jacob

07/30/2021, 1:51 PM

Hi, I am using `dagster-celery-docker`` I can pass env vars from the celery container via

env_vars

. Is there a way to define additional environment variables in the configuration of

celey-docker

i.e.

FOO: bar

? Cheers, Alex

Jonathan Mak

07/30/2021, 2:15 PM

How can I use dbt flag

--full-refresh

when using dagster-dbt? I tried to do something like

Copy code

context.resources.dbt.run(models=[m], full_refresh = full_refresh)

And got the run command compiled but I get this error when it actually tries to run

Copy code

dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2)

Wei Mei

07/30/2021, 2:25 PM

happy friday everyone. Am inquiring to see if dagster has available SLA definition/warnings?

szalai1

07/30/2021, 5:40 PM

hey, getting an odd error, from a very simple pipeline

keyError: 'result'

chrispc

07/30/2021, 11:27 PM

Hi team. I was wondering if you could help me with two questions. I am working whit the sensors. I upgrade Dagster version to 0.12.4. I am trying to return a SkipReason when the file I am pointing doesn't have new modifications. Here my sensor: I am getting this error (Is this the expected behavior?). Also, I would like to know how can I set the mode dependently in which environment I am working? (could be it configured using the GUI?) Thanks in advance.

Makoto

07/30/2021, 11:44 PM

I have two pipelines, A and B. If I want B to run upon successful run of A, would using sensors be the right way to do it? I’m thinking pipeline A would have a solid to write a file to S3 to indicate it was done, and then a sensor would pick that up and runs pipeline B. Please let me know if there’s a better way.

chrispc

07/31/2021, 10:39 PM

Hi team, sorry for making a lot questions. I am trying to configure the RetryPolicy but I am not sure about how to use the Backoff and Jitter. What is expected in this parameters? what is the Enum parameter? Thanks!

Xu Zhang

08/01/2021, 12:18 AM

Guys, I want to report a Dagit hanging issue that just happened that i cannot reproduce for now. I have two hosts that runs the exact same Dagit version with same config, they got started together and I checked they were both working after deployment. now one of the hosts’ Dagit is giving browser timeout error while the other is fine. Dagit process is running and the port is also being listened.

Xu Zhang

08/01/2021, 12:19 AM

after rebooting the problematic Dagit, it is all good now.

Jonathan Mak

08/01/2021, 4:05 AM

I tried to run

execute(_shell_command_ = cmd, _output_logging_ = 'STREAM', _log_ = context.log, _cwd_ = '/my/dir/path')

with dagster-shell but the temporary script directory was still set somewhere else, am I doing anything wrong there?