dagster #announcements

Jeff Hulbert

03/19/2021, 1:23 PM

I'm attempting to use the helm chart to deploy Dagster but I keep getting the example user code instead of my code even though I updated the dagster-user-deployments section with my user code image. Am I missing a configuration in values.yaml?

Copy code

dagster-user-deployments:
  # Creates a workspace file with the gRPC servers hosting your user code.
  enabled: true

  # If you plan on deploying user code in a separate Helm release, set this to false.
  enableSubchart: true

  # List of unique deployments
  deployments:
    - name: "dagster-code"
      image:
        repository: "<http://repo.azurecr.io/dagster-code|repo.azurecr.io/dagster-code>"
        tag: latest
        # Change with caution! If you're using a fixed tag for pipeline run images, changing the
        # image pull policy to anything other than "Always" will use a cached/stale image, which is
        # almost certainly not what you want.
        pullPolicy: Always
      # Arguments to `dagster api grpc`.
      # Ex: "dagster api grpc -m dagster_test.test_project.test_pipelines.repo -a define_demo_execution_repo"
      # would translate to:
      # dagsterApiGrpcArgs:
      #   - "-m"
      #   - "dagster_test.test_project.test_pipelines.repo"
      #   - "-a"
      #   - "define_demo_execution_repo"
      dagsterApiGrpcArgs:
        - "-f"
        - "/opt/dagster/app/repository.py"
      port: 3030

King Chung Huang

03/19/2021, 7:04 PM

I've just posted my thoughts on

PresetDefinition

PartitionSetDefinition

as a discussion. Originally wrote this last year, but never got around to finishing it off. I'd love to know what others think of it! tl;dr: I think the distinction between the two is unnecessarily complicated. They can be combined by adding general variable support to

PresetDefinition

. https://github.com/dagster-io/dagster/discussions/3904

👀 4

esztermarton

03/19/2021, 7:08 PM

Hi all 👋 I'm attempting to deepcopy a ModeDefinition object. It's giving me an error. Am attaching the code snippet - would be great to understand - is it wrong to be doing this? I want to do it so that I can have a default mode defined, and allow pipelines to customise this (without mutating the original object).

Yan

03/21/2021, 8:54 AM

Hi Is there way in dagster cli to run pipeline from specific solid taking inputs from some parent run? Thanks

geoHeil

03/21/2021, 10:53 AM

When trying to create a dagster repository as a separate python package in https://github.com/geoHeil/dagster-demo/tree/master/use_cases/usecase_b I have installed it with:

pip install .

A manual import:

Copy code

from usecase_b.hello_b import hello

works nicely. However, when trying to start dagit I get a stacktrace of:

No module named 'hello_b'

geoHeil

03/21/2021, 10:53 AM

Copy code

Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 188, in load_python_module
    module = importlib.import_module(module_name)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/usecase_b/repository_b.py", line 2, in <module>
    from hello_b import hello_b_pipeline
ModuleNotFoundError: No module named 'hello_b'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/__main__.py", line 4, in <module>
    grpc_command()
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/cli/api.py", line 402, in grpc_command
    server = DagsterGrpcServer(
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 805, in __init__
    self._api_servicer = DagsterApiServer(
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 217, in __init__
    self._repository_symbols_and_code_pointers.load()
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 98, in load
    self._loadable_repository_symbols = load_loadable_repository_symbols(
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 122, in load_loadable_repository_symbols
    loadable_targets = get_loadable_targets(
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/utils.py", line 37, in get_loadable_targets
    else loadable_targets_from_python_package(package_name)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/cli/workspace/autodiscovery.py", line 23, in loadable_targets_from_python_package
    module = load_python_module(package_name, remove_from_path_fn=remove_from_path_fn)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 210, in load_python_module
    raise error
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 182, in load_python_module
    module = importlib.import_module(module_name)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/usecase_b/repository_b.py", line 2, in <module>
    from hello_b import hello_b_pipeline
ModuleNotFoundError: No module named 'hello_b'
/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/cli/workspace/workspace.py:105: UserWarning: Error loading repository location u_B:dagster.core.errors.DagsterUserCodeProcessError: ModuleNotFoundError: No module named 'hello_b'

Stack Trace:
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 805, in __init__
    self._api_servicer = DagsterApiServer(
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 217, in __init__
    self._repository_symbols_and_code_pointers.load()
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 98, in load
    self._loadable_repository_symbols = load_loadable_repository_symbols(
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 122, in load_loadable_repository_symbols
    loadable_targets = get_loadable_targets(
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/utils.py", line 37, in get_loadable_targets
    else loadable_targets_from_python_package(package_name)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/cli/workspace/autodiscovery.py", line 23, in loadable_targets_from_python_package
    module = load_python_module(package_name, remove_from_path_fn=remove_from_path_fn)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 210, in load_python_module
    raise error
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/core/code_pointer.py", line 182, in load_python_module
    module = importlib.import_module(module_name)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/usecase_b/repository_b.py", line 2, in <module>
    from hello_b import hello_b_pipeline


Stack Trace:
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/cli/workspace/workspace.py", line 100, in _load_handle
    handle = self.create_handle_from_origin(origin)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/cli/workspace/workspace.py", line 85, in create_handle_from_origin
    return origin.create_handle()
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/core/host_representation/origin.py", line 190, in create_handle
    return ManagedGrpcPythonEnvRepositoryLocationHandle(self)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/core/host_representation/handle.py", line 295, in __init__
    self.grpc_server_process = GrpcServerProcess(
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 1066, in __init__
    self.server_process = open_server_process(
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 981, in open_server_process
    wait_for_grpc_server(server_process, output_file)
  File "/usr/local/Caskroom/miniconda/base/envs/dagster-demo/lib/python3.9/site-packages/dagster/grpc/server.py", line 918, in wait_for_grpc_server
    raise DagsterUserCodeProcessError(

  warnings.warn(
Loading repository...

geoHeil

03/21/2021, 10:53 AM

Can you help me to install the package in a way that dagster is happy?

nikki

03/21/2021, 5:30 PM

Hi all, I feel like I’ve scoured everywhere I can scour for an answer to this: How can I obfuscate or hide sensitive fields in the “Playground” tab/view of the dagit UI? I see there is the StringSource config type but that seems to be for picking up env vars in the run_config. I want to designate certain fields in presets as “secret” and then have them be replaced with “****” or “HIDDEN” or whatevs in the Playground view. Is this possible?

lewis card

03/22/2021, 12:31 PM

Hi, What is the best way to debug a dagster gRPC server that I've kicked using the API, following this example https://github.com/dagster-io/dagster/tree/0.11.0/examples/deploy_docker ? The gRPC server container is not logging anything as far as I can tell, while

dagster api grpc-health-check

returns nothing. The error I am getting in dagit is:

Copy code

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1616415994.552532757","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5396,"referenced_errors":[{"created":"@1616415994.552531220","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"

Thanks for your help

szalai1

03/22/2021, 12:57 PM

Hi, we are upgrading to user deployments, question. As it's no longer possible to specify the executor image in the execution mode. Is there any way to specify different executor container on different pipeline from the same user code deployment ? (e.g data science deployment, executor with gpu driver, executor w/o drivers)

bklau-zap

03/22/2021, 2:51 PM

Q: what databases are supported for Dagster?

Deveshi

03/22/2021, 3:00 PM

Hi, just wondering which API is referenced in the comments here. And, whether it would allow me to access resources from inside a custom logger with the latest release? https://github.com/dagster-io/dagster/issues/3846

Jeff Hulbert

03/22/2021, 3:48 PM

Has anyone had any issues with getting the timeline to dynamically refresh while a pipeline is executing. I can reload the page during execution and get the timeline to update but not in real-time. This is both when running Dagit within an Azure Web App or within k8s (behind Ambassador Edge). Chrome web console has a message like "failed: WebSocket is closed before the connection is established." Are there some timeout settings that may need to be adjusted when Dagit is behind a load balancer type setup?

Ben Torvaney

03/22/2021, 4:36 PM

Hi folks, I’d like to share some configuration across a handful of solids. What’s the best way to do this? Conceptually, creating a new resource would fit; however, it seems a bit like overkill to me, since the desired configuration is very simple (a list of integer IDs).

Michael Lynton

03/22/2021, 9:45 PM

Hi - I am really new at this but looking for some direction. I went through the tutorials and I know just enough to be dangerous. I’m trying to re-build an existing python script which takes a couple urls separately, downloads them (csv), then does some renaming and deduping, and then combines the dataframes and loads to a db. (The existing script doesn’t leveral any functional programming either) I started with a basic solid, repurposing the

read_csv

concept from the tutorials. Since I wanted to use two csvs and pass each value into that solid, I created a composite solid (not sure if that is unneeded here - is that the right approach?). I did this, so that I could pass each url/csv into the read_csv function and get a dataframe output to work with downstream. I was able to run the pipeline just with my composite solid, which I felt was a win. I get some basic logging with number of lines in each dataframe. Pretty cool. My composite solid looks like this:

@composite_solid

def read_csv_multi():

read_intl = read_csv.alias("read_intl")

read_us = read_csv.alias("read_us")

return (read_intl(), read_us())

Now, where I am stuck is, I’m not sure if this is the right approach because, like I said, I want to use each dataframe (Intl & US) in other solids downstream, but probably separate solids because they are different structures and need to be processed differently. Do I need to do something with yielding output definitions on the composite solid, so that I can use the output as input in each of my “rename/prep” solids? I am pretty sure that I tried to do something like

yield Output(read_us(), "us_data")

inside of the composite solid above, but I got a weird message like:

Copy code

@composite_solid read_csv_multi returned problematic value of type <class 'generator'>. Expected return value from invoked solid or dict mapping output name to return values from invoked solids

Kinda feels like I’m spinning my wheels all to avoid duplicating my read_csv solid. Sure, I could duplicate it but I feel like I don’t really need to, right? Thanks for any pointers

user

03/22/2021, 10:34 PM

Johann Miller just published a new version: 0.11.1. See https://github.com/dagster-io/dagster/releases/tag/0.11.1 for the release notes.

johann

03/22/2021, 10:50 PM

0_11_1_Changelog

Stany Angoyi

03/23/2021, 8:39 AM

Hi there, I'm exploring Dagster (and Dagit), this awesome tool that you guys gave to the community. My question is, what's the best way to set the directory (

base_dir

) for sqlite to store run information? Can I use the sqlite that comes shipped with Python? Currently, when it's set to

base_dir: /home/stany/anaconda3/bin/sqlite3

base_dir: /home/stany/anaconda3/envs/dagster/bin/sqlite3

, it returns the following error:

yaml.scanner.ScannerError: mapping values are not allowed here

Any suggestion? Thanks, Stany

NawafSheikh

03/23/2021, 9:20 AM

Hi, I wanted to know if it’s possible to run pipeline on multiple cpu cores with using simple daemon run, like without using Kubernetes, docker or celery etc.? If not, then what is the best method to deploy dagster which doesn’t have much resource extensive pipelines i.e. most of the pipelines are used to fetch, read, a small parsing and then write a report?

Deveshi

03/23/2021, 12:16 PM

Hi, we have multiple aws environments and I am hosting my Dagster on a centralised EC2 instance. Does the s3_resource support 'Profiles'? Say I have two pipelines and they need to upload files to S3 buckets in separate env, using different credentials. As per the documentation, it says s3_resource is based on boto3.resource, that picks up default profile https://docs.dagster.io/_apidocs/libraries/dagster-aws#dagster_aws.s3.s3_resource

Rolf Schick

03/23/2021, 12:30 PM

How can I direkt a pipeline to execute a solid after another without using its output?

Steve Pletcher

03/23/2021, 8:21 PM

is there any way to configure a user deployment in dagster's helm chart to use the same service account as the main chart? the new version in 0.11.0 that spins it out as a subchart also appears to hardcode its service account name to be, by necessity, unique from the main service account.

Yugal Sharma

03/24/2021, 6:21 AM

is there a way in dagster to get user input at runtime? For example using input function of python

Cesar Benitez

03/24/2021, 7:51 AM

Hello,

Cesar Benitez

03/24/2021, 7:51 AM

I have a question abaout how to avoid :

Cesar Benitez

03/24/2021, 7:51 AM

OSError: [Errno 12] Cannot allocate memory

Cesar Benitez

03/24/2021, 7:52 AM

during pipeline execution

Sasha Gorelikov

03/24/2021, 10:42 AM

HI! Can you edit documentation on https://docs.dagster.io/deployment/run-coordinator Priorities It says:

dagster/priority

on a run (either in Dagit or on the pipeline definition in code), you can specify an integer value as the run’s priority but dagster(0.11.1) works with string

dagster.check.CheckError: Value in dictionary mismatches expected type for key dagster/priority. Expected value of type <class 'str'>. Got value 1 of type <class 'int'>.

thanks.

Sasha Gorelikov

03/24/2021, 10:45 AM

jonathan

03/24/2021, 1:48 PM

Hi there! Quick question, is there a way to create a success hook that triggers once the pipeline is complete (vs a solid)?