dagster #announcements

Xu Zhang

11/16/2020, 7:07 AM

I managed to use

InProcessRepositoryLocationOrigin

to build what i need which is managing pipelines completely in memory w/o dealing with physically Python files, but still would love to know what happened above.

Richard Fisher

11/16/2020, 7:21 AM

Hi, how can I use the solid selection syntax to query for a solid inside a composite? I’ve tried

composite_name.solid_name

, but this doesn’t seem to work. I’m trying this through the Dagit playground, but I’d like it to run through

execute_pipeline

Charm Malubag

11/16/2020, 12:49 PM

Hi! Is there a way to limit the number of runs fetched from the dagster DB when opening up the runs page in the dagit UI? Or if none, is there a way to safely reset the DB so the historical runs get removed from the DB? We have tried

dagster run wipe

and

dagster schedule wipe

, and though these were able to reset the runs and schedules from the schedules page, the DB still seems to contain historical data, and the runs page still takes long to load.

Xu Zhang

11/16/2020, 10:00 PM

Guys, im using Gunicorn to start the Flask app that Dagit created: the command i used is

Copy code

['/export/content/lid/apps/dagster-web/i001/bin/gunicorn', '--bind', 'localhost:5002', '--chdir', '/export/content/lid/apps/dagster-web/i001', '--pid', '/export/content/lid/apps/dagster-web/i001/logs/dagster-web.pid', '--name', 'dagster-web', '--config', '/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagsterweb/gunicorn_conf.py', 'dagsterweb.webapp:app', '--access-logformat', '%(h)s %(p)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s" %(T)ss %(D)sus', '--log-config', '/export/content/lid/apps/dagster-web/i001/logs/gunicorn_logging.ini']

however, im seeing this error in the log:

Xu Zhang

11/16/2020, 10:00 PM

The way i get the Flask app from Dagit is:

Copy code

with DagsterInstance.get() as instance:

    target = WorkspaceFileTarget(paths=WORKSPACE_PATHS)
    with workspace_from_load_target(target) as workspace:
        if not workspace:
            raise Exception("Failed to created workspace")

        return create_app_from_workspace(workspace, instance)

Xu Zhang

11/16/2020, 10:00 PM

Copy code

/export/apps/python/3.7/bin/python3.7: Error while finding module specification for 'dagster.grpc' (ModuleNotFoundError: No module named 'dagster')
Loading repository...
KafkaClient cannot be created for Log Handler in DEV fabric. No events will be sent!
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/nbformat/notebooknode.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  from collections import Mapping
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  from collections import MutableMapping, Sequence  # noqa
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  from collections import MutableMapping, Sequence  # noqa
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/graphene/relay/connection.py:2: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  from collections import Iterable, OrderedDict
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/cli/workspace/workspace.py:50: UserWarning: Error loading repository location dagsterweb.examples.hello:(DagsterIPCProtocolError) - dagster.serdes.ipc.DagsterIPCProtocolError: Timeout: read stream has not received any data in 15 seconds
Stack Trace:
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/cli/workspace/workspace.py", line 43, in _load_handle
    handle = RepositoryLocationHandle.create_from_repository_location_origin(origin)
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/core/host_representation/handle.py", line 47, in create_from_repository_location_origin
    return ManagedGrpcPythonEnvRepositoryLocationHandle(repo_location_origin)
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/core/host_representation/handle.py", line 172, in __init__
    lazy_load_user_code=True,
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/grpc/server.py", line 1095, in __init__
    lazy_load_user_code=lazy_load_user_code,
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/grpc/server.py", line 1008, in open_server_process
    wait_for_grpc_server(output_file)
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/grpc/server.py", line 955, in wait_for_grpc_server
    event = read_unary_response(ipc_output_file, timeout=timeout)
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/serdes/ipc.py", line 39, in read_unary_response
    messages = list(ipc_read_event_stream(output_file, timeout=timeout))
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/serdes/ipc.py", line 142, in ipc_read_event_stream
    timeout=timeout
  location_name=location_name, error_string=error_info.to_string()
Exception ignored in: <function ManagedGrpcPythonEnvRepositoryLocationHandle.__del__ at 0x10e571c20>
Traceback (most recent call last):
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/core/host_representation/handle.py", line 245, in __del__
    self.is_cleaned_up,
  File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_c0ef7b83c436eed3e1fde58bf84f5616e03c48d7605c710cc48b8158b3748442/site-packages/dagster/core/host_representation/handle.py", line 241, in is_cleaned_up
    return not self.client
AttributeError: 'ManagedGrpcPythonEnvRepositoryLocationHandle' object has no attribute 'client'

Xu Zhang

11/17/2020, 5:52 AM

for people who are interested, the above exception should be caused by the following reason:

Copy code

I think this is something due to how dagsterweb got packed. it got packed into a .pyz with gunicorn + Flask app inside as a standalone app, so yes, at the very beginning, it indeed get booted up via /export/apps/python/3.7/bin/python3.7 , that is why at the running time, we see /export/apps/python/3.7/bin/python3.7 being sys.executable; however, that super-long site-packages directory is dynamically added into the sys.path, which means at running time, dagster libs can be located successfully within the same process; but since we are using sys.executable to execute another app, we are out of luck.
the only solution i could think of is to install dagster lib into the site packages of /export/apps/python/3.7/bin/python3.7 , but that would be ugly as typically it would cumbersome to install a lib on the host without going through the build & deploy pipeline.

Xu Zhang

11/17/2020, 8:09 AM

resolved the issue above using the following code before initializing the Dagster instance:

Copy code

CURRENT_PATH = ":".join([p.strip() for p in sys.path if p])

EXECUTABLE_PATH = check_output([sys.executable, '-c', 'import sys; print(":".join([p.strip() for p in sys.path if p]))']).decode().strip()

os.putenv('PYTHONPATH', ':'.join([CURRENT_PATH, EXECUTABLE_PATH]))

schrockn

11/17/2020, 1:40 PM

Release Schedule and Dropping Python 2 Support Just wanted to give a couple of quick updates for everyone. 1) We have pushed our release our 0.10.0 release (code-named “The Edge of Glory”) to January 7th, 2021. We felt like our current release schedule was too close to the holiday break. We will giving more details about this release as the date approaches. We talked about our new fault-tolerant scheduling infrastructure at our last community meeting:

https://youtu.be/6nKCd2TaXik?t=2136▾

. There are other improvements coming as well. 2) Starting with that release, we will no longer support Python 2. Based on the lack of objections in this discussion https://github.com/dagster-io/dagster/discussions/3208 as well as information from pypistats and our own telemetry, we have determined that usage of Dagster by Python 2 users is very minimal. Dropping Python 2 will reduce our surface area and allow us to build more “python-3-native” public APIs for Dagster, which we are excited about.

3️⃣ 1

ladydaga 13

Jaakko Kangasharju

11/18/2020, 8:46 AM

Is there a way to see the date of the run in dagit on an individual run's page? The event list has timestamps for each event but those show only the time, not the date, and I'm not seeing the date anywhere

Felix Yuan

11/18/2020, 10:34 AM

anybody knows why 'segmentation fault' occurs while running 'dagit -f hello_dagster.py'

Keval

11/18/2020, 7:29 PM

When calling solid with passing argument getting this error. So initial argument can not be passed directly? it always needs to return from another solid? How to pass dynamic value to solid in that case? Can some one help me? I am newbie so sorry for such silly question

Copy code

download_file("file_path")

Jake Smart

11/19/2020, 2:55 AM

Is there a way to access the repository name from a run context/solid handle?

Istvan Darvas

11/19/2020, 2:15 PM

Hi Guys! I am very new with Dagster, actually I am evaluating with this cool technology. I am trying to setup a pipeline with external parameters. So i have read the documentation and watch some presentation on youtube and I saw some solution for external (not scheduled) based pipeline invocation. Well I am trying to simulate this scenario, but it is not clear for me how can I give parameters to this event based pipeline during development, because I always get error 😉. I am going to put some code, screenshot into the thread. Any help would be appreciated.

Keval

11/19/2020, 9:01 PM

How to loop pipeline? I have csv having URL list I want to develop pipeline that will take each URL > Download File from URL > Run other solid and convert data into target format. So should I just create loop in python and pass each argument from CSV? or is there any other better approach ?

Frank Dekervel

11/19/2020, 9:33 PM

hello, i'm trying to figure out how to use dagster-pyspark. i run spark on k8s so i have to set a lot of spark config variables (k8s serviceaccount, ...) i try pyspark_resource.configured({"spark_conf":{}}) but it seems to ignore a lot of my variables, eg spark.kubernetes.authenticate.driver.serviceAccountName

👀 1

user

11/19/2020, 11:22 PM

Chris DeCarolis just published a new version: 0.9.20. See https://github.com/dagster-io/dagster/releases/tag/0.9.20 for the release notes.

🙌 1

👍 1

schrockn

11/19/2020, 11:53 PM

Hey all! I wanted to highlight@Ben Gotow’s improvements to our asset view in the Asset Catalog in 0.9.20. This is a huge improvement in usability.

schrockn

11/19/2020, 11:53 PM

If you want a nice video runthrough of the new screen you can check this out: https://www.loom.com/share/0afedc3ffc0a45479ba6a8e9ae34d0e7

👍 5

dwall

11/19/2020, 11:54 PM

amaze

amaze 5

Istvan Darvas

11/20/2020, 7:27 AM

Hi Guys! How can I use TZ Aware datetimes in dataframes? I put some code into the thread.

Xu Zhang

11/20/2020, 8:14 AM

Website mobile view bug report: scroll down & up can not reach to the bottom or top the first try (second try would work)

Xu Zhang

11/20/2020, 8:15 AM

Image from iOS.MP4

Xu Zhang

11/20/2020, 8:16 AM

Tried on both safari and chrome. iOS 14.2.1

Istvan Darvas

11/20/2020, 12:21 PM

Guys! Can I skip type checking, in dagster_pandas, for particular colums? because changeing between TZ Naiv and Aware will cause me a lot of trouble. What is the best approache tackle with this? Is there a solution to handle TZ for PandasColumn.datetime_column or can I skip type check pandas.TimeStamp? or other way?

Istvan Darvas

11/20/2020, 2:15 PM

what is the best executor for a stream processing if there is any? So a pipeline workflow would calculate aggregates in a hierachy (small computes, but high concurency)

Paul Wyatt

11/20/2020, 5:42 PM

Is there a way to run a pipeline with hooks applied as reconstructable? When attempting to do so I see

dagster.core.errors.DagsterInvariantViolationError: Reconstructable target should be a function or definition produced by a decorated function, got <class 'dagster.core.definitions.pipeline.PipelineDefinition'>

The use case is writing tests for hook-decorated pipelines. Another potentially workable solution would be if hooks could be applied as part of an

execute_pipeline

call.

Nate Loker

11/20/2020, 9:46 PM

Hey, I am working through the airline_demo and I keep getting an error when I execute load_q2_sfo_weather.compute. I will add the error output in the thread. Any help is appreciated.

Xu Zhang

11/21/2020, 5:25 AM

not sure whether this is an issue specifically happened on my company’s host or not: in the pipeline run page, the waterfall does not stop when the run has succeeded. I had to refresh the page to get the latest data:

Kapture 2020-11-20 at 21.22.58.mp4

Xu Zhang

11/21/2020, 5:26 AM

(I had to refresh the page at around 18 seconds of the video in order to make the run detail page to show correct info)