dagster #announcements

Michael T

02/11/2021, 9:34 PM

Hi - once I have decorated a function with @solid, the only way to call it is with a @pipeline or @composite_solid()? That is - I can’t “just call” the function?

Michael T

02/11/2021, 9:35 PM

^with^within

Noah K

02/11/2021, 9:35 PM

Correct

Noah K

02/11/2021, 9:35 PM

The result is actually a SolidDefinition object

Noah K

02/11/2021, 9:35 PM

Not a normal function 🙂

Michael T

02/11/2021, 9:36 PM

Right. So one has to buy into the dagster environment in toto.

Noah K

02/11/2021, 9:37 PM

No?

Noah K

02/11/2021, 9:37 PM

I mean you can just have the solid call some other normal function

Noah K

02/11/2021, 9:37 PM

It's still code 🙂

Noah K

02/11/2021, 9:37 PM

You could also do

foo_soild = solid()(foo)

Noah K

02/11/2021, 9:37 PM

to avoid overwriting

foo

Michael T

02/11/2021, 9:38 PM

Right - but that ends up constructing a parallel hierarchy in a way, of “solid”-ified functions.

Noah K

02/11/2021, 9:38 PM

Indeed

Noah K

02/11/2021, 9:38 PM

There is no magic 🙂

Noah K

02/11/2021, 9:38 PM

Write the code that does the thing you want

Michael T

02/11/2021, 9:39 PM

😉

paul.q

02/11/2021, 9:47 PM

We use Elastic search with logstash+filebeat to ingest our logs. We developed a custom JSON logger to plug in to dagster to support what we want to do. Among other things, we like to use the "extra" kwarg for logger methods (debug, warn, etc) to pass a dictionary of useful context that's meaningful to our logging dashboards. So I was surprised when it failed with "do not allow until explicit support is handled". Looking further I found this. Short of overriding all the methods in our custom logger to find a way to cram the "extra" blob into the log output in some other way does anyone have: 1. Any suggestion as to how we might achieve what we're after, or 2. An ideas as to when this part of log_manager will get some treatment to accomodate "extra"? thanks Paul

Basil V

02/12/2021, 1:14 AM

Hello, is there a way to reference an entire

.env

file for the DockerRunLauncher config rather than listing out each env variable? Currently it is:

Copy code

config:
    env_vars:
      - DAGSTER_PG_USERNAME
      - DAGSTER_PG_PASSWORD
      - DAGSTER_PG_DB
      - DAGSTER_PG_HOST

Ideally something like:

Copy code

config:
    env_file: .env

👀 1

bklau-zap

02/12/2021, 3:26 PM

Can we add a new

Community

channel to address questions from adopters and Dagster-ecosystem?

Paul Wyatt

02/12/2021, 7:53 PM

I've got schedules running successfully in 0.10 , but the actual schedule page is showing a croniter error

Copy code

Operation name: RunsRootQuery

Message: failed to find prev date

Path: ["repositoriesOrError","nodes",1,"schedules",1,"futureTicks"]

Locations: [{"line":183,"column":3}]

Stack Trace:
  File "/usr/local/lib/python3.6/dist-packages/graphql/execution/executor.py", line 452, in resolve_or_error
    return executor.execute(resolve_fn, source, info, **args)
  File "/usr/local/lib/python3.6/dist-packages/graphql/execution/executors/sync.py", line 16, in execute
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dagster_graphql/schema/schedules/schedules.py", line 96, in resolve_futureTicks
    tick_times.append(next(time_iter).timestamp())
  File "/usr/local/lib/python3.6/dist-packages/dagster/utils/schedules.py", line 20, in schedule_execution_time_iterator
    date_iter.get_prev(datetime.datetime)
  File "/usr/local/lib/python3.6/dist-packages/croniter/croniter.py", line 133, in get_prev
    return self._get_next(ret_type or self._ret_type, is_prev=True)
  File "/usr/local/lib/python3.6/dist-packages/croniter/croniter.py", line 210, in _get_next
    nth_weekday_of_month, is_prev)
  File "/usr/local/lib/python3.6/dist-packages/croniter/croniter.py", line 479, in _calc
    raise CroniterBadDateError("failed to find prev date")

Any advice?

Jai Kumaran

02/13/2021, 9:18 AM

Hi, I am using the execute_pipeline API. I would like to update my UI with the real time status using the events

dagster.core.storage.event_log

. I see there is a SQLite-backed event log storage, but I need a websocket probably for real time updates. Is that how it is done on the dagit UI?

mrdavidlaing

02/14/2021, 12:54 PM

Pipeline design question: I want to use some functionality from a Java library (specifically https://github.com/scify/JedAIToolkit) in a number of @solids in my pipeline. What options do I have? What are the trade offs being made for each option? 🧵

schrockn

02/15/2021, 5:40 PM

Hey everyone! Data Council just published a survey where they are asking folks to submit OSS projects they are excited about. It would be awesome if you could communicate your excitement for your favorite data orchestrator and three-legged octopus! It is super easy to fill out: https://form.jotform.com/210275762605050 Thanks! 🙏🏻🙏🏻🙏🏻

dagsir 3

✅ 8

🙌 1

Stas Sajin

02/15/2021, 6:51 PM

I was wondering how folks handle admin related stuff in Dagster. How can you make sure that users have access to certain pipelines or repos and not others? How can you admin which resources can be used by which systems/users?

Rafał Wojdyła

02/16/2021, 11:05 AM

👋 I'm trying to use

MemoizableIOManager

but I having hard time, and right now hitting 2 issues/questions, I would highly appreciate your help. I'm on dagster==0.10.5. 1.

has_output

appears to be getting called with context set to

None

, thus I can't validate if the external data exists, this is a simplified version of my current POC: https://gist.github.com/ravwojdyla/6a546e3fa65459b17413aac10c643f25 2. when I hard-code

had_output

to return

True

(just as a POC), I'm getting this error:

Copy code

for step in self.get_steps_to_execute_by_level()[0]:
IndexError: list index out of range

I was expecting to get an error in that case, but that specific error made me think how the memoization appears to work, is it fair to say that for memoization to work the "central" scheduler needs to have access the the previously run "steps" (to fetch config etc)?

Hamza Khurshid Butt

02/16/2021, 1:44 PM

Hi All! 🙂 I have a scenario where i am passing different fileNames to

RunRequest

method inside

sensor

, which are further processed by various solids, now when this sensor will run again after some interval, it is not going to re run the fileNames that have already been passed to it due to Idempotence using run keys
feature of sensor, So lets say my specific file does not get processed correctly or encounters any problem during processing, in this case it will not be rerun ever by sensor whereas i want to re run only for this specific file. Any Solutions ??

Richard Fisher

02/16/2021, 2:14 PM

Hi, we’re using the

dagster api grpc-health-check

command as our kubernetes livenessProbe for our pipelines container (every 5 seconds). This is causing significant load on the CPU. Aside from reducing the frequency of the liveness probe, are there any other fixes for this/plans to streamline this in a future release? We’re running 0.10.0

Scott Jones

02/16/2021, 3:11 PM

Hi everybody! 🤓 Does anybody here have experience running Julia tasks from Dagster? I found one repo on GitHub, where someone was using the

pyjulia

package to call into Julia from Python, I wanted to know if are other methods people had used, any recommendations. I am concerned about calling Julia, which uses asynchronous threads / tasks, from Python with it’s GIL.

👍 1

Rubén Lopez Lozoya

02/16/2021, 6:41 PM

Hey, is there any way to use mode definitions to, for example, set the value of a variable that holds the slack channel to which the hook failure post messages? If this is not appropiate, how would you guys do that? Or even to use mode defs so that the hook failure is only active in one of the modes (e.g production mode)

Cagatay K

02/16/2021, 9:59 PM

Is anyone having problems running

dagit

with the

–path-prefix

option to set a different root path than

? The "Playground" tab of the pipeline view that would let me edit the config and launch a run fails with a blank page in the browser (Firefox or Safari), and I see the errors in the screenshot in browser console. It looks like the path prefix doesn't take for the static resources, and the browser makes requests to

/static/css/...

and

/static/js/...

paths. This is on Dagster

0.10.5

and with the non-empty prefix

/pipelines