jay
10/01/2021, 6:18 PMrun_config
. Dagster cannot serialize the Dataframe so I converted it to JSON but then I get this error: `OverflowError: string longer than INT_MAX bytes`I tried to to compress the string using zlib but then I am getting TypeError: Object of type bytes is not JSON serializable
has anyone encountered this?Benoit Perigaud
10/02/2021, 9:53 PMdagster-daemon run
command is always eating 100% of one of my CPUs.
Here is the error in journalctl (I'm on dagster 0.12.12):
Oct 03 08:49:03 raspberrypi bash[10714]: 2021-10-03 08:49:03 - dagster-daemon - ERROR - Thread for SCHEDULER did not shut down gracefully
Oct 03 08:49:03 raspberrypi bash[10714]: Traceback (most recent call last):
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/bin/dagster-daemon", line 8, in <module>
Oct 03 08:49:03 raspberrypi bash[10714]: sys.exit(main())
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/lib/python3.7/site-packages/dagster/daemon/cli/__init__.py", line 135, in main
Oct 03 08:49:03 raspberrypi bash[10714]: cli(obj={}) # pylint:disable=E1123
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/lib/python3.7/site-packages/click/core.py", line 829, in __call__
Oct 03 08:49:03 raspberrypi bash[10714]: return self.main(*args, **kwargs)
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/lib/python3.7/site-packages/click/core.py", line 782, in main
Oct 03 08:49:03 raspberrypi bash[10714]: rv = self.invoke(ctx)
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
Oct 03 08:49:03 raspberrypi bash[10714]: return _process_result(sub_ctx.command.invoke(sub_ctx))
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
Oct 03 08:49:03 raspberrypi bash[10714]: return ctx.invoke(self.callback, **ctx.params)
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/lib/python3.7/site-packages/click/core.py", line 610, in invoke
Oct 03 08:49:03 raspberrypi bash[10714]: return callback(*args, **kwargs)
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/lib/python3.7/site-packages/dagster/daemon/cli/__init__.py", line 48, in run_command
Oct 03 08:49:03 raspberrypi bash[10714]: controller.check_daemon_loop()
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/lib/python3.7/site-packages/dagster/daemon/controller.py", line 237, in check_daemon_loop
Oct 03 08:49:03 raspberrypi bash[10714]: self.check_daemon_heartbeats()
Oct 03 08:49:03 raspberrypi bash[10714]: File "/home/pi/.envs/dagster/lib/python3.7/site-packages/dagster/daemon/controller.py", line 212, in check_daemon_heartbeats
Oct 03 08:49:03 raspberrypi bash[10714]: failed_daemons=failed_daemons
Oct 03 08:49:03 raspberrypi bash[10714]: Exception: Stopping dagster-daemon process since the following threads are no longer sending heartbeats: ['SCHEDULER']
Oct 03 08:49:04 raspberrypi systemd[1]: dagster-daemon.service: Main process exited, code=exited, status=1/FAILURE
Oct 03 08:49:04 raspberrypi systemd[1]: dagster-daemon.service: Failed with result 'exit-code'.
Oct 03 08:49:04 raspberrypi systemd[1]: dagster-daemon.service: Service RestartSec=100ms expired, scheduling restart.
Oct 03 08:49:04 raspberrypi systemd[1]: dagster-daemon.service: Scheduled restart job, restart counter is at 6.
Oct 03 08:49:04 raspberrypi systemd[1]: Stopped Daemon for dagster.
Oct 03 08:49:04 raspberrypi systemd[1]: Started Daemon for dagster.
The heath page tells me: "Not running - No recent heartbeat"marcos
10/04/2021, 2:05 AMlog_output()
waits until all 25 numbers have been returned before executing. I had hoped that function would start right away after the first number was returned. Is that type of functionality possible?Rubén Lopez Lozoya
10/04/2021, 10:14 AMdagster-postgres
library dependency from our code since it was not being imported anywhere and we had no issues developing locally with and without Docker. However, once we deployed Dagster to our cluster using the provided Helm chart, our pipelines would get stuck in STARTING
because our deployment missed the mentioned dependency. Is there any way to have this dependency included somehow in the dagster core package or be automatically handled by the Helm chart itself somehow? It's really confusing having to add a library that is not imported anywhere 😞Arun Kumar
10/04/2021, 6:06 PMAnaqi Afendi
10/04/2021, 8:13 PMAndy H
10/04/2021, 10:18 PMdagster_aws.s3.sensor
?Sandeep Mankikar
10/04/2021, 10:42 PMSimon Späti
10/05/2021, 8:40 AMinput_solid
that handles that and distribute it to other solids. Downside: This is very messy and you do not get the actual data flow anymore.
I read the message from @sandy, would you suggest to create a RootInputManager for that use case which would contain mf_handler_version
, mf_converter_version
and input_file
, that could then be used as an input for all the solids needed. expectation_config_file
and data
can be directly stated as an input to the specific solid. Do you agree?
Or do you see other solutions? @alex I saw other comments from you where you suggested resource, but that be a bit of an overkill for our use case, right? Appreciate any help or hint a lot!Cody Hutchens
10/05/2021, 3:25 PMgeoHeil
10/05/2021, 4:04 PM<http://context.log.info|context.log.info>(os.path.join(context.run_id, context.step_key, context.name))
nor yield EventMetadataEntry.string(self._get_path(context), label="xxxxx")
seem to show up in the dagster logs.Gillian Sharer
10/05/2021, 4:42 PMgeoHeil
10/05/2021, 5:34 PMChris Evans
10/05/2021, 8:55 PM0.12.11
. Firstly, Dagit seems to detect errors around missing config for ops that are deselected. Secondly, even if a subset of a graph is selected in the Dagit playground, all ops will end up running when execution is launched.
from dagster import graph, op
@op(config_schema={"param": str})
def hello(context):
...
@op(config_schema={"param": str})
def hello2(context):
...
@graph
def test_graph():
hello()
hello2()
test_job = test_graph.to_job()
Dalin Kim
10/06/2021, 3:25 AMMatthew Smicker
10/06/2021, 7:26 PMdc = get_dc() # this solid returns an object that has a bucket attribute
export_operation(bucket=dc.bucket)
I could make a basic solid that takes an object of the specific type of dc and returns the bucket attribute as a string but was hoping to avoid that. Appreciate any advice on patterns to follow - or if this approach is a bad one, appreciate that feedback as well 🙂 i.e. should I stick to basic types and yield all the objects attributes of potential interest (there are ~15)?Tyler Ellison
10/06/2021, 8:31 PMAmardeep Singh
10/07/2021, 5:45 AMStepan Dvoiak
10/07/2021, 1:35 PMdagit
with https on an external server? I do have ssl certs but cant find an option for dagit cli
The main problem is that Safari browser is forcing https connections and I cant disable it. The server has ssl certs for all sites that run on itRubén Lopez Lozoya
10/07/2021, 2:56 PMMartim Passos
10/07/2021, 4:39 PMENTRYPOINT ["dagster", "pipeline", "execute", "-f", "pipelines/IIIF_pipeline.py", "--preset", "debug"]
but runs successfully with no ENTRYPOINT
and
docker run my-container dagster pipeline execute -f pipelines/IIIF_pipeline.py --preset debug
Marc Keeling
10/07/2021, 5:54 PMAdam McCartney
10/08/2021, 9:07 AMThomas
10/08/2021, 12:17 PMNoah Sanor
10/08/2021, 2:19 PMNoah Sanor
10/08/2021, 2:25 PMsolid_selection
argument to only run some of the solids in the pipeline. When asserting validate_run_config
we are getting failures because the test does not know that we don't need the config values for solids not being run. Is there a solution or workaround for this besides adding "dummy" config values for the solids we aren't running?Chris Chan
10/08/2021, 7:20 PMGayathri Chakravarthy
10/08/2021, 9:29 PMKoby Kilimnik
10/10/2021, 10:43 AMKoby Kilimnik
10/10/2021, 10:45 AMKoby Kilimnik
10/10/2021, 10:45 AM