hey all trying to figure out what the best cleanest way to d dagster #announcements

hey all, trying to figure out what the best/clean...

addy

05/05/2020, 3:04 PM

hey all, trying to figure out what the best/cleanest way to do the following would be: I have my pipeline configured to automatically send intermediate materializations to s3, works great saves to

<s3://bucket/prefix/step.result>

, I'm also using

dagster-pandas

and

TypeCheck

to create summary statistics on dataframes (and other types) I would like to send these to s3 as well so I have all of my intermediate results and

event_specific_data

in one place, I suppose I could do this with materializations, but then I'm already automatically materializing them, and I'm already creating the

EventMetadataEntry

in the type checks, and would feel hacky to re-add them to a materialization loving dagster so far, and it's crazy how fast you guys are improving it, started using it like 3 weeks ago and every day it feels a little more awesome to use

🎉 1

alex

05/05/2020, 4:10 PM

hmm ya this is tough

alex

05/05/2020, 4:11 PM

this is a little crazy - but you can get at the structured event stream from a custom logger, you could use that to grab the

event_specific_data

off of the type checks and send it somewhere

alex

05/05/2020, 4:14 PM

this is something were planning on writing some better support for - for now it would look a little like this:

Copy code

import logging
from dagster import logger
class EventListener(logging.Handler):
    def emit(self, record):
        event = record.dagster_meta.get('dagster_event')
        if not event.is_dagster_event:
            return

        dagster_event = event.dagster_event

        # bad name - just means output event
        if dagster_event.is_successful_output:
            # send dis
            dagster_event.step_output_data.type_check_data

@logger
def logger_based_event_listener(init_context):
    klass = logging.getLoggerClass()
    logger_ = klass(name, level=level)
    logger_.addHandler(EventListener())
    return logger_

alex

05/05/2020, 4:26 PM

would feel hacky to re-add them to a materialization

ya its not great but it might be your best bet

addy

05/08/2020, 2:04 PM

thanks, gonna play around with this for now, awesome that your planning on adding this later!

6 Views

Open in Slack

Previous Next