https://dagster.io/ logo
Powered by
d

dwall

10/09/2019, 4:26 PM
trying to understand an error I'm seeing re: input typing (in thread)
here is a snippet of what the code looks like:
Copy code
@resource(config={
    'token': Field(
        String,
        description='Your Stitch Connect API Token.',
        is_optional=False,
        is_secret=True
    )},
    description='This resource is for connecting to the Stitch Data Connect API.'
)
def stitch_resource(context):
    return PyStitch(token=context.resource_config.get("token"))

@solid(required_resource_keys={'stitch'})
def get_stitch_streams(context, sources: List[Int]) -> List:
    available_streams = []
    for source in context.resources.stitch.list_sources():
        for stream in context.resources.stitch.list_streams(source_id=source.get("id")):
            data = (source.get("name"), stream.get("stream_name"))
            available_streams.append(data)
    return available_streams

@pipeline(
    mode_defs=[
        ModeDefinition(resource_defs={'stitch': stitch_resource})
    ]
)
def resources_pipeline():
    get_stitch_streams()

environment_dict = {
    'resources': {
        'stitch': {
            'config': {
                'token': STITCH_TOKEN
            }
        }
    },
    'solids': {
        'list_stitch_sources': {
            'inputs': {
                'sources': {
                    'value': [1234]
                }
            }
        }
    }
}

if __name__ == '__main__':
    execute_pipeline(
        resources_pipeline,
        environment_dict=environment_dict
    )
and here is the error I'm seeing:
Copy code
agster.core.errors.DagsterInvalidConfigError: Pipeline "resources_pipeline" config errors:
    Error 1: Type failure at path "root:solids:list_stitch_sources:inputs:sources" on type "None". Value at path root:solids:list_stitch_sources:inputs:sources must be list. Expected: [{ json: { path: Path } pickle: { path: Path } value: Int }].
can I not supply complex types like 
List[Int]
via the python API?
m

max

10/09/2019, 4:27 PM
that seems like a bug, yeah
let me dig in
d

dwall

10/09/2019, 4:28 PM
thanks @max!
m

max

10/09/2019, 5:12 PM
ok, thanks for bearing with me
it's a little verbose, but currently you can write the following:
Copy code
environment_dict = {
    'resources': {
        'stitch': {
            'config': {
                'token': STITCH_TOKEN
            }
        }
    },
    'solids': {
        'list_stitch_sources': {
            'inputs': {
                'sources': [{'value': 1234}]
            }
        }
    }
}
and everything should work
the idea here is to support cases where, e.g., you have an input list of data frames that might be specified in some complex way, like 
[{'path': '/foo.csv', 'sep': ';', ...}, ...]
let us know if this feels super unnatural, though -- we are going to explore the possibility of special casing List[Scalar], or whether it makes sense to just disallow heterogeneous loading of the separate list elements in an input of type List[T]
2 Views
Powered by