Can I define my solids in a python class and use t...
# announcements
Can I define my solids in a python class and use them in defining the pipeline using the class object, when I creat my dag?
i think so? Not sure we’ve tried. The decorators turn the functions in to instances of
etc so as long as those instances get wired up correctly i think it should work
I recommend giving it a shot with a small toy version to verify
ūüíĮ 1
the top level thing that returns the
will have to be a function
This is what I tried:
Copy code
from dagster import execute_pipeline, pipeline, solid, DagsterType
import pandas as pd
import json
import os

class Foo:
       def __init__(context, self):
       self.target_sheets = {}
       self.targets = None
       self.target_path = r'/Users/foo/target'
       self.target_sheets = self.get_sheet_config()

       def get_sheet_config(self):
       with open(r'config.json') as sheets:
           return json.load(sheets)

       def change_current_directory(self, path):
       dir = os.path.join(os.path.expanduser('~'), path)
       return dir
   def check_for_targets(self, context):
       <|>(f'Changing paths: {target_path}')
       files = os.listdir(target_path)
       <|>(f'Available files: {files}')
       if len(files) > 0:
           return True
           <|>(f'Path and files: {target_path}, {files[0]}')
           return False

   def check_sheet(self, context, check: bool):
       if check:
               self.targets = pd.ExcelFile(file_name)
               if sorted(list(self.targets.sheet_names)) == sorted(self.target_sheets.keys()):
                   return True
                   return False
           except Exception as ex:
               <|>(f'Error to check {ex}')
               return False
           <|>(f'Targets file does not exist')
           return False

foo = Foo()
# dagit -f -n targets_pipeline`
def foo_pipeline():
And the error is ..
Loading repository... Traceback (most recent call last):  File "/opt/anaconda3/envs/grand/bin/dagit-cli", line 8, in <module>    sys.exit(main())  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/dagit/", line 209, in main    cli(auto_envvar_prefix='DAGIT')  # pylint:disable=E1120  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/click/", line 829, in call    return self.main(*args, **kwargs)  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/click/", line 782, in main    rv = self.invoke(ctx)  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/click/", line 1066, in invoke    return ctx.invoke(self.callback, **ctx.params)  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/click/", line 610, in invoke    return callback(*args, **kwargs)  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/dagit/", line 123, in ui    host_dagit_ui(host, port, storage_fallback, reload_trigger, port_lookup, **kwargs)  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/dagit/", line 131, in host_dagit_ui    return host_dagit_ui_with_execution_handle(  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/dagit/", line 154, in host_dagit_ui_with_execution_handle    app = create_app_with_execution_handle(handle, instance, reloader)  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/dagit/", line 223, in create_app_with_execution_handle    context = DagsterGraphQLInProcessRepositoryContext(  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/dagster_graphql/implementation/", line 114, in init    self.repository_definition = handle.build_repository_definition()  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/dagster/core/definitions/", line 406, in build_repository_definition    obj = self.entrypoint.perform_load()  File "/opt/anaconda3/envs/grand/lib/python3.8/site-packages/dagster/core/definitions/", line 441, in entrypoint
I generally prefer a class to define things rather than a file. Also decorating a class as solid is the best thing but I shouldn't be suggesting features. Kind of new to dagster
I dont see
so if you’re trying to target a
via the CLI it will have to be a top level function your
function on the class are going to need to be
- we can work with instance methods
No that is an old comment
how were you invoking dagit? is there any part of the stack trace thats missing? I dont see a specific error
Thanks...I will do that. Also, thanks for the prompt responses
dagit -f -n foo_pipeline
another thing to try before running dagit is just
to make sure it evaluates correctly
Hey it works. Changes: Make the solids also static methods and it also didn't like the f string...I don't know why
ūüĎć 2
I found out about the f string using Python
f string issue only when I run Python
Thanks for the help
fwiw, you can also use the
class directly!
I started using dragster today, beside reading about it and really like it. I have to read about SolidDefinition and jump on it.