https://dagster.io/ logo
Join Slack
Powered by
Hi there, I'm battling to figure out how to conver...
# ask-community
m
Hi there, I'm battling to figure out how to convert to the new pythonic resources, even after following this guide here. In the "legacy" code approach, I actually have a class in another package with general functions for interacting with a postgresql database. It has an 
__init__
function that also creates the engine used to then create the session to interact with the database. The config needed to be supplied are the connection details for the database. I then have a resource in the dagster code that imports this class and extends the methods by adding Dagster logging, etc. I have tried modifying the super class in Dagster by adding 
ConfigurableResource
and declaring attributes for the connection details at the top of the class. I am getting errors though about the attributes defined in the parent class not being present in the Dagster resource class. To try simplify, I also tried moving the 
__init__
function to the Dagster child class, but then get errors as follows: 
'ILDatabase' is a Pythonic resource and does not support manipulating undeclared attribute 'engine' as it inherits from 'pydantic.BaseModel' without extra="allow".
Are there any examples (I haven't been able to find anything) of interacting with a database, such as postgresql, using the new pythonic resources? My code examples posted in replies to this for more context.
The parent class defined used by multiple other repositories, the Dagster pipelines being one of them:
Copy code
class StriataDataIL(object):
    """
    A class for the Postgres database, making use of the SQL Alchemy ORM.
    """

    def __init__(
        self,
        postgres_user,
        postgres_password,
        postgres_host,
        postgres_port,
        postgres_schema,
        postgres_name,
    ):
        postgres_db = {
            "drivername": "postgresql",
            "username": postgres_user,
            "password": postgres_password,
            "host": postgres_host,
            "port": postgres_port,
            "database": postgres_name
        }

        self.engine = sqlalchemy.create_engine(
            URL(**postgres_db),
            connect_args={"options": "-csearch_path={},public".format(postgres_schema)},
        )

        self.Base = automap_base()
        self.Base.prepare(self.engine, reflect=True)

    def create_session(self):
        """
        Create a database connection session.
        """
        engine = self.engine
        Session = sessionmaker(bind=engine.connect(), autoflush=False)

        return Session()
The child class that inherits from the parent class in the Dagster code to define a resource:
Copy code
class ILDatabase(StriataDataIL):
    """
    A class for the Postgres database, which extends the striatadata StriataDataIL class to integrate with Dagster.
    """
    # INSERT FUNCTIONS -----------------------------------------------------------
    ...
How I've tried to modify this to update to the new pythonic resources and config:
Copy code
class ILDatabase(ConfigurableResource, StriataDataIL):
    """
    A class for the Postgres database, which extends the striatadata StriataDataIL class to integrate with Dagster.
    """

    postgres_user: str
    postgres_password: str
    postgres_host: str
    postgres_port: str
    postgres_schema: str
    postgres_name: str

    ...
q
I believe you can use 
PrivateAttr()
from pydantic here. You can do something like this
Copy code
from pydantic import PrivateAttr

class ILDatabase(ConfigurableResource):

    engine: engine_class_type = PrivateAttr()
    Base: base_class_type = PrivateAttr()
    create_session: Session_type = PrivateAttr()

    def setup_for_execution(self, context: InitResourceContext) -> None:
        self.engine = sqlalchemy.create_engine(
            URL(**postgres_db),
            connect_args={"options": "-csearch_path={},public".format(postgres_schema)},
        )
        self.Base = automap_base()
        self.Base.prepare(self.engine, reflect=True)
        self.create_session = sessionmaker(bind=engine.connect(), autoflush=False)
m
Thanks! Let me investigate this. I was also just looking at using the 
@property
decorator.
q
Yes, the 
create_session
can also be a property on the class
Copy code
@property
def create_session(self, ...): ....
🙏 1
7 Views
Open in Slack
PreviousNext
Powered by