How does everyone organize generic ops? For instance I have an op that converts a list of dictionaries to a data frame, and can be used by anything that needs it. Does it make sense for that to be an op? or should it just be a core function call?

I have a directory for each code location (which we split by department/function, e.g. sales, engineering, reporting..) but I also have a directory in that level for generic ops/jobs/resources etc. without a repository or

Definitions

obj. In this way I can import the generics from that directory/package for use across each code location. This pattern has worked out pretty well for us so far :daggy-love:

@Guy McCombe Can you give a visual representation of your directory structure? I am trying to do something similar where I can use common ops across code locations.

I have an op that converts a list of dictionaries to a data frame, and can be used by anything that needs it. Does it make sense for that to be an op?

This really depends on how you intend to use this and how your graphs are structured. If you use a list of dicts as a stored intermediate representation, then it could make sense, if you’re generally passing dataframes between ops, then it should probably just be a function you invoke inside ops and or an IO manager.

Sure, here’s our tree, @Clayton Casey:

rvvup
├── __init__.py
├── data
│   ├── __init__.py
│   ├── assets
│   ├── dbt_project
│   ├── jobs
│   ├── ops
│   ├── resources
│   ├── schedules
│   └── utils
├── reporting
│   ├── __init__.py
│   ├── assets
│   ├── jobs
│   ├── resources
│   └── schedules
├── sales
│   ├── __init__.py
│   ├── assets
│   ├── jobs
│   ├── ops
│   └── resources
└── universal
    ├── __init__.py
    ├── resources
    ├── sensors
    └── utils

Hi @Guy McCombe, thanks for the tree, that's very helpful. What does your rvuup -> init.py look like? I'm assuming that is what loads the code repos. Also, how are you deploying? We are using Helm

We’re using helm too.

/rvvup/__init__.py

is blank, but there’s an

__init__

in each department directory (sales, reporting, etc.) which contains the

Definitions

for that code workspace location. They look a little like this:

from dagster import Definitions
from dagster import load_assets_from_package_module

from rvvup.data import assets
from <http://rvvup.data.jobs|rvvup.data.jobs> import ALL_JOBS
from rvvup.data.resources import RESOURCES
from rvvup.data.schedules import ALL_SCHEDULES
from rvvup.universal.sensors import ALL_SENSORS
from rvvup.universal.utils.constants import ENV

ALL_ASSETS = load_assets_from_package_module(assets)

defs = Definitions(
    assets=ALL_ASSETS,
    schedules=ALL_SCHEDULES,
    sensors=ALL_SENSORS,
    jobs=ALL_JOBS,
    resources=RESOURCES[ENV],
)

Ok, that was my next question. Do those locations each have their own images?

I don’t think that they are different images

For our purposes, I was thinking it might be good to have the different code locations running in different pods, but then I am not sure how that would translate into dagit showing all of the code locations

Eek, I’m not sure how much I can help you out on that front, I’m mostly just a python and graphs guy I don’t really know too much about pods etc