Rubén Lopez Lozoya

02/26/2021, 2:49 PM
Hey guys I got a question regarding solid naming conventions. Let's say I have a repo where I have one pipeline for foods and another one for drinks. Ideally, I'd like to have my pipelines consist on just 3 composite solids (extract, transform, load) and then inside those composites just call other solids. The problem is that my ETL composite solids are different for foods and drinks, but still I would like to name the composite solids in the same way. Problem: Dagster does not allow me to have solids with the same names even though they are imported from different files. How should I solve this? I'd rather not have solids named c_solid_food_extract, c_solid_drink_extract, c_solid_food_transform, c_solid_drink_transform, c_solid_food_load and c_solid_drink_load.


02/26/2021, 3:37 PM
Here are some tools to consider
Copy code
# can specify definition name by argument

def extract():
Copy code
# can name invocations to make pipeline appear cleaner

def food_etl():
    e = food_extract.alias('extract')()
    t = food_transform.alias('transform')(e)
    l = food_load.alias('load')(t)
Copy code
# unique names *per repository* so could also split in to two repos

def food():

def drinks():

Rubén Lopez Lozoya

02/26/2021, 4:07 PM
Cool, thanks a lot Alex! You guys are so swift 🙂
👍 1