in the spark code a ton of stuff was baked into the code. so if job X read date partition

2019-01-01

from table A and wrote to table B, the scala code would know that table A would correspond to

<gs://our-data/table_a/2019/01/01/*.parquet>

and to

case class A(params...

) and then table B would be

<gs://our-data/table_b/2019/01/01/*.parquet>

and a paired case class also