Timo Vink
08/10/2023, 10:49 PMDagsterInstance
outside of a job/asset/sensor/etc. and without setting DAGSTER_HOME
?
Alternatively: Is there a way to initialize a DynamicPartitionsDefinition
with a static set of partitions (so that I don't need the instance at all)?Timo Vink
08/10/2023, 10:50 PMdagster dev
I would like to ensure a DynamicPartitionsDefinition
is always initialized with at least one partition so that I actually have something to materialize. I want dagster to be ephemeral (so don't want to set DAGSTER_HOME) and would prefer not to create a @sensor
or similar because I just need to add the partition as a one-off (there's nothing to sense -- don't need it to run on a 30s interval).Timo Vink
08/10/2023, 10:50 PMDagsterInstance.get()
I get an exception saying DAGSTER_HOME isn't set. If I use DagsterInstance.ephemeral()
I believe I'm getting a separate dagsterinstance from the one that was created by dagster dev
?
My hacky workaround for now is:
# Hacky way to get the current dagster instance without setting DAGSTER_HOME
tmp_dirs = [d for d in os.listdir('.') if d.startswith('tmp')]
latest_dir = max(tmp_dirs, key=lambda d: os.path.getmtime(d))
instance = DagsterInstance.from_config(latest_dir)
Daniel Gafni
08/11/2023, 7:56 AMinstance.add_dynamic_partitions(name, keys)
in the same script which starts Dagster for you.
I haven't tested this but it should work, you just need to get the correct DagsterInstance