Navneet Sajwan
09/21/2022, 6:29 AMGintvilė Bergerytė
09/21/2022, 7:12 AMk8s_job_op
dynamically: https://docs.dagster.io/_apidocs/libraries/dagster-k8s#ops
For example my op would look something like this:
make_magic = k8s_job_op.configured(
{
"image": "my_image",
"args": ["python3", "script.py"],
"env_vars": [
"ENV_VAR_1",
"ENV_VAR_2"
]
"image_pull_secrets": [{"name": "secret1"}],
},
name="make_magic",
)
and I would like to reuse this op by somehow passing the image
, args
and env_vars
values. Is it possible to achieve that? Maybe k8s_job_op
could somehow access resources
?owen
09/21/2022, 5:21 PMHarry James
09/21/2022, 10:36 AMMarvinK
09/21/2022, 2:19 PMdocker.errors.NotFound: 404 Client Error for <http+docker://localhost/v1.41/containers/7d273d70cfca87e7f2e292ff71a4277e64eaeb7f62a963594b2b1bcdb0335912/start>: Not Found ("network docker_network not found")
Why does the networking not work?
This is my dagster.yaml
run_launcher:
module: dagster_docker
class: DockerRunLauncher
config:
env_vars:
- DAGSTER_POSTGRES_USER
- DAGSTER_POSTGRES_PASSWORD
- DAGSTER_POSTGRES_DB
network: docker_network
container_kwargs:
volumes: # Make docker client accessible to any launched containers as well
- /var/run/docker.sock:/var/run/docker.sock
Romain
09/21/2022, 4:04 PMJeffery
09/21/2022, 4:09 PMCharlie Bini
09/21/2022, 4:18 PMcontext.job_def.graph.name
returns sync_all
which is the name of the parent graph. these are all subgraphsBalázs Dukai
09/21/2022, 4:41 PMYang
09/21/2022, 6:48 PMDusty Shapiro
09/21/2022, 7:23 PMdagster-user-deployments
config?Antoine Valette
09/21/2022, 10:31 PMLeo Qin
09/22/2022, 1:36 AMwith_resources
, but when I try to materialize the asset from dagit i get an error that says __ASSET_JOB cannot be executed with the provided config. Please fix the following errors: Missing required config entry "ops" at the root
, if i shift-click and have it scaffold the job, i can tell that it knows all the configs needed, but the ones I set during asset definition aren't being respected. Any ideas what's going on?peay
09/22/2022, 11:35 AMGrigoriy Sterin
09/22/2022, 3:06 PMOptional[Dict[str, str]]
like this:
@graph
def my_graph(my_dict: Optional[Dict[str, str]]):
pass
I'm getting the following error:
dagster/core/workspace/context.py:554: UserWarning: Error loading repository location repo.py:TypeError: Optional[t] requires a single type. Got <dagster.core.types.python_dict._TypedPythonDict object at 0x7fbb4a0662b0>.
Is there any way around this? Thank you.Carter
09/22/2022, 3:45 PMMarc Keeling
09/22/2022, 3:53 PMHookContext
and OpExecutionContext
is?
It seems like there is some overlap there, but wanted to see what the experts say.Ryan Riopelle
09/22/2022, 4:47 PMSimon Vadée
09/22/2022, 5:10 PMclass PandasIOManager(IOManager):
def __init__(self, con_string: str):
self._con = con_string
def handle_output(self, context, obj: Iterator[pd.DataFrame]):
obj.to_sql(table, con=self._con, if_exists="append")
def load_input(self, context) -> Iterator[pd.DataFrame]:
"""Load the contents of a table as a pandas DataFrame."""
return pd.read_sql(f"SELECT * FROM {table}", con=self._con, chunksize=10)
@asset(
ins={"my_model": AssetIn(input_manager_key="pandas_df_manager")},
io_manager_key="pandas_df_manager",
)
def downstream_asset(upstream_asset):
for chunk in upstream_asset:
# do stuff
yield pd.Dataframe(result)
I get
dagster._core.errors.DagsterInvariantViolationError: Compute function for op "downstream_asset" yielded a value of type <class 'pandas.core.frame.DataFrame'> rather than an instance of Output, AssetMaterialization, or ExpectationResult. Values yielded by ops must be wrapped in one of these types. If your op has a single output and yields no other events, you may want to use `return` instead of `yield` in the body of your op compute function. If you are already using `return`, and you expected to return a value of type <class 'pandas.core.frame.DataFrame'>, you may be inadvertently returning a generator rather than the value you expected.
an tried wrapping my output without success 🙃
I made some research across issues and messages from this slack but everyone is using @op
and DynamicOutput
which I can’t do since I’m using SDAs produces by the dagster-dbt
integration 🤯 .
I found some people saying that SDA were not meant to be dynamic (outputs must be known when building the graph or something...) but then I don’t know how to do! Maybe I should use a different dagster API to achieve what I’m trying to do? help me out!Matthew Karas
09/22/2022, 6:48 PMZach
09/22/2022, 8:44 PMwh git:(master) ✗ poetry add dagster dagster-postgres
Using version ^1.0.10 for dagster
Using version ^1.0.5 for dagster-postgres
Updating dependencies
Resolving dependencies... (0.0s)
SolverProblemError
Because no versions of dagster-postgres match >1.0.5,<2.0.0
and dagster-postgres (1.0.5) depends on dagster (1.0.5), dagster-postgres (>=1.0.5,<2.0.0) requires dagster (1.0.5).
So, because wh depends on both dagster (^1.0.10) and dagster-postgres (^1.0.5), version solving failed.
UPDATE: Updating Poetry to the latest version fixes this. Feel free to delete if this is too irrelevant. 🙂Julian Mudd
09/22/2022, 9:24 PMpipenv install dagster dagit requests
Upon trying to run any command with either the dagster
or dagit
CLIs, I get this error:
ImportError: dlopen(/Users/papi/.local/share/virtualenvs/dagster-oVuOtnKT/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (__ZN6google8protobuf15FieldDescriptor12TypeOnceInitEPKS1_)
I saw in an earlier message a few months ago about an issue with the grpcic
library on Mac M1s. I tried installing with the --no-binary
flag but it didn't change anything. Would love some help here!Dave Muse
09/22/2022, 9:38 PMdaniel
09/22/2022, 9:55 PMSlackbot
09/22/2022, 10:25 PMWilliam
09/23/2022, 2:58 AMSanidhya Singh
09/23/2022, 3:17 AMFrank Dekervel
09/23/2022, 6:55 AMRoel Hogervorst
09/23/2022, 7:23 AMMarvinK
09/23/2022, 9:47 AM@op()
def set_environment(context):
os.environ['DATA'] = "someValue"
<http://context.log.info|context.log.info>(os.environ['DATA'])
@op(ins={"start": In(Nothing)})
def do_something(context):
<http://context.log.info|context.log.info>(os.environ['DATA']) <-- Env-Variable is not present anymore
@job()
def do_it_all_with_simplified_config():
do_something(start=set_environment())
Why is the env variable in do_something not present?
I set it in set_environment and can log it.
greetings