Gabriel Felipe
05/16/2023, 12:14 PMLeo Qin
05/16/2023, 7:19 PMAnushree Agrawal
05/17/2023, 1:43 AMAki Iwa
05/17/2023, 6:31 AMexecution:
config:
max_concurrent: 2
....
• job.py
....
@job(executor_def=multiprocess_executor)
def elt_job():
.....
output_1, output_2 = op_a(var=output_1)
op_b(var=output_1)
output_3 = op_c(var=output_2)
...
• op.py
...
@op
def op_a(context, var: str)
....
return "output_1", "output_2"
@op(ins={"var": In()})
def op_b(context, var: str)
....
@op(ins={"var": In()})
def op_c(context, var: str)
....
yield Output(value="output_3", output_name="output_3")
...
Phil Sheard
05/17/2023, 2:49 PMmultiprocessing
required by Dagster.Brandon Zhang
05/18/2023, 12:22 AMAshwin Kamath
05/18/2023, 1:03 AMassets
are structured in a way that I have a dynamic partition definition that is a sparse set of dates in the upstream asset and a monthly time window partition definition in the downstream asset. Has anyone run into issues with partition mappings pertaining to dynamic partitions?Edson Henrique
05/18/2023, 12:57 PMPhil Dreizen
05/18/2023, 6:48 PMdocker_container_op
. I’ve had success doing this locally with dagster dev
, but it’s been failing on Dagster Cloud. I’ve tried this with both the very simple example of using the busybox container to echo something, and using a private container registry. Both work locally, but fail on Dagster Cloud. The error I get seems to be unrelated to my own code.
dagster._check.CheckError: Failure condition: Couldn't import module dagster_cloud_serverless_agent.serverless.user_code_launcher when attempting to load the configurable class dagster_cloud_serverless_agent.serverless.user_code_launcher.ServerlessUserCodeLauncher
I’ll post the full error in the thread. 🧵
Is Dagster Cloud supposed to be able to support this feature?Arturo Martínez
05/19/2023, 2:24 PMnatalie nakamine
05/22/2023, 6:06 PMLeo Qin
05/23/2023, 3:14 AMtyping-extensions
version differed between the early and late deploys (was 4.5.0
, now 4.6.0
) - dagster calls for typing_extensions>=4.4.0
.
The new version (4.6.0
) was released between 5-6pm US Pacific Time... changenotes here. Our CI/CD pipeline installs dagster fresh on each run, so there is some reason to believe that one of the changes may not be forward compatible.
Pinning typing-extensions
to 4.5.0
resolved the issue.Oliver Gate
05/23/2023, 7:14 AMXuan Mai Ho
05/23/2023, 8:03 PMprod
with:
dagster-cloud serverless deploy . --location-name my_location --package-name mypackage
Currently using serverless. Thanks!Arturo Martínez
05/23/2023, 9:29 PMsubmit_job_execution
function for executing jobs in Dagster cloud and I see that the function has a parameter called run_config
, do you have an example of how can I pass a dictionary or a dataframe or both using that parameter? Thanks!Steve Jackson
05/24/2023, 8:50 PMArturo Martínez
05/24/2023, 9:32 PMsubmit_job_execution
function.Here is what Im trying to do: https://pastebin.pl/view/50193ecd (pass some parameters to my asset using submit_job_execution
)whats the correct way to create the dictionary for run_config
?Shalabh Chaturvedi
05/25/2023, 5:46 PMDaniel Gafni
05/26/2023, 10:45 AMdagsterCloud:
deployments:
- prod
branchDeployments: true
As I understand, it should appear for both prod
and Branch Deployments, right?
Why can this happen?Qwame
05/26/2023, 2:25 PMArturo Martínez
05/26/2023, 4:15 PMdagster-cloud serverless deploy-python-executable ./my-project --location-name whatever --package-name my_project
sometimes I receive this error: https://pastebin.pl/view/5e6d9ebf . Im using poetry environment and my requirements.txt is: https://pastebin.pl/view/3e70ddecZach
05/27/2023, 12:36 AMAugustin YAICH
05/29/2023, 2:01 PMNoud Jaspers
05/31/2023, 12:42 PMYou do not have permission to activate serverless deployments.
What should I do to fix this?Shane Kalepp
05/31/2023, 8:01 PMError: buildx failed with: ERROR: failed to solve: process "/bin/sh -c if [ -f \"setup.py\" ]; then pip install .; fi" did not complete successfully: exit code: 1
Josh Kutsko
05/31/2023, 9:09 PMError: No Dagster Cloud agent is actively heartbeating. Make sure that you have a Dagster Cloud agent running.
Checking the agents window, it looks like we have one inactive agent, and the message ‘Activating serverless deployment’ that’s been spinning for a few hours (screenshot). Can someone help me get this back up and running?Steve Jackson
06/01/2023, 7:06 PMResource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'." (RequestToken: 80eb7a80-f711-36a6-3712-2d14c936d93a, HandlerErrorCode: GeneralServiceException)
Steve Jackson
06/01/2023, 7:25 PMZach
06/01/2023, 11:57 PMJosh Kutsko
06/02/2023, 6:06 PM