David Jayatillake
11/14/2022, 3:41 PMDavid Jayatillake
11/14/2022, 7:31 PMDavid Jayatillake
11/16/2022, 12:32 AMShalabh Chaturvedi
11/16/2022, 3:18 AM.github/workflows/deploy.yml
and `.github.workflows/branch_deployments.yml`:
a. Open deploy.yml
, copy and save the DAGSTER_CLOUD_URL
environment variable. You will need this later.
b. Replace both yml files in your repo with the updated files from here: https://github.com/dagster-io/dagster-cloud-serverless-quickstart/tree/main/.github/workflows
c. Replace the DAGSTER_CLOUD_URL
environment variable with the saved value.
2. Ensure you are using version 1.0.17 of dagster-cloud. Your setup.py or requirements.txt should not be pinned to an earlier version.
3. Add ENABLE_FAST_DEPLOYS: 'true'
to the top level env
section in both yml files.
To test this out:
1. Make a change to your code to trigger the updated workflows and ensure the code update works. Any subsequent code changes should get deployed fast - in about 30-40 seconds.
2. If you need to disable this feature, just remove the ENABLE_FAST_DEPLOYS
line.
To use fast deploys with your secrets, you must use the new secrets UI. This is now available under Environment variables in the Deployment tab.
We’d be happy to hear your feedback if you try this out!Manish Khatri
11/18/2022, 8:18 PMOperation name: AssetViewDefinitionQuery
Message: Internal Server Error (Trace ID: 3366240499784658422)
Path: ["assetOrError","definition","configField"]
Locations: [{"line":120,"column":3}]
Is there anyway this can be fixed?David Merritt
11/21/2022, 5:08 PMJacob Marcil
11/21/2022, 11:09 PM2022-11-19
This will run the backfill for that specific day. The problem I have is that dagster do the following things;
1. Delete the specific range rows from the table.
2. Create a temporary stage
3. PUT the asset into that stage
4. COPY the content of that stage to the table.
My problem is that the Delete
statement use the Between
clause like this
DELETE FROM DAGSTER_DB.MY_SCHEMA.MY_TABLE WHERE date BETWEEN '2022-11-19 00:00:00' AND '2022-11-20 00:00:00'
But since the Between
clause is inclusive both ways in Snowflake it translate to
DELETE FROM DAGSTER_DB.MY_SCHEMA.MY_TABLE WHERE date >= '2022-11-19 00:00:00' AND date <= '2022-11-20 00:00:00'
Then when I load data dagster always delete the next partition….
For example if I have some data for 2022-11-20
all data for that day will be deleted when I run the partition for 2022-11-19
Is there anything I can do to avoid that? Am I missing something obvious?
ThanksDagster Jarred
11/30/2022, 7:25 PMFélix Tremblay
12/04/2022, 3:57 PMZach P
12/19/2022, 11:44 PMRavishankar S R
12/22/2022, 10:41 AMLeo Qin
01/03/2023, 5:32 PMMaterializing unexpected asset key: AssetKey([a,b])
where a,b
is an asset from run X! Also, while asset materialization events for run X were successfully logged, there were no such events for run Y even though the logs indicate that the dbt job was successful.
I notice that the run has isolation:disabled
so i suspect this has something to do with it - any advice on what happened and how to prevent it from happening in the future?Manish Khatri
01/05/2023, 4:44 PMJoel Olazagasti
01/06/2023, 6:28 PMDagster Serverless Docker Deploy
job. on layer #10 of the dockerfile I just spin indefinitely on Preparing metadata (setup.py): started
The problem may have been when I removed the github code location and re-added it? Any insight into what steps I can take to resolve from here?David Merritt
01/12/2023, 7:55 PMAliénor Lougerstay
01/18/2023, 8:09 PMAliénor Lougerstay
02/14/2023, 7:53 PMJacob Marcil
02/15/2023, 4:33 PMbuild_schedule_from_partition_job
but they doesn’t seem to work for hourly partition.
https://docs.dagster.io/concepts/partitions-schedules-sensors/schedulesZach P
02/16/2023, 12:37 AMJacob Marcil
02/16/2023, 9:05 PMValueError: Tried to get asset partitions for an output that correponds to a partitioned asset that is not partitioned with a TimeWindowPartitionsDefinition.
Anyone know what I should do?Deividas Jodogalvis
02/23/2023, 3:35 PMJacob Marcil
02/23/2023, 7:58 PMmulti_asset
with partitions and I can’t find where I could configure the partitions for outs
.
I thought that they would take the partition_def
from the multi_asset
but it seems like they are not.
I’m always receiving the following error.
dagster._check.CheckError: Failure condition: The output has no asset partitions
Here a snipper of the configuration of my multi_asset
@multi_asset(
partitions_def=hourly_partitions,
op_tags={"kind": "s3"},
required_resource_keys={"aws_secrets"},
retry_policy=default_retry_policy,
outs={
"found": AssetOut(
is_required=False,
metadata={"priority": "high"},
io_manager_key="io_manager",
),
"not_found": AssetOut(
is_required=False,
metadata={"priority": "low"},
io_manager_key="io_manager",
),
},
)
def do_things(context):
json_response = {'what_ever':3}
if "tokens_found" in context.selected_output_names:
yield Output(value=json_response, output_name="found")
if "tokens_not_found" in context.selected_output_names:
yield Output(value=json_response, output_name="not_found")
Is there a way I can add a partition_Def
in the Output()
?Pete Hunt
03/08/2023, 4:29 AMSterling Paramore
04/05/2023, 8:27 PMZach P
04/06/2023, 6:08 PMdagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC
with either Error Code: Unreachable
or Error Code: Unavailable
. Is there some service issue going? I’ll post more of the errors in the thread.David Merritt
04/10/2023, 4:55 AMDavid Merritt
04/14/2023, 3:44 PMSterling Paramore
04/14/2023, 4:31 PMFélix Tremblay
05/23/2023, 10:23 PMAgent xxxxxxxx detected that run worker failed.
Over the past 7 days, we have encountered three instances of such errors, which account for approximately 0.1% of the total runs.
I was wondering if anyone has investigated the root causes behind these worker failures.
Additionally, I wanted to suggest the possibility of implementing an automatic process wherein the agent can initiate the launch of a new worker when such failures occur. This could help mitigate the impact of these issues and ensure smoother operations.
Thanks!Todd de Quincey
06/02/2023, 3:50 AM