user
05/09/2020, 6:18 PMBen Smith
05/10/2020, 2:40 PMdagit -f file.py -n nodename
throws an error from (what seems to be) within dagster's config system. Anyone else seeing this?
File "c:\users\bsmith\appdata\local\continuum\anaconda3\lib\site-packages\dagster\config\field_utils.py", line 225, in is_potential_field
potential_field
File "c:\users\bsmith\appdata\local\continuum\anaconda3\lib\site-packages\dagster\config\field.py", line 117, in resolve_to_config_type
desc=VALID_CONFIG_DESC,
dagster.core.errors.DagsterInvalidDefinitionError: You have passed an instance of DagsterType Path to the config system (Repr of type: <dagster.core.types.dagster_type._Path object at 0x0000019207D41F08>). The DagsterType and config schema systems are seperate. Valid config values are:
Cris
05/11/2020, 2:36 AMsephi
05/11/2020, 7:03 AMexecute_pipeline
.
1. We set the DAGSTER_HOME
environment
2. When running dagit
we can see that the logs are saved under the storage
and under history/runs
3. When running the execute_pipeline
from within a python file the logs are not saved (under the environment_dict we have the configurations for the solids
that we are running in the pipeline
.
What is the correct manner to configure exectue_pipeline
?
Update
I guess we need to define the instance .ezechiel syx
05/11/2020, 10:42 AMChris Roth
05/11/2020, 7:09 PMcontext.pdb.set_trace()
to work with docker-compose? i added stdin_open: true
and tty: true
to my docker-compose service but it still doesn't let me interact with the terminalChris Roth
05/11/2020, 7:48 PMcontext.log
in the libraries that i'm importingChris Roth
05/11/2020, 9:17 PMChris Roth
05/11/2020, 9:17 PMuser
05/11/2020, 11:01 PMsashank
05/11/2020, 11:04 PMChris Roth
05/12/2020, 1:33 AMChris Roth
05/12/2020, 2:02 AMAuster Cid
05/12/2020, 8:27 PMChris Roth
05/12/2020, 9:28 PMMessage: An error occurred (404) when calling the HeadObject operation: Not Found
when trying to view raw output, anyone seen this before?sephi
05/13/2020, 8:55 AMsephi
05/13/2020, 11:55 AMdagster instance migrate
I get the following GraphQL errors on the`runs` tab in `dagit`:
Message: Cannot query field "pipelineSnapshotId" on type "Pipeline"
Path:
Locations: [{"line":38,"column":7}]
Message: Cannot query field "rootRunId" on type "PipelineRun". Did you mean "runId"?
Path:
Locations: [{"line":28,"column":3}]
Message: Cannot query field "parentRunId" on type "PipelineRun".
Path:
Locations: [{"line":29,"column":3}]
To solve the current issue - I am going to start from a clean slate and keep the old db for reference if requiredTobias Macey
05/13/2020, 2:39 PMawesome-dagster
resource to list community examples, resource plugins, etc. that other can leverage in their own work?Tobias Macey
05/13/2020, 7:26 PMpranavsingh
05/13/2020, 8:02 PMTim Chan
05/13/2020, 8:50 PMTobias Macey
05/13/2020, 8:56 PMCris
05/13/2020, 10:15 PMimport logging
logger = logging.get_logger(__name__)
class Processor:
def process(self):
logger.log("processing...")
I could see this log in the dagster execution logs when used in a solid
@solid
def main_process():
processor = Processor()
processor.process()
Currently I'd pass the context logger directly to the object, but I was wondering wether there could be a better approach.Nam D. Nguyen
05/14/2020, 3:46 AMsephi
05/14/2020, 8:54 AMdict
type within a solid configuration?
I'm not able to understand how to define a general dict from the error :
Passing Dict
sets an error
DagsterInvalidDefintionError: Attempted to pass <dagster.core.types.python_dict.DagsterDictApi object> to a Field that expects a valid dagster type usable in config (e.g. Dict, Int, String et al).The
Permissive
is too strict - I just need a general dict
type
Tried
Shape({})
> but still problematic
1. A Python primitive type that resolve to dagster config
types: int, float, bool, str.
2. A dagster config type: Int, Float, Bool, String, StringSource, Path, Any,
Array, Noneable, Selector, Shape, Permissive, etc.
3. A bare python dictionary, which is wrapped in Shape. Any
values in the dictionary get resolved by the same rules, recursively.
4. A bare python list of length one which itself is config type.
Becomes Array with list element as an argument.
Cris
05/14/2020, 4:06 PMBen Sully
05/14/2020, 4:36 PMdagster_aws.emr
subpackage. i've started with the types module, but it looks like the dagster_aws.emr.types
module uses both regular python enums and dagster enums - why is that? have i missed something in the docs?user
05/14/2020, 11:15 PMalex
05/14/2020, 11:20 PMCris
05/15/2020, 3:13 AM$ VAR=val dagit ...
), how can I make these available for the scheduled jobs also, it seems they are not starting as the env vars are not set for the cron.