marcos
10/04/2021, 2:05 AMlog_output()
waits until all 25 numbers have been returned before executing. I had hoped that function would start right away after the first number was returned. Is that type of functionality possible?Rubén Lopez Lozoya
10/04/2021, 10:14 AMdagster-postgres
library dependency from our code since it was not being imported anywhere and we had no issues developing locally with and without Docker. However, once we deployed Dagster to our cluster using the provided Helm chart, our pipelines would get stuck in STARTING
because our deployment missed the mentioned dependency. Is there any way to have this dependency included somehow in the dagster core package or be automatically handled by the Helm chart itself somehow? It's really confusing having to add a library that is not imported anywhere 😞Arun Kumar
10/04/2021, 6:06 PMAnaqi Afendi
10/04/2021, 8:13 PMAndy H
10/04/2021, 10:18 PMdagster_aws.s3.sensor
?Sandeep Mankikar
10/04/2021, 10:42 PMSimon Späti
10/05/2021, 8:40 AMinput_solid
that handles that and distribute it to other solids. Downside: This is very messy and you do not get the actual data flow anymore.
I read the message from @sandy, would you suggest to create a RootInputManager for that use case which would contain mf_handler_version
, mf_converter_version
and input_file
, that could then be used as an input for all the solids needed. expectation_config_file
and data
can be directly stated as an input to the specific solid. Do you agree?
Or do you see other solutions? @alex I saw other comments from you where you suggested resource, but that be a bit of an overkill for our use case, right? Appreciate any help or hint a lot!Cody Hutchens
10/05/2021, 3:25 PMgeoHeil
10/05/2021, 4:04 PM<http://context.log.info|context.log.info>(os.path.join(context.run_id, context.step_key, context.name))
nor yield EventMetadataEntry.string(self._get_path(context), label="xxxxx")
seem to show up in the dagster logs.Gillian Sharer
10/05/2021, 4:42 PMgeoHeil
10/05/2021, 5:34 PMChris Evans
10/05/2021, 8:55 PM0.12.11
. Firstly, Dagit seems to detect errors around missing config for ops that are deselected. Secondly, even if a subset of a graph is selected in the Dagit playground, all ops will end up running when execution is launched.
from dagster import graph, op
@op(config_schema={"param": str})
def hello(context):
...
@op(config_schema={"param": str})
def hello2(context):
...
@graph
def test_graph():
hello()
hello2()
test_job = test_graph.to_job()
Dalin Kim
10/06/2021, 3:25 AMMatthew Smicker
10/06/2021, 7:26 PMdc = get_dc() # this solid returns an object that has a bucket attribute
export_operation(bucket=dc.bucket)
I could make a basic solid that takes an object of the specific type of dc and returns the bucket attribute as a string but was hoping to avoid that. Appreciate any advice on patterns to follow - or if this approach is a bad one, appreciate that feedback as well 🙂 i.e. should I stick to basic types and yield all the objects attributes of potential interest (there are ~15)?Tyler Ellison
10/06/2021, 8:31 PMAmardeep Singh
10/07/2021, 5:45 AMStepan Dvoiak
10/07/2021, 1:35 PMdagit
with https on an external server? I do have ssl certs but cant find an option for dagit cli
The main problem is that Safari browser is forcing https connections and I cant disable it. The server has ssl certs for all sites that run on itRubén Lopez Lozoya
10/07/2021, 2:56 PMMartim Passos
10/07/2021, 4:39 PMENTRYPOINT ["dagster", "pipeline", "execute", "-f", "pipelines/IIIF_pipeline.py", "--preset", "debug"]
but runs successfully with no ENTRYPOINT
and
docker run my-container dagster pipeline execute -f pipelines/IIIF_pipeline.py --preset debug
Marc Keeling
10/07/2021, 5:54 PMAdam McCartney
10/08/2021, 9:07 AMThomas
10/08/2021, 12:17 PMNoah Sanor
10/08/2021, 2:19 PMNoah Sanor
10/08/2021, 2:25 PMsolid_selection
argument to only run some of the solids in the pipeline. When asserting validate_run_config
we are getting failures because the test does not know that we don't need the config values for solids not being run. Is there a solution or workaround for this besides adding "dummy" config values for the solids we aren't running?Chris Chan
10/08/2021, 7:20 PMGayathri Chakravarthy
10/08/2021, 9:29 PMKoby Kilimnik
10/10/2021, 10:43 AMKoby Kilimnik
10/10/2021, 10:45 AMKoby Kilimnik
10/10/2021, 11:51 AMKoby Kilimnik
10/10/2021, 11:51 AM