May I ask two conceptual questions all around sql_solid and delta lake (delta.io)?

Hello Folks, I am the host of the Data Engineering Podcast and Podcast.init, and I am working with O'Reilly on a new book that showcases the work and challenges of data engineers. I am collecting short essays (~400 - 500 words) from industry veterans and experts such as yourself that will be compiled into the final work. This project is a collection of pearls of wisdom collected from leading experts like you -- whether you are/have been a data engineer or you work closely with data engineers in your daily work. The collection will include multiple and varied perspectives on key topics, ideas, skills, and future-focused thoughts that data engineers should be aware of. We’re looking for articles on a wide range of topics: building pipelines, batch and stream processing, data privacy, data security, data governance and lineage, data storage, career advice, data team makeup and culture--and everything in between, from the granular issues to big-picture concepts. What are the challenges? How did you fail? What have you learned? When did the light bulb go off? What do you wish you knew when you started? Submissions are being accepted via the Google form found at boundlessnotions.com/97things, which also contains additional details and instructions. You can feel free to submit multiple articles, but please have them to me within the next 3 - 4 weeks so that we can be sure to hit our publishing deadline. You can find out more about the existing "97 Things Every Programmer Should Know" book here at https://www.safaribooksonline.com/library/view/97-things-every/9780596809515/ If you know anyone else who might be interested in participating then please pass this invitation along to your friends and colleagues as well. Thank you for your time and interest, and we look forward to learning from you!

I'm trying to add dagstermill to my pipeline and having a few problems. The first 2 I was able to workaround, but the last is going to a bug fix in dagstermill.

hey, i'm running

dagster-celery worker start -y celery_config.yaml

with an sqs broker and i get

ValueError: invalid literal for int() with base 10:

. i think the solution is to safequote these strings (eg https://docs.celeryproject.org/en/latest/getting-started/brokers/sqs.html), but i'm not sure if there's a way to do this without forking dagster-celery. any ideas?

see also https://stackoverflow.com/questions/49417851/django-celery-4-valueerror-invalid-literal-for-int-with-base-10-when-start

hello dagster community 👋

hi, quick question. is it possible to use an environment variable in the

celery_config.yaml

that is passed to

dagster-celery

?

prha just published a new version: 0.7.6.

The config editor change the phil just posted is a big advance in that UI (@dwall @ramshackle-jamathon i think you both got sneak peeks of this :-)) This will make dealing with large config documents dramatically easier, as you will quickly be able to see what solid or resource is misconfigured and navigate to it in one click via the pane in the bottom right. Also errors are more prominently featured. Please let us know what you think!

When running dagit with a repository.yaml, is it possible to specify either: 1. Multiple repositories in a repository.yaml file? or 2. Multiple repository.yaml files to dagit?

In case it helps, want to share my biggest challenge from the newguy perspective: all of the dagster Google results for dev questions return "readthdocs" links. Lots of time spent banging on code that was half-deprecated. Now that I'm out of that secret tunnel, I can't sing your praises loud enough. Game changer.

I think there may be some issues with the dagster-bash solids. I am on 0.7.5. When I run a basic pipeline, using the bash_script_solid, I am seeing the solid executed on when loading the repository into dagit, as startup. My solid definition looks something like the following:

@pipleine
def my_pipeline():
    run_cmd_in_venv = bash_script_solid(
        '.../dagster_pipelines/hello_world.sh',
        name = 'hello'
    )

    execute_solid(
        run_cmd_in_venv
    )

the output on starting dagit is as follows:

Loading repository...
2020-04-03 13:29:16 - dagster - DEBUG - ephemeral_hello_solid_pipeline - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - PIPELINE_START - Started execution of pipeline "ephemeral_hello_solid_pipeline".
2020-04-03 13:29:16 - dagster - DEBUG - ephemeral_hello_solid_pipeline - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - ENGINE_EVENT - Executing steps in process (pid: 29701)
 event_specific_data = {"error": null, "marker_end": null, "marker_start": null, "metadata_entries": [["pid", null, ["29701"]], ["step_keys", null, ["['hello.compute']"]]]}
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - DEBUG - ephemeral_hello_solid_pipeline - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - STEP_START - Started execution of step "hello.compute".
               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - using temporary directory: /tmp
               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - Temporary script location: /tmp/hellop5yj64ld
               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - Running command: 
echo "Hello"
               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - Hello

               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - Command exited with return code 0

... you can see that it executes the solid, which is pretty undesirable. Am I doing something wrong?

This is posted to #general and comes from a bot named webhookbot.

hey, quick question. one of my solids calls a command line tool that writes a large json file (gigabytes) to disk (or s3) and then a downstream solid needs to read that file from disk (or s3). not sure if the tool that i'm using can handle stdin / stdout. i'm using celery + s3 for intermediates. what's the right way to pass this json file around? should i just pass the s3 url as the intermediate between solids?

Error with spark on windows

Java gateway process exited before sending its port number

(full stack trace see in first comment). It works fine on my Macbook and linux, but in my a bit more complicated work environment with windows and WSL plus proxy, my local spark connection gives the above error. I tried several things with JAVA options for proxy etc. but before I search more, do you might have an idea where I should start searching or how to fix that? Would be much appreciated.

hey everyone. I'm trying to find some example code on building solids that depend on each other, but don't necessarily pass anything meaningful to the dependent solids

is there any "best practice" way to do this?

was hunting through the examples directory and missed something like this

I'm trying to run scripts in parallel on a cluster through

qsub -sync y

calls within solids, but it is only submitting and running one job at a time. When I omit the

-sync y

part, the pipeline completes immediately once all the jobs are submitted, which doesn't let me see the progress in

dagit

I'm a newbie here, but if someone could help me that would be awesome 😄 How to return some value from notebook solid? I've tried 

dagstermill.yield_result

 didn't help Also why is documentation so wild, you start intro with defining solids with decorator and after a while you are using SolidDefinitions, is there a way for dagstermill solid to be defined with decorator? Sorry if my questions are stupid, really just trying to learn this is basically my structure and I would like my solid transformation_joins.. to be ipynb that can return value / variable from notebook, if that's possible.

Hi all, question: I'm trying to typecheck a float_column using a custom-made (via create_dagster_pandas_dataframe_type) dataframe in dagster_pandas. The problem is there are null values (nans) in some of the values. I'm getting a

Violated InRangeColumnConstraint (Column must have values between -inf and inf inclusive.)

for that column with the nans as offending values, is that normal?

Is it possible to load data into redshift and data is output of an notebook solid? Thanks!

Hi guys, we're building an ETL Framework to work on top of spark. The framework will have specific logging, custom components like lookups, loading from specific databases, etc, all as a layer above Spark. We currently have a simple pipeline component that was built in-house. I came across Dagster and from what i've read and tried in the tutorials i believe it could be a very good candidate for replacing our pipeline component. Would love to chat with anyone who have worked with Dagster on similar implementations

Hello, another question: are the stack traces truncated or something when they are raised by Dagster? I have an error raised during execution of a pipeline (a call to pandas.DataFrame.to_sql) and I see only that it is a programming error, but the actual error I get from postgres is not there. I only have what I believe is up to the top layer of the stack, but not the actual exception data. (@Frank Dekervel)

This is probably a dumb question. Is it possible (or advisable) to make full use of dagster without setting up configs in yaml files? I'm not sure if I understand the benefit entirely, and it feels like it adds extra layers of complication. I guess I also don't like yaml's syntax, so that's probably some bias on my part. Thanks in advance for helping with my ignorance. 🙂

Hi, is it possible to create a solid that takes an arbitrary number of inputs? Something like this:

@solid(...)
def my_solid(context, **kwargs):
    #some stuff

and then:

@pipeline
def my_pipeline():
    task1 = my_solid.alias('task1')
    task2 = my_solid.alias('task2')
    task3 = my_solid.alias('task3')
    task4 = my_solid.alias('task4')
    task4(task1(), task2(), task3())

ahoy! i'm on 0.7.6 and noticed that the docs show

compute_log_manager

as the

dagster.yaml

property but it only worked when i changed it to

compute_logs

- maybe docs need to be updated?

also, how do i get the raw log output to work? i'm currently using sqs/celery and seeing nothing when i view raw logs