Hello! I am quite new to Dagster and am experimenting with it lately. Perhaps I am missing something, I am trying to find out if Dagster support resource management natively to limit parallelism during executions (i.e. similar to https://luigi.readthedocs.io/en/stable/api/luigi.task.html?highlight=resources#luigi.task.Task.resources), but haven't found anything particularly interesting on the documentation site (other than https://docs.dagster.io/docs/deploying/dask#managing-compute-resources-with-dask which forces me to use dask, which I am avoiding for now) Would any kind soul point me to the right direction/share some dagster recipe for achieving resources management please, thanks a bunch 🙏

hi… is there any option to run dagit UI with custom base_url instead of simple

/

… i want to run the dagit like:

<http://localhost:3000/engine>

Is there any guidelines for when something should be a solid config vs solid input?

Curious how folks are approaching data quality checks and relatedly (hopefully) generating/simulating test data.

it'd be interesting to have hypothesis annotations for inputs and outputs on solids

and perhaps have some initial population of those based on observed data flowing through a pipeline

Hi, I want to execute some code when there are any failures. Specifically, I'm trying to log failure information to a table. What would be the best approach? Currently this is how I'm envisioning it.

Hi, When running a composite_solid we need to output 2 objects. if we run with

@composite_solid(

output_defs=[OutputDefinition(name='path', dagster_type=String),

OutputDefinition(name='dataframe', dagster_type=DataFrame),

])

def my_solid():

...

path = solid_return_path()

dataframe = solid_return_dataframe()

yield Output (path, 'path')

yield Output (dataframe, 'dataframe')

we get the following error:

dagster.core.errors.DagsterInvalidDefinitionError: @composite_solid my_solid returned problematic value of type <class 'generator'>. Expected return value from invoked solid or dict mapping output name to return values from invoked solids.

If we change it to :

return path, dataframe

the pipeline runs. Any suggestions?

I've been getting a bunch of these errors. does anyone know what might be causing them?

hm, it looks to me like you might be running out of a system resource like file descriptors... @prha does this ring a bell

I haven't found it in the docs yet but if it's there lmk and I'll keep looking instead of asking here, but is it possible to run pipelines inside virtualenv (kind of like airflow virtualenv operator)? Or a solution to support different envs in se deployment using venv or something else

@max still getting resource unavailable

compute_logs:
  module: dagster.core.storage.local_compute_log_manager
  class: NoOpComputeLogManager
  config:
    base_dir: /tmp/datafarm

prha just published a new version: 0.7.16.

running

0.7.16

what did i do wrong?

new errors

why is it running cron code?

Hi! Sorry for dumb question 😅 Is it possible to get a config structure (dict or yaml) of existing solid? I guess it is, cuz dagit autocompletes and checks config, but I could not find how it happens.

hi getting this error after update to 0.7.16

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column "snapshot_id" of relation "runs" does not exist
LINE 1: INSERT INTO runs (run_id, snapshot_id, pipeline_name, status...

🙋 Roll call! Who else is here?

Guys, I could not find it in the docs, but may be you can point me out to the possibility. I am looking for a way to store execution status on S3 without AWS RDS. Is there such possiblity?

Is there a way to configure

execute_pipeline_iterator

to send logs to the event stream?

What kind of advance we get when running this line in our project.