Heya! What would be the preferred way providing fe...
# ask-community
Heya! What would be the preferred way providing feedback / suggestions to dagster.io blog posts? For context, some things are broken or outdated with https://dagster.io/blog/dagster-meltano-integration-tutorial.. published on April 4, 2023 😕
👀 2
Hey Janis! I’m from the Meltano team - can you share what you’re seeing that’s wrong?
There seem to be some errors in the blog post as well as it seems some of the steps are not possible to walk through anymore. I was just about to go through it once. While at it, I can make a summary on what seems to not work. I'll get back to the thread in an hour or so 👍
Apart from some minor things that could be written better, these are critical ones.. When following the instructions to the letter (apart from using your own github token), one will not be able to get past "_Option 1: import jobs from Meltano_" 1. Dagster requires Meltano library installed in its env for it to be able to run
function. I do not know, if this is by design or not, but it is not installed with
. Maybe the blog assumes this module to be available globally and in $PATH? One will be be presented with following error in terminal's stdout
Copy code
..dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster/_core/workspace/context.py:589: UserWarning: Error loading repository location dag_melt_project:FileNotFoundError: [Errno 2] No such file or directory: 'meltano'
No jobs will be visible in To fix this, one would
Copy code
source ~/dag-melt/dagster/dag-venv/bin/activate && pip install meltano && deactivate
then restart
dagster dev
2. One would not be able to run the job that now is available with fix in #1 because of following error
Copy code
dagster._check.CheckError: Value in dict mismatches expected type for key MELTANO_CLI_LOG_CONFIG. Expected value of type <class 'str'>. Got value /Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster_meltano/logging.yaml of type <class 'pathlib.PosixPath'>.
I do not know, if this is a breaking change in Meltano (because the tutorial tells the user to install the latest version or something else).. anyhow, I have not been able to work around this yet. Honestly would not think anyone to continue with a tutorial at this point.
@Fraser Marlow it looks like clarifying the instructions on installing Meltano would be good as well as recommending to pin a specific version. Looks like the post was using 2.16?
Thanks for the specific feedback @JP!
Does not seem to be a breaking change at Meltano side. The blog post hints at usage of Meltano 2.16.1, so I've reinstalled it to be that particular version to no avail... same issue. Looking at the stack trace, it seems to be something in dagster_meltano lib where it is not serialising the pathlib's path object intro string 🤷
👀 1
This fixes the problem..
Copy code
change line 34 from
Copy code
"MELTANO_CLI_LOG_CONFIG": Path(__file__).parent / "logging.yaml",
Copy code
"MELTANO_CLI_LOG_CONFIG": str(Path(__file__).parent / "logging.yaml"),
The meltano job runs without issues.. I now suspect this may be a breaking change in Python 3.11. I'll try to reproduce this with 3.10
From looks of it the default_env property is always used, its type hints say it returns a
for keys and values, however
Path(__file__).parent / "logging.yaml"
does not return a
, but a
object, hence the error.. I've checked this on Python 3.11, 3.10 and 3.9 🤷 From what I can tell from git blame, this property was introduced back in Feb 18, 2023 as part of New dagster_meltano package #15 and therefore I'm clueless as to how this was working at the time of writing the blog post 😞 Would you suggest me to open github issue on this?
Looks like this is a bug in
where it was allowing non-string entries to exist in the dict, which was "fine" until a change in
package in dagster/python_modules/dagster/dagster/_check/__init__.py at master · dagster-io/dagster · GitHub I suspect previously the dict values were not validated, but now they are, hence the error. Specifically this function file:
Copy code
Copy code
It is little too late to go further down the rabbit hole on this. I'll open a PR on
sometime tomorrow on 🤷
Yeah, either a PR or an issue on that repo would be good 👍 Thanks!
Wanted to tag @Jules Huisman (Quantile) as well since he’s the main contributor to that project.
❤️ 1
Thanks @JP for the head’s up. I will run back through the tutorial with your notes i hand and update it. Indeed we made significant changes to the Dagster Config and Resources APIs, introducing Pydantic-like type checking.
OK, sorry for the delay. I got to rerun through the tutorial tonight. I am now on
Copy code
dagster, version 1.2.6
meltano, version 2.16.1
and everything ran smoothly. Thanks @JP for your help on identifying and troubleshooting this one!
👍 2