JP
05/14/2023, 11:26 PMTaylor Murphy
05/15/2023, 2:33 PMJP
05/15/2023, 4:53 PMload_jobs_from_meltano_project
function. I do not know, if this is by design or not, but it is not installed with dagster_meltano
. Maybe the blog assumes this module to be available globally and in $PATH?
One will be be presented with following error in terminal's stdout
..dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster/_core/workspace/context.py:589: UserWarning: Error loading repository location dag_melt_project:FileNotFoundError: [Errno 2] No such file or directory: 'meltano'
No jobs will be visible in http://127.0.0.1:3000/locations/dag_melt_project/jobs
To fix this, one would
source ~/dag-melt/dagster/dag-venv/bin/activate && pip install meltano && deactivate
then restart dagster dev
2. One would not be able to run the job that now is available with fix in #1 because of following error
dagster._check.CheckError: Value in dict mismatches expected type for key MELTANO_CLI_LOG_CONFIG. Expected value of type <class 'str'>. Got value /Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster_meltano/logging.yaml of type <class 'pathlib.PosixPath'>.
I do not know, if this is a breaking change in Meltano (because the tutorial tells the user to install the latest version or something else).. anyhow, I have not been able to work around this yet.
Honestly would not think anyone to continue with a tutorial at this point.Taylor Murphy
05/15/2023, 5:49 PMJP
05/15/2023, 5:50 PMdag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster_meltano/meltano_resource.py
change line 34 from
"MELTANO_CLI_LOG_CONFIG": Path(__file__).parent / "logging.yaml",
to
"MELTANO_CLI_LOG_CONFIG": str(Path(__file__).parent / "logging.yaml"),
The meltano job runs without issues..
dict
with str
for keys and values, however Path(__file__).parent / "logging.yaml"
does not return a str
, but a PosixPath
object, hence the error.. I've checked this on Python 3.11, 3.10 and 3.9 🤷
From what I can tell from git blame, this property was introduced back in Feb 18, 2023 as part of New dagster_meltano package #15 and therefore I'm clueless as to how this was working at the time of writing the blog post 😞
Would you suggest me to open github issue on this?dagster-meltano
where it was allowing non-string entries to exist in the dict, which was "fine" until a change in dagster
package in dagster/python_modules/dagster/dagster/_check/__init__.py at master · dagster-io/dagster · GitHub
I suspect previously the dict values were not validated, but now they are, hence the error.
Specifically this function
file:
dagster/_check/__init__.py
function:
_check_mapping_entries
It is little too late to go further down the rabbit hole on this. I'll open a PR on dagster-meltano
sometime tomorrow on 🤷Taylor Murphy
05/15/2023, 6:58 PMFraser Marlow
05/15/2023, 8:15 PMJP
05/16/2023, 8:29 AMFraser Marlow
05/18/2023, 12:53 AMdagster, version 1.2.6
meltano, version 2.16.1
and everything ran smoothly. Thanks @JP for your help on identifying and troubleshooting this one!