Hi there I m completely new to Meltano I managed to set up a Meltano #troubleshooting

Hi there, I'm completely new to Meltano. I managed...

roel_peters

01/30/2022, 7:45 PM

Hi there, I'm completely new to Meltano. I managed to set up an EL pipeline, but my transformations don't seem to work. This is my debug log. I've tried googling and searching the community but nothing popped up. Help is very much appreciated! Main error below, full log in comments.

Copy code

2022-01-30T18:59:20.601399Z [info     ]   at path []: Additional properties are not allowed ('seed-paths', 'packages-install-path', 'model-paths' were unexpected) cmd_type=transformer job_id=test_pipeline name=dbt run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 stdio=stderr

roel_peters

01/30/2022, 7:46 PM

Part 1: ```022-01-30T185916.914393Z [info ] Running transformation... job_id=test_pipeline name=meltano run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 2022-01-30T185916.925389Z [debug ] Variable '$MELTANO_TRANSFORM__PACKAGE_NAME' is missing from the environment. 2022-01-30T185916.931386Z [debug ] Variable '$MELTANO_TRANSFORM__PACKAGE_NAME' is missing from the environment. 2022-01-30T185916.937390Z [debug ] Variable '$MELTANO_TRANSFORM__PACKAGE_NAME' is missing from the environment. 2022-01-30T185916.941394Z [debug ] Invoking: ['C:\\Users\\roelp\\Documents\\Projects\\meltano_run\\sample_project\\.meltano\\transformers\\dbt\\venv\\Scripts\\dbt', 'clean'] 2022-01-30T185916.952389Z [debug ] Env: {'ALLUSERSPROFILE': 'C:\\ProgramData', 'APPDATA': 'C:\\Users\\roelp\\AppData\\Roaming', 'CHOCOLATEYINSTALL': 'C:\\ProgramData\\chocolatey', 'CHOCOLATEYLASTPATHUPDATE': '132534034809883263', 'CHROME_CRASHPAD_PIPE_NAME': '\\\\.\\pipe\\crashpad_13396_GZEHJFJROCKPUIHF', 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', 'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files', 'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files', 'COMPUTERNAME': 'LAPTOP-7NO76L8N', 'COMSPEC': 'C:\\WINDOWS\\system32\\cmd.exe', 'DRIVERDATA': 'C:\\Windows\\System32\\Drivers\\DriverData', 'FPS_BROWSER_APP_PROFILE_STRING': 'Internet Explorer', 'FPS_BROWSER_USER_PROFILE_STRING': 'Default', 'HOMEDRIVE': 'C:', 'HOMEPATH': '\\Users\\roelp', 'LOCALAPPDATA': 'C:\\Users\\roelp\\AppData\\Local', 'LOGONSERVER': '\\\\LAPTOP-7NO76L8N', 'NUMBER_OF_PROCESSORS': '8', 'ONEDRIVE': 'C:\\Users\\roelp\\OneDrive - P-TMC p-tmc.com', 'ONEDRIVECOMMERCIAL': 'C:\\Users\\roelp\\OneDrive - P-TMC p-tmc.com', 'ONEDRIVECONSUMER': 'C:\\Users\\roelp\\OneDrive', 'ORIGINAL_XDG_CURRENT_DESKTOP': 'undefined', 'OS': 'Windows_NT', 'PATH': 'C\\Users\\roelp\\Documents\\Projects\\meltano run\\sample project\\.meltano\\transformers\\dbt\\venv\\Scripts;C\\Users\\roelp\\Documents\\Projects\\meltano_run\\.venv\\Scripts;C\\WINDOWS\\system32;C\\WINDOWS;C\\WINDOWS\\System32\\Wbem;C\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C\\WINDOWS\\System32\\OpenSSH\\;C\\Program Files\\PuTTY\\;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Program Files\\Git\\cmd;C:\\Program Files\\nodejs\\;C\\ProgramData\\chocolatey\\bin;C\\Program Files\\Google\\Chrome\\Application;C:\\Program Files\\Amazon\\AWSCLIV2\\;C:\\Program Files\\Docker\\Docker\\resources\\bin;C\\ProgramData\\DockerDesktop\\version bin;C\\Users\\roelp\\Documents\\Projects\\draft\\wget.exe;C\\Users\\roelp\\AppData\\Local\\Microsoft\\WindowsApps;C\\Users\\roelp\\AppData\\Local\\Google\\Cloud SDK\\google-cloud-sdk\\bin;C\\Users\\roelp\\Desktop\\curl\\bin;C\\Users\\roelp\\AppData\\Roaming\\npm;C:\\Program Files\\heroku\\bin;C:\\Program Files\\Google\\Chrome\\Application;C:\\Program Files\\7-Zip;', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL', 'PROCESSOR_ARCHITECTURE': 'AMD64', 'PROCESSOR_IDENTIFIER': 'AMD64 Family 23 Model 24 Stepping 1, AuthenticAMD', 'PROCESSOR_LEVEL': '23', 'PROCESSOR_REVISION': '1801', 'PROGRAMDATA': 'C:\\ProgramData', 'PROGRAMFILES': 'C:\\Program Files', 'PROGRAMFILES(X86)': 'C:\\Program Files (x86)', 'PROGRAMW6432': 'C:\\Program Files', 'PSMODULEPATH': 'C\\Users\\roelp\\Documents\\WindowsPowerShell\\Modules;C\\Users\\roelp\\AppData\\Local\\Google\\Cloud SDK\\google-cloud-sdk\\platform\\PowerShell;C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules', 'PUBLIC': 'C:\\Users\\Public', 'SESSIONNAME': 'Console', 'SYSTEMDRIVE': 'C:', 'SYSTEMROOT': 'C:\\WINDOWS', 'TEMP': 'C:\\Users\\roelp\\AppData\\Local\\Temp', 'TMP': 'C:\\Users\\roelp\\AppData\\Local\\Temp', 'USERDOMAIN': 'LAPTOP-7NO76L8N', 'USERDOMAIN_ROAMINGPROFILE': 'LAPTOP-7NO76L8N', 'USERNAME': 'roelp', 'USERPROFILE': 'C:\\Users\\roelp', 'WINDIR': 'C:\\WINDOWS', 'TERM_PROGRAM': 'vscode', 'TERM_PROGRAM_VERSION': '1.6…

roel_peters

01/30/2022, 7:47 PM

Part 2:

Copy code

2022-01-30T18:59:20.591398Z [info     ] Encountered an error while reading the project: cmd_type=transformer job_id=test_pipeline name=dbt run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 stdio=stderr
2022-01-30T18:59:20.596397Z [info     ]   ERROR: Runtime Error         cmd_type=transformer job_id=test_pipeline name=dbt run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 stdio=stderr
2022-01-30T18:59:20.601399Z [info     ]   at path []: Additional properties are not allowed ('seed-paths', 'packages-install-path', 'model-paths' were unexpected) cmd_type=transformer job_id=test_pipeline name=dbt run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 stdio=stderr
2022-01-30T18:59:20.606398Z [info     ]                                cmd_type=transformer job_id=test_pipeline name=dbt run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 stdio=stderr
2022-01-30T18:59:20.611399Z [info     ] Error encountered in C:\Users\roelp\Documents\Projects\meltano_run\sample_project\transform\dbt_project.yml cmd_type=transformer job_id=test_pipeline name=dbt run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 stdio=stderr
2022-01-30T18:59:20.615400Z [info     ] Encountered an error:          cmd_type=transformer job_id=test_pipeline name=dbt run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 stdio=stderr
2022-01-30T18:59:20.619399Z [info     ] Runtime Error                  cmd_type=transformer job_id=test_pipeline name=dbt run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 stdio=stderr
2022-01-30T18:59:20.624405Z [info     ]   Could not run dbt            cmd_type=transformer job_id=test_pipeline name=dbt run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058 stdio=stderr
2022-01-30T18:59:20.835899Z [error    ] Transformation failed          code=2 job_id=test_pipeline message=  Could not run dbt name=meltano run_id=1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058
2022-01-30T18:59:20.863896Z [debug    ] ELT could not be completed: `dbt clean` failed
For more detailed log messages, check the generated log file 'C:\Users\roelp\Documents\Projects\meltano_run\sample_project\.meltano\logs\elt\test_pipeline\1d3a9cc0-ba21-45b6-a0d2-1ea4a40bd058\elt.log' or re-run the command using the '--log-level=debug' CLI flag. 
Traceback (most recent call last):
  File "c:\users\roelp\documents\projects\meltano_run\.venv\lib\site-packages\meltano\cli\elt.py", line 248, in _run_elt
    await _run_transform(log, elt_context, output_logger)
  File "c:\users\roelp\documents\projects\meltano_run\.venv\lib\site-packages\meltano\cli\elt.py", line 346, in _run_transform
    await dbt_runner.run(**kwargs, log=transformer_log_writer)
  File "c:\users\roelp\documents\projects\meltano_run\.venv\lib\site-packages\meltano\core\runner\dbt.py", line 64, in run
    await self.invoke(dbt, log=log, command="clean")
  File "c:\users\roelp\documents\projects\meltano_run\.venv\lib\site-packages\meltano\core\runner\dbt.py", line 56, in invoke
    raise RunnerError(
meltano.core.runner.RunnerError: `dbt clean` failed

roel_peters

01/30/2022, 8:29 PM

My dbt version is 0.21.1, but my dbt_project config says require-dbt-version: [">=1.0.0", "<2.0.0"].

roel_peters

01/30/2022, 8:30 PM

How do I upgrade my dbt version within Meltano? Just using pip doesn't seem to do anything.

roel_peters

01/30/2022, 8:43 PM

Updated meltano.yml, used meltano install, meltano upgrade, and now it's error hell

roel_peters

01/30/2022, 8:46 PM

Downgraded back to 0.21.1, and removed seed-pathts, packages-install-paths and model-paths from the dbt_project.yml

roel_peters

01/30/2022, 8:56 PM

Well wadayaknow, it works.

visch

01/31/2022, 3:07 AM

It working on Windows? Going to blow my mind if

meltano elt

works on Windows 😄

roel_peters

01/31/2022, 7:53 AM

Well... it does.

roel_peters

01/31/2022, 7:54 AM

meltano run works too

aaron_phethean

01/31/2022, 10:59 AM

@roel_peters great work getting it going. The meltano engineers are currently upgrading dbt to 1.0.0 - which is great if you actually wanted to get dbt 1.0.0 working. Longer story - meltano automatically brings a dbt project into your meltano project with a plugin dependency called files-dbt. Essentially a bundle of files that get you started with dbt for a specific version. It's this default getting started dbt project that is being upgraded to dbt 1.0.x Once catch, my experience end of last week was that this broke the files-dbt dependency on the current and older meltano version, latest will temporarily be broken until the next meltano release. If you are using an older meltano version, creating a new meltano project will always be broken as it is going to create a dbt project for you with the wrong configuration options. One to watch, not sure this is a problem per se. We did end up forking that files-dbt and creating a release for 0.20.x, and changing the default profile to use the target environments instead of internal meltano only environments - https://github.com/Matatika/files-dbt/releases/tag/v0.20.x.0

roel_peters

01/31/2022, 1:47 PM

Seems like a great moment to get started with Meltano. 🥲 /s

aaron_phethean

01/31/2022, 1:53 PM

It'll be ok! 🙂

visch

01/31/2022, 2:02 PM

note that Meltano is not supported on Native windows @roel_peters , WSL 2 it is https://docs.meltano.com/guide/installation#:~:text=from%20your%20browser.-,Requirements,-Before%20you%20install

aaron_phethean

01/31/2022, 4:10 PM

Surely no one runs windows in production! Do they? (I know I'm showing my age, when Windows wasn't as stable as it is now...)

visch

01/31/2022, 4:11 PM

Generally windows is used all over in production (outside of Meltano/singer) not sure if that's what you mean. I also have Meltano running in prod with Windows

visch

02/02/2022, 10:28 PM

@roel_peters , just verified

meltano elt

does not work on Windows on the latest Meltano version. Just an fyi for you if you try to run elt

Open in Slack

Previous Next