https://linen.dev logo
Join Slack
Powered by
Heyo, I posted <this> in <#CMN8HELB0|> as I didn't...
# troubleshooting
r
Heyo, I posted this in #CMN8HELB0 as I didn't notice this channel exists. When testing extractors I'm running into an issue of 
Plugin configuration is invalid
although I see that the CSV file was loaded as I tested it with a different CSV file to check if it's something to do with the entries in the file. Testing command failed again at:
Copy code
Plugin configuration is invalid
{"type": "RECORD", "stream": "SUPPORT", "record": {"id": "2957642436270096567", "display_name": "b98ee58e372a7398b882a22abcd0b149", "email":
"622474637484ce932692c2e646ba05d2", "department": "Support", "manager_id": "2636555004332212249", "manager_name": "67c835d0879ab870f0ffbc7f355b2b5b",
"manager_email": "6897008ad2a760717537fbd0d8d69c9e", "2nd_level_manager_id": "2790094842804306626", "is_manager": "FALSE"}, "time_extracted":
"2023-10-03T22:32:58.394266+00:00"}

Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
{"type": "RECORD", "stream": "SUPPORT", "record": {"id": "2957642436270096567", "display_name": "b98ee58e372a7398b882a22abcd0b149", "email": "622474637484ce932692c2e646ba05d2", "department": "Support", "manager_id": "2636555004332212249", "manager_name": "67c835d0879ab870f0ffbc7f355b2b5b", "manager_email": "6897008ad2a760717537fbd0d8d69c9e", "2nd_level_manager_id": "2790094842804306626", "is_manager": "FALSE"}, "time_extracted": "2023-10-03T22:32:58.394266+00:00"}
I've also tried with Google Sheets extractor and I'm getting the same. Any ideas on what could be the issue, I've discussed it a bit in the thread in #CMN8HELB0, yml file is there also
r
meltano config <plugin> test
will fail when either no 
RECORD
message is received by Meltano, or some error occurs in the underlying tap invocation process resulting in a non-zero exit code - in the latter case, Meltano grabs the last line of output from the tap process in an attempt to provide the user with some context as to why the test failed (of course, whether this message is meaningful or not is largely dependent on the tap implementation). https://github.com/meltano/meltano/blob/4dcafc24bf0469794999950f5828943b981b80ff/src/meltano/core/plugin_test_service.py#L61-L105 In your case, it looks like the last line of output was a 
RECORD
message, so I would have expected the test to pass as soon as that was encountered. You can try running 
meltano --log-level debug config <plugin> test
, which will print out every line of output as a debug log, or directly invoke the tap itself with 
meltano invoke <plugin>
and see if you get any errors there.
I see you've posted your output of 
meltano --log-level debug tap-csv test
in the other thread - having a look at that now...
I set up a simple example to try and replicate your issue: 
test.csv
(from the 
RECORD
message in your logs)
Copy code
id,summary,creator,creator_email,created,reporter,assignee,statuscategorychangedate,updated,resolutiondate,resolution,priority,status,impacted_series,impacted_clients,kb,components,started_noticed_at,product_affected,escalToTS,escalToEng,TTE_TS,TTE_ENG,TTR,NOMMF,LOMMF
INC-2377,Summarry,MB,mb@mail.mail,2023-09-27 12:45:10,MB,,2023-09-27 12:45:11,2023-09-27 12:48:19,,Some resolution,P3,Ops Investigating,2594770,Clients,,Data Platform,2023-09-27 10:30:00,Widgets,FALSE,FALSE,,,,2,"assignee, ticket_unassigned"
meltano.yml
Copy code
version: 1
default_environment: dev
project_id: 4f0b707f-cacd-4d85-aa55-891398480ba2
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-csv
        config:
          files:
          - entity: test
            path: test.csv
            keys:
            - id
          flattening_enabled: false
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-csv
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-csv.git>
When I run
Copy code
meltano config tap-csv test
I get this error:
Copy code
2023-10-04T00:08:53.670146Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
Catalog discovery failed: command ['/tmp/project/.meltano/extractors/tap-csv/venv/bin/tap-csv', '--config', '/tmp/project/.meltano/run/tap-csv/tap.b5f0116b-69dd-4fd5-8c34-7f57402c8b5b.config.json', '--discover'] returned 1 with stderr:
 2023-10-04 01:08:54,135 | ERROR    | tap-csv              | No CSV file definitions found.
That makes sense to me because the config is defined under the 
dev
environment, so I assume you're running
Copy code
meltano --environment dev config tap-csv test
, in which case everything works fine for me, and I wasn't able to reproduce your issue:
Copy code
2023-10-04T00:12:41.112583Z [info     ] Environment 'dev' is active
Plugin configuration is valid
I can emulate your issue by modifying my local 
meltano
to always set a return code of 
1
for 
meltano config <plugin> test
- then I get the same error as you! 😅 This makes me think that for some reason the tap process is exiting with an error of some kind. I suggest you start by running 
meltano invoke tap-csv
and see what you get.
r
Thanks a lot for such a detailed reply, I'll try it later today
r
That all looks fine to me. I noticed some Windows paths though, so I assume you are running Windows? Maybe there's some difference in the way Windows handles a process termination vs a UNIX-based OS (i.e. resulting in positive non-zero exit code vs a negative one, which would align with what I was able to emulate before)? I don't think many people use Meltano with Windows, so 
meltano config <plugin> test
could very well have been broken this whole time! 😅 As simple and convenient as that command is, 
meltano invoke tap-csv
is probably a safer bet (and more immediately informative) here, for the time being. You can also do
Copy code
meltano run tap-csv target-jsonl
as a test sync from the CSV to a target that outputs a 
.jsonl
file for each defined stream in the 
output/
directory (requires no configuration).
Looking more likely: https://docs.python.org/3/library/subprocess.html#subprocess.Popen.returncode
A negative value 
-N
indicates that the child was terminated by signal 
N
(POSIX only).
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.terminate
On POSIX OSs the method sends SIGTERM to the child. On Windows the Win32 API function TerminateProcess() is called to stop the child.
e
I noticed some Windows paths though, so I assume you are running Windows? Maybe there's some difference in the way Windows handles a process termination vs a UNIX-based OS
Oh that'd be worth logging an issue, at least to document this behavior.
r
https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess So probably what's happening when 
terminate
is called: On UNIX OSs, the 
SIGTERM
signal is sent to the child process, and the negative of the signal number (
15
) is used as the process exit code (
-15
). This registers as 
meltano config <plugin> test
success (is a negative exit code). On Windows, 
TerminateProcess()
is called with the resolved 
SIGTERM
signal number (
15
), and then Python just returns the same value back as the process return code (
15
). This registers as 
meltano config <plugin> test
failure (is not a negative exit code).
r
Oh well, I just span up Ubuntu on WSL and lo and behold, I got this 
Plugin configuration is valid
😅😅😅 I should've asked about this in #C01TCRBBJD7 a lot sooner 😅😅 You're an absolute legend @Reuben (Matatika). Thanks a lot to both, appreciate all the help and hassle. This took a bit but glad we've managed to figure it out and also found what the issue is. Appreciate it a lot guys 💙
2 Views
Open in Slack
PreviousNext
Powered by