https://linen.dev logo
Join Slack
Powered by
Hello, I'm using `tap-s3-csv` variant `fishtown an...
# troubleshooting
b
Hello, I'm using 
tap-s3-csv
variant 
fishtown analytics
The variant shows an interesting configuration of modified the headers that's whats I need so thats why I'm trying to use it. When I run 
elt
or 
invoke
I'm getting the below error.
Copy code
Catalog discovery failed: command ['/meltano-project/.meltano/extractors/tap-s3-csv-fa/venv/bin/tap-s3-csv', '--config', '/meltano-project/.meltano/run/tap-s3-csv-fa/tap.fe4d1544-bc99-42fb-b46f-25326b7034b1.config.json', '--discover'] returned 2
https://github.com/dbt-labs/tap-s3-csv
Apparently the tap doesn't recognized discover, so I try to remove it from the capabilities but incurs in other error.
```usage: tap-s3-csv [-h] -c CONFIG [-s STATE] tap-s3-csv: error: unrecognized arguments: --discover [2021-10-08 153136,443] [74|MainThread|root] [DEBUG] Deleted configuration at /meltano-project/.meltano/run/tap-s3-csv-fa/tap.fe4d1544-bc99-42fb-b46f-25326b7034b1.config.json [2021-10-08 153136,444] [74|MainThread|meltano.cli.utils] [DEBUG] Catalog discovery failed: command ['/meltano-project/.meltano/extractors/tap-s3-csv-fa/venv/bin/tap-s3-csv', '--config', '/meltano-project/.meltano/run/tap-s3-csv-fa/tap.fe4d1544-bc99-42fb-b46f-25326b7034b1.config.json', '--discover'] returned 2 Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/meltano/cli/__init__.py", line 44, in main cli(obj={"project": None}) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in call return self.main(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/cli/params.py", line 23, in decorate return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/cli/params.py", line 56, in decorate func(project, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/cli/invoke.py", line 74, in invoke _invoke(invoker, project, plugin_name, plugin_args, session, dump, command_name) File "/usr/local/lib/python3.6/site-packages/meltano/core/utils/__init__.py", line 47, in run_async loop.run_until_complete(future) File "/usr/local/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete return future.result() File "/usr/local/lib/python3.6/site-packages/meltano/cli/invoke.py", line 88, in _invoke handle = await invoker.invoke_async(*plugin_args, command=command_name) File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 275, in invoke_async async with self._invoke(*args, **kwargs) as ( File "/usr/local/lib/python3.6/site-packages/async_generator/_util.py", line 34, in aenter return await self._agen.asend(None) File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 260, in _invoke async with self.plugin.trigger_hooks("invoke", self, args): File "/usr/local/lib/python3.6/site-packages/async_generator/_util.py", line 34, in aenter return await self._agen.asend(None) File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 87, in trigger_hooks await self.class.trigger(self, f"before_{hook_name}", *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 115, in trigger raise err File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 107, in trigger await hook_func(target, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/tap.py", line 251, in discover_catalog_hook await self.discover_catalog(plugin_invoker) File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/tap.py", line 297, in discover_catalog await self.run_discovery(plugin_invoker, catalog_path) File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/tap.py", line 358, in run_discovery f"Catalog discovery failed: command {plugin_invoker.exec_args('--discover')} returned {exit_code}" meltano.core.plugin.error.PluginExecutionError: Catalog discovery failed: command ['/meltano-project/.meltano/extractors/tap-s3-csv-fa/venv/bin/tap-s3-csv', '--config', '/me…
Complete 
-log-level=debug
shows:
t
Which version of Meltano are you on?
b
@taylor I'm on 1.83. I will update to the .84 and try.
t
I don’t think that would solve it. Just trying to cover all the bases
Do you want to try another variant of tap-csv? https://hub.meltano.com/taps/csv
b
I checked it, but I didn't try because all my .csv files are inside S3, I saw that this tap is for local, am I right? I used the default variant of 
tap-s3-csv
, with my team did some change to the tap, because was not syncing and then worked. But I want to change the header name because not all my files have headers. I changed the type with 
schema
extra of meltano, I tried to change the name attribute but is stays in stand by. So I'm trying this variant 
fishtown analytics
, in the description mentioned that it could change via the configuration.
t
the docs are confusing. The gotchas say “input files MUST have a header”, but then the config says 
// if the files don't have a header row, you can specify the field names
I’m not sure how it’s going to distinguish between files that do and don’t have a header
e
@boggdan_barrientos so you removed 
discover
from the tap's capabilities but did you also remove 
catalog
?
b
@edgar_ramirez_mondragon that's right. In a try I removed first discover but I check that it depends of catalog. So in the next try I removed both. The log-level=debug posted it the thread shows it with all capabilities.
e
And you're still getting errors when you remove both capabilities?
b
Yes. 🥲 This is the 
--log-level=catalag
``` DEBUG ELT could not be completed: Cannot start extractor: Applying catalog rules failed: catalog file is missing. Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/tap.py", line 412, in apply_catalog_rules with catalog_path.open() as catalog_file: File "/usr/local/lib/python3.6/pathlib.py", line 1183, in open opener=self._opener) File "/usr/local/lib/python3.6/pathlib.py", line 1037, in _opener return self._accessor.open(self, flags, mode) File "/usr/local/lib/python3.6/pathlib.py", line 387, in wrapped return strfunc(str(pathobj), *args) FileNotFoundError: [Errno 2] No such file or directory: '/meltano-project/.meltano/run/elt/2021-10-08T170102--tap-s3-csv-fa--target-redshift/e9a36cdb-8f08-41dc-84ef-f53d9176a4bf/tap.properties.json' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 67, in invoke stderr=asyncio.subprocess.PIPE, # Log File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 275, in invoke_async async with self._invoke(*args, **kwargs) as ( File "/usr/local/lib/python3.6/site-packages/async_generator/_util.py", line 34, in aenter return await self._agen.asend(None) File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 260, in _invoke async with self.plugin.trigger_hooks("invoke", self, args): File "/usr/local/lib/python3.6/site-packages/async_generator/_util.py", line 34, in aenter return await self._agen.asend(None) File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 87, in trigger_hooks await self.class.trigger(self, f"before_{hook_name}", *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 115, in trigger raise err File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 107, in trigger await hook_func(target, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/tap.py", line 371, in apply_catalog_rules_hook self.apply_catalog_rules(plugin_invoker, exec_args) File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/tap.py", line 435, in apply_catalog_rules ) from err meltano.core.plugin.error.PluginExecutionError: Applying catalog rules failed: catalog file is missing. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 239, in _run_elt await _run_extract_load(elt_context, output_logger) File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 282, in _run_extract_load loader_out=loader_out_writer, File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 230, in run loader_out=loader_out, File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 70, in invoke raise RunnerError(f"Cannot start extractor: {err}") from err meltano.core.runner.RunnerError: Cannot start extractor: Applying catalog rules failed: catalog file is missing. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 227, in _redirect_output yield File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 248, in _run_elt raise CliError(f"ELT could not be completed: {err}") from err meltano.cli.utils.CliError: ELT could not be completed: Cannot start extractor: Applying catalog rules failed: catalog file is missing. meltano | ELT could not be completed: Cannot start extractor: Applying catalog rules failed: …
e
Ah, you need to remove any 
metadata
and 
schema
overrides you may have added for that tap
And 
select
, 
select_filter
too
b
@edgar_ramirez_mondragon Thanks! I remove properties from the capabilities too and that's work. The tap allows insert the header but only read csv with 
','
So I'm modifying the tap itself to support other delimiters.
I made the changes to the tap, now is working as I want, the tap admits the delimiter. What is the process to put it as a variant?
e
I think you need to open an MR in the Hub, right @taylor?
t
@boggdan_barrientos you can make an MR to this file to get it added! https://gitlab.com/meltano/hub/-/blob/main/_data/taps/csv.yml
Open in Slack
PreviousNext
Powered by