https://linen.dev logo
Join Slack
Powered by
I did a proof-of-concept of <tap-bigquery> a few m...
# troubleshooting
s
I did a proof-of-concept of tap-bigquery a few months ago and it seemed to work. I’m trying it again now (using more recent versions of everything) and running into an error that is hard to track down. It seems to be coming from something in meltano rather than the tap. Any help appreciated. Full stack trace in 🧵 :
Copy code
not enough values to unpack (expected at least 2, got 1)
The tap starts running. It queries data out of bigquery and then it raises the error. ```INFO Running query: SELECT * FROM 
mse-prod-field-1.customer_backfill.package
WHERE 1=1 AND datetime '2010-01-01 000000.000000' <= CAST(insert_timestamp as datetime) AND CAST(insert_timestamp as datetime) < datetime '2022-09-08 221249.740235' ORDER BY insert_timestamp LIMIT 100 2022-09-08T221251.178240Z [debug ] Deleted configuration at /opt/dagster/app/eta_wh/meltano/.meltano/run/tap-bigquery--field-customer_backfill/tap.17bda005-8afc-4828-bafc-bfdedd9f8c56.config.json 2022-09-08T221251.223269Z [error ] Cannot start plugin tap-bigquery--field-customer_backfill: not enough values to unpack (expected at least 2, got 1) Traceback (most recent call last): File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/block/singer.py", line 346, in start self.process_handle = await self.invoker.invoke_async( File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/plugin_invoker.py", line 446, in invoke_async async with self._invoke(*args, **kwargs) as ( File "/usr/local/lib/python3.8/contextlib.py", line 171, in aenter return await self.gen.__anext__() File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/plugin_invoker.py", line 422, in _invoke async with self.plugin.trigger_hooks("invoke", self, args): File "/usr/local/lib/python3.8/contextlib.py", line 171, in aenter return await self.gen.__anext__() File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/behavior/hookable.py", line 87, in trigger_hooks await self.class.trigger(self, f"before_{hook_name}", *args, **kwargs) File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/behavior/hookable.py", line 115, in trigger raise err File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/behavior/hookable.py", line 107, in trigger await hook_func(target, *args, **kwargs) File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/plugin/singer/tap.py", line 474, in apply_catalog_rules_hook self.apply_catalog_rules(plugin_invoker, exec_args) File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/plugin/singer/tap.py", line 515, in apply_catalog_rules metadata_rules.extend(config_metadata_rules(config["_metadata"])) File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/plugin/singer/tap.py", line 106, in config_metadata_rules tap_stream_id, *props, rule_key = key.split(".") ValueError: not enough values to unpack (expected at least 2, got 1) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/logging/output_logger.py", line 138, in redirect_logging yield File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/block/extract_load.py", line 438, in run await self.run_with_job() File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/block/extract_load.py", line 464, in run_with_job await self.execute() File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/block/extract_load.py", line 427, in execute async with self._start_blocks(): File "/usr/local/lib/python3.8/contextlib.py", line 171, in aenter return await self.gen.__anext__() File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/block/extract_load.py", line 553, in _start_blocks await block.start() File "/root/.local/pipx/venvs/meltano/lib/python3.8/site-packages/meltano/core/block/singer.py", line 353, in start raise RunnerError(f"Cannot start plugin {self.string_id}: {err}") from err meltano.core.runner.RunnerError: Cannot start plugin tap-bigquery--field-customer_backfil…
Oh no! Yeah, if I fall back to meltano==1.105.0, then this tap works. Looks like this tap was not developed with the meltano SDK
Ok, and it looks like the breaking change was introduced in 2.3.0. It works it 2.2.0, fails with the above error in 2.3.0
e
Hi @sterling_paramore! What does the 
metadata
value look like for 
tap-bigquery
in your project?
s
Not quite sure what you mean by 
metadata
value. There’s a 
metadata
field in my 
meltano.yml
file:
Copy code
environments:
- name: main
  config:
    plugins:
      extractors:
      - name: tap-bigquery
        config:
          credentials_path: .secrets/client_secrets.json
          start_datetime: '2010-01-01 00:00:00'
          start_always_inclusive: true
        metadata:
          add_record_metadata: true
Oh, hmmm. If I remove the 
metadata
section, it does work. This seemed to be important when I was doing the POC a few months ago. Why would that feature no longer be supported?
e
Oh, it’s definitely supported but we released some related bug fixes (#6380, #3322) in 
2.3.0
. Though that metadata entry does look funny. It’s supposed to be a mapping:
Copy code
extractors:
- name: tap-postgres
  metadata:
    some_stream_id:
      replication-method: INCREMENTAL
      replication-key: created_at
      created_at:
        is-replication-key: true
unless the tap uses the singer catalog in a non-standard way
(and I can’t find any reference to 
add_record_metadata
in the tap’s codebase)
s
Strange. I wonder why I thought I needed it because it doesn’t seem to do anything in 2.2.0 anyway. Must have slipped into my config while I trying to solve something else.
e
Ideally the fact that 
metadata
isn’t a proper object in your config should be caught earlier (e.g. #2964)
s
That would be great, I knew I was in trouble when I saw the generic error 
not enough values to unpack (expected at least 2, got 1)
Open in Slack
PreviousNext
Powered by