Hi everyone, I've been bashing my head against this one for a while now and was hoping someone here could help Im trying to do

meltano run tap-jira target-bigquery

but I've been getting this error: Does anyone have an idea what could be causing this ? For more context I used the pipx route to install meltano

Scroll up father there should be more stack trace above you can share

This is at the top of the stack trace, I can send more but it doesnt fit on one screen 😓

version: 1
default_environment: dev
project_id: 327e219d-d3d0-4c26-8a4d-0d0a2b3ac31a
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-jira
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-jira.git>
    config:
      auth:
        username: REDACTED
      domain: REDACTED
      email: REDACTED
      api_token: 
        REDACTED
    select:
    - issues.*
  loaders:
  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
    config:
      bucket: <gs://REDACTED/tmp>
      dataset: meltano_jira
      credentials_path: REDACTED.json
      project: REDACTED
      upsert: 'true'
      column_name_transforms:
        add_underscore_when_invalid: true
        lower: true
      flattening_enabled: true
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

I was able to run it into json files and that worked fine, so Im assuming the issue lies with the loader

Try isolating the loader process to observe the error directly:

meltano invoke tap-jira > tap.out
cat tap.out | meltano invoke target-bigquery

Yeah the other easiest way is to pipe stdout/err to a file and you can attach it or go to .meltano/run and pull up the pipeline logs that are in there for this run (all of them are stored there)

Thanks guys, I tried this and got one step closer to finding a culprit... could this be due to python versioning or something ?

Traceback (most recent call last):
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/bin/target-bigquery", line 8, in <module>
    sys.exit(TargetBigQuery.cli())
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 572, in cli
    target = cls(  # type: ignore[operator]
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/target.py", line 350, in __init__
    super().__init__(*args, **kwargs)
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/target_base.py", line 71, in __init__
    super().__init__(
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 113, in __init__
    self._validate_config(raise_errors=validate_config)
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/target.py", line 587, in _validate_config
    return super()._validate_config(False, False)
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 242, in _validate_config
    errors = [e.message for e in validator.iter_errors(self._config)]
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 242, in <listcomp>
    errors = [e.message for e in validator.iter_errors(self._config)]
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/jsonschema/validators.py", line 384, in iter_errors
    for error in errors:
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/singer_sdk/typing.py", line 149, in set_defaults
    yield from validate_properties(
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/jsonschema/_keywords.py", line 296, in properties
    yield from validator.descend(
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/jsonschema/_keywords.py", line 340, in anyOf
    errs = list(validator.descend(instance, subschema, schema_path=index))
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/jsonschema/validators.py", line 421, in descend
    resolver = self._resolver.in_subresource(
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/referencing/_core.py", line 694, in in_subresource
    id = subresource.id()
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/referencing/_core.py", line 226, in id
    id = self._specification.id_of(self.contents)
  File "/home/data-enablement/meltano/projects/jirapull/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/referencing/jsonschema.py", line 56, in _legacy_dollar_id
    id = contents.get("$id")
AttributeError: 'str' object has no attribute 'get'

That's happening during config validation before the target is even doing anything really. I tried it out and the problem is that the target is expecting

upsert

to be a boolean or array of strings, not a string as you have currently. The Meltano plugin definition declares the setting as a string, which is probably where this issue stems from (assuming you ran

meltano config target-bigquery set upsert true

) and explains the weird unhandled error. You should change to

upsert: true

in your

meltano.yml

. You'll hit another validation error right after requiring you to supply a value for

flattening_max_depth

, but after that it should be ready to attempt loading data.

You're a life saver @Reuben (Matatika)! Thank you so so so much