looking at the

tap.properties.json

, I see that point is included:

"point": {},

but without any type defined. The database type is a PostGIS geometry so it's not surprising there is no support for it.

so postgres generates a complete schema always, even though you exclude some columns in the actual data. fine. so I try to amend the schema in

tap-postgres

with

schema:
          '*':
            point:
              type: 'null'
              default: null

(also tried string and such). but the result is the same. Looking in the

run

dir, the metadata for the tap matches what I've configured. But the log output when running etl shows that an empty schema is transmitted:

2022-01-13T12:49:57.192021Z [info     ] INFO public-e_address schema: { [...] 'point': {} [...]

right. so this appears to be https://gitlab.com/meltano/meltano/-/issues/2469

and there is an unmerged fix (since oct!) here: https://github.com/transferwise/pipelinewise-tap-postgres/pull/129 using the that fork fixes the schema, and now it works.

I'm evaluating Meltano for my company and finding it rather frustrating that these aren't working together out of the box, even after deselecting the problematic columns

Will post back shortly once that is updated.

Could you let us know (either way) if you have any luck with this approach? As mentioned, this is a brand new feature and we're still gathering feedback from real-world applications.

plugins:
  extractors:
  - name: tap-jira
    variant: singer-io
    pip_url: git+<https://github.com/singer-io/tap-jira.git>
    config:
      base_url: *******
      start_date: '2022-03-01'
      username: *******
    select:
    - issues.*
    - '!issues.renderedFields'
    - '!issues.versionedRepresentations'
  loaders:
  - name: target-bigquery
    variant: adswerve
    pip_url: git+<https://github.com/adswerve/target-bigquery.git@0.11.3>
    config:
      credentials_path: *******
      dataset_id: *******
      project_id: *******
  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    mappings:
    - name: remove-bad-cols
      config:
        stream_maps:
          issues:
            renderedFields: null
            versionedRepresentations: null

I've tested this basic config to remove two problematic fields (

issues.renderedFields

and

issues.versionedRepresentations

) but I still encounter the same error as before when running

meltano run tap-jira remove-bad-cols target-bigquery

Perhaps I'm missing something obvious?

the final exception is same as before using the mapper:

google.api_core.exceptions.BadRequest: 400 POST <https://bigquery.googleapis.com/upload/bigquery/v2/projects/***/jobs?uploadType=resumable>: Field renderedFields is type RECORD but has no schema

Oddly enough I do see

target-bigquery

complaining early on about the

issues

schema

2022-03-23T22:16:27.021628Z [info     ] WARNING the pipeline might fail because of undefined fields: an empty object/dictionary indicated as {} cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery

and above that I see

target-bigquery

still printing out the schema with the two columns I've attempted to remove (removed other parts since the log message is long)

2022-03-23T22:16:27.021085Z [info     ] INFO issues schema: {[...] 'renderedFields': {'type': ['null', 'object'], 'patternProperties': {'.+': {}}},'versionedRepresentations': {'type': ['null', 'object'], 'patternProperties': {'.+': {'type': ['null', 'object'], 'patternProperties': {'.+': {}}}}},} cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery

@jose_riego_valenzuela what's the best way to install your version of target bigquery? I've changed my meltano.yml with this code

- name: target-bigquery

variant: transferwise

pip_url: git+<https://github.com/jmriego/pipelinewise-target-bigquery.git>

and then did a

meltano install loader target-bigquery