https://linen.dev logo
Join Slack
Powered by
I'm pretty confused by `tap-postgres`. Am I not su...
# troubleshooting
t
I'm pretty confused by 
tap-postgres
. Am I not supposed to provide a 
streams
field like the other taps? This yml is just selecting everything
Copy code
plugins:
  extractors:
    - name: nipap
      inherit_from: tap-postgres
      variant: meltanolabs
      pip_url: meltanolabs-tap-postgres
      config:
        log_level: debug
        host: <http://pg-01.utbb.net|pg-01.utbb.net>
        port: 5432
        database: nipap
        user: meltano
        password: 9oiv?|,VU
        filter_schemas: 
          - public
        streams:
          - name: public.ip_net_plan
            primary_keys:
              - id
            replication_keys: last_modified
            schema:
              properties:
                id:
                  type: [integer]
                prefix:
                  type: [string]
e
a 
streams
field like the other taps
I don't think other taps accept that setting, at least none that I can think of 😅 You probably want https://docs.meltano.com/concepts/plugins/#select-extra instead
t
Here's an extractor I wrote using the rest tap. It took a stream argument
Copy code
plugins:
  extractors:
  - name: uisp
    inherit_from: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: <https://uisp.utbb.net/nms/api/v2.1>
      auth_method: api_key
      api_keys:
        X-Auth-Token: ${UISP_TOKEN}
      streams:
        - name: onus
          path: /devices
          params:
            type: onu
          primary_keys:
          - identification_id
          schema:
            properties:
              identification_id:
                type: string
              identification_name:
                type: string
              attributes_parentId:
                type: string
              identification_mac:
                type: string
              overview_createdAt:
                type: string
                format: date-time
              identification_updated:
                type: string
                format: date-time
    pagination_request_style: single_page_paginator
    pagination_response_style: single_page_paginator
    load_schema: raw_uisp
👀 1
How do I set the replication key with the select extra approach? I see on that link there's a metadata field I can set the replication key in. But that requires a stream ID. How do I know my stream ID if I can't define a stream like I do with the rest tap?
e
How do I know my stream ID if I can't define a stream like I do with the rest tap?
Those are pre-defined by what's discoverable in your postgres database. You can run 
meltano select tap-postgres --list --all
to see the available streams.
t
Okay so my stream ID is the same as what I'm selecting? So would that look like this?
Copy code
metadata:
  public-.ip_net_pool.*
    replicaiton-key: created_at
Copy code
select:
        - public-ip_net_plan.*
      metadata:
        public-ip_net_plan.*:
With that select I get many streams back from my list
Copy code
[selected   ] public-ip_net_plan.added
	[selected   ] public-ip_net_plan.alarm_priority
	[selected   ] public-ip_net_plan.authoritative_source
	[selected   ] public-ip_net_plan.avps
	[selected   ] public-ip_net_plan.children
	[selected   ] public-ip_net_plan.comment
	[selected   ] public-ip_net_plan.country
	[selected   ] public-ip_net_plan.customer_id
	[selected   ] public-ip_net_plan.description
	[selected   ] public-ip_net_plan.display_prefix
	[selected   ] public-ip_net_plan.expires
	[selected   ] public-ip_net_plan.external_key
	[selected   ] public-ip_net_plan.free_addresses
	[automatic  ] public-ip_net_plan.id
	[selected   ] public-ip_net_plan.indent
	[selected   ] public-ip_net_plan.inherited_tags
	[selected   ] public-ip_net_plan.last_modified
	[selected   ] public-ip_net_plan.monitor
	[selected   ] public-ip_net_plan.node
	[selected   ] public-ip_net_plan.order_id
	[selected   ] public-ip_net_plan.pool_id
	[selected   ] public-ip_net_plan.prefix
	[selected   ] public-ip_net_plan.status
	[selected   ] public-ip_net_plan.tags
	[selected   ] public-ip_net_plan.total_addresses
	[selected   ] public-ip_net_plan.type
	[selected   ] public-ip_net_plan.used_addresses
	[selected   ] public-ip_net_plan.vlan
	[selected   ] public-ip_net_plan.vrf_id
So if I do the metadata thing am I setting the replication key on each column of this table? There's only one comlumn per row that I want to be the replication ke
e
Ok, so each of those those is a selected field. The structure is 
<stream>.<field>
. In your case 
public-ip_net_plan
is the stream. If you want to select all tables starting with 
ip_
then you might want something like:
Copy code
select:
        - public-ip_*.*  # select all fields in tables starting with 'ip_' in schema 'public'
Then, if they all have the the same replication key:
Copy code
metadata:
        public-ip_*:
          replication-method: INCREMENTAL
          replication-key: last_modified
t
Thank you very much
Copy code
plugins:
  extractors:
  - name: uisp
    inherit_from: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: <https://uisp.utbb.net/nms/api/v2.1>
      auth_method: api_key
      api_keys:
        X-Auth-Token: ${UISP_TOKEN}
      streams:
        - name: onus
          path: /devices
          params:
            type: onu
          primary_keys:
          - identification_id
          schema:
            properties:
              identification_id:
                type: string
              identification_name:
                type: string
              attributes_parentId:
                type: string
              identification_mac:
                type: string
              overview_createdAt:
                type: string
                format: date-time
              identification_updated:
                type: string
                format: date-time
    pagination_request_style: single_page_paginator
    pagination_response_style: single_page_paginator
    load_schema: raw_uisp
With that rest tap I was able to have multiple streams for this one provide (uisp). I was also able to name each stream and reference them in my transformer
Copy code
WITH
source AS (
	SELECT * FROM {{ source("uisp", "onus") }}
),
Am I understanding correctly that there's no concept of a stream in the postgresql tap? How do I have multiple sources from different tables?
I guess it has multiple streams they're just automatically defined as one stream per table
e
I guess it has multiple streams they're just automatically defined as one stream per table
Correct!
4 Views
Open in Slack
PreviousNext
Powered by