Anyone had experience getting the twilio tap to work with postgres target? I'm using the transferwise variant for both. First I tried with a twilio subaccount SID and that just returned a bunch of 404s from the API so I switched to the twilio master account ID and data seems to be pulled correctly. The error i'm getting is in the target ```tap-twilio | time=2021-06-22 215550 name=singer level=INFO message=METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.024429798126220703, "tags": {"endpoint": "usage", "http_status_code": 200, "status": "succeeded"}} tap-twilio | time=2021-06-22 215550 name=singer level=INFO message=METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"endpoint": "usage"}} tap-twilio | time=2021-06-22 215550 name=singer level=INFO message=Stream usage, batch processed 1 records tap-twilio | time=2021-06-22 215550 name=singer level=INFO message=START Syncing: usage_records tap-twilio | time=2021-06-22 215550 name=singer level=INFO message=START Sync for Stream: usage_records, parent_stream: usage, parent_id: None tap-twilio | time=2021-06-22 215550 name=singer level=INFO message=stream: usage_records, bookmark_field: end_date, last_datetime: 2021-04-01T000000+00:00 tap-twilio | time=2021-06-22 215550 name=singer level=INFO message=START Sync for Stream: usage_records, Date window from: 2021-04-01 000000+00:00 to 2021-05-01 000000+00:00 tap-twilio | time=2021-06-22 215550 name=singer level=INFO message=URL for Stream usage_records: https://api.twilio.com/2010-04-01/Accounts/xxxxx/Usage/Records.json?StartDate=2021-04-01&amp;EndDate=2021-05-01 target-postgres | time=2021-06-22 215550 name=target_postgres level=INFO message=Table '"message_media"' exists target-postgres | time=2021-06-22 215550 name=target_postgres level=INFO message=Table '"message_media"' exists target-postgres | time=2021-06-22 215550 name=target_postgres level=INFO message=Table '"usage"' does not exist. Creating... CREATE TABLE IF NOT EXISTS twilio."usage" ("_sdc_batched_at" timestamp without time zone, "_sdc_deleted_at" character varying, "_sdc_extracted_at" timestamp without time zone, "subresource_uris" jsonb, PRIMARY KEY ("")) target-postgres | Traceback (most recent call last): target-postgres | File "/home/meltano/meltano/twilio-qa/.meltano/loaders/target-postgres/venv/bin/target-postgres", line 8, in <module> target-postgres | sys.exit(main()) target-postgres | File "/home/meltano/meltano/twilio-qa/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/__init__.py", line 373, in main target-postgres | persist_lines(config, singer_messages) target-postgres | File "/home/meltano/meltano/twilio-qa/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/__init__.py", line 219, in persist_lines target-postgres | stream_to_sync[stream].sync_table() target-postgres | File "/home/meltano/meltano/twilio-qa/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/db_sync.py", line 589, in sync_table target-postgres | self.query(query) target-postgres | File "/home/meltano/meltano/twilio-qa/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/db_sync.py", line 313, in query target-postgres | params target-postgres | File "/home/meltano/meltano/twilio-qa/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/psycopg2/extras.py", line 143, in execute target-postgres | return super(DictCursor, self).execute(query, vars) target-postgres | psycopg2.errors.SyntaxError: zero-length delimited identifier at or near """" target-postgres | LINE 1: ...thout time zone, "subresource_uris" jsonb, PRIMARY KEY ("")) target-postgres | ^ target-postgres | meltano | Loading failed (1): (see above) meltano | ELT coul…

Yes, agreed. In your logs, I see

PRIMARY KEY ("")

which is probably driven by the key properties being set to a list containing a one-item empty string. It should be either an empty set or a set/list of string key names.

thanks @aaronsteers I forked it and made the change to set key key_properties as [], seeing other issues now

@peter_kosztolanyi 👆 Looks like Mark is running into some issues - is this tap fairly new and would you expect some issues?

Needed this to get it working to accept no primary key https://gist.github.com/reedox/e60b6f7baedfabcfb221ed8783b8c80f#file-gistfile1-txt-L43

Good luck! I know Peter (now cc'd) is really good about managing these repos. I'm sure he'll be happy to take a PR if you are able to add fixes/improvements.

hello, we don’t use the usage endpoints in tap-twilio, it’s not included in our stream selection on prod and perhaps the reason why we haven’t seen this issue. Extracting

UsageRecords

and

UsageTriggers

added by this PR, which as you said doesn’t define PK for the

users

stream and we should address. I have very limited knowledge about tap-twilio but maybe you can try one of these: 1. Define a PK for users stream and send PR 2. Disable the users stream in the selection 3. target_postgres has the option to disable PK check:

primary_key_required: False

, but I would not recommend disabling it because most probably that would cause more issues than it solves.