https://linen.dev logo
Join Slack
Powered by
Hi there, I'm trying to use the `tap-klaviyo` plug...
# plugins-general
f
Hi there, I'm trying to use the 
tap-klaviyo
plugin (meltanolabs variant), but i'm having issues and wanted to aks here if anyone had the same problems, before opening an issue on github. I want to extract all my lists and profiles form Klaviyo. Extracting Lists succeeds without error but does not extract anything, and extracting Profiles fails with the error 
Bad Request for path: /api/profiles
. Here is the relevant part from my Meltano Config:
Copy code
- name: tap-klaviyo
	config:
		revision: "2023-02-22"
		start_date: "2023-01-01"
	select:
		- lists.*
		- profiles.*
I already tried the following: • creating a new Full Access API Token • creating a new Read-Only Everything API Token (i prefer that over a full access token, as read-only should suffice for extraction) • using older revisions ("1", "2", "2022-10-17") • omitting the 
start_date
or using a different 
start_date
• requesting only specific profile attributes (eg 
profiles.id
) • enabling 
replication-method: FULL_TABLE
The same issue also occurs with the 
events
endpoint. What is curios is that it does extract 20 profiles (or 200 events when selecting that) before failing!
p
@edgar_ramirez_mondragon any ideas for Felix? Is this something you've seen before? I'm not familiar with the tap at all but hearing that it extracts data then eventually fails makes me think wonder if the last iteration is failing due to a missing value in the pagination request or something 🤔
f
@pat_nadolny Thanks for your reply!
any ideas for Felix? Is this something you've seen before? I'm not familiar with the tap at all but hearing that it extracts data then eventually fails makes me think wonder if the last iteration is failing due to a missing value in the pagination request or something
It definitely isn't the last request, as it fails after 20 of about 25k records. I decided to open an issue on GitHub: https://github.com/MeltanoLabs/tap-klaviyo/issues/13
p
Ah ok thats helpful! Its definitely strange that some requests succeed then eventually start failing, some assumption must be wrong about how the requests are generated. Thanks for opening that issue
f
Yes I figured that it is probably a bug with the plugin, klaviyo itself or a combination of both, rather than me using the plugin wrong. Thus a github issue is probably the better place to figure this out 🙂
p
Yes good call - this meltanolabs variant is new so feedback like this is really helpful. We'll try to address the issues quickly
e
My guess is a either a missing header or an unexpected query parameter
f
Probably. Unfortunately the plugin doesn't log anything other than 
Bad Request
so it's hard to debug.
t
Looks like we have a PR to fix this issue! @edgar_ramirez_mondragon is tagged to review https://github.com/MeltanoLabs/tap-klaviyo/pull/14
f
amazing, thank you!
Hi @edgar_ramirez_mondragon, thanks again for your help. I don't want to Issue-Spam on GitHub, but I noticed that flattening does not work, I tried all kinds of 
flattening_max_depth
, but always get the same issue:
Copy code
Run invocation could not be completed as block failed: Cannot start plugin tap-klaviyo: Catalog discovery failed: command ['[REDACTED]/.meltano/extractors/tap-klaviyo/venv/bin/tap-klaviyo', '--config', '[REDACTED]/.meltano/run/tap-klaviyo/tap.db611d05-319e-4ea0-b591-28080339eea7.config.json', '--discover'] returned 1 with stderr:
 2023-05-22 22:09:23,807 | INFO     | tap-klaviyo          | Added 'listperson' as child stream to 'lists'
Traceback (most recent call last):
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/bin/tap-klaviyo", line 8, in <module>
    sys.exit(TapKlaviyo.cli())
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 516, in cli
    tap = cls(  # type: ignore[operator]
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 103, in __init__
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 628, in register_raw_streams_from_catalog
    self.register_raw_stream_schema(
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 671, in register_raw_stream_schema
    self.default_mapper_type(
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 93, in __init__
    self.transformed_schema = self.flatten_schema(self.transformed_schema)
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 141, in flatten_schema
    return flatten_schema(
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/helpers/_flattening.py", line 205, in flatten_schema
    new_schema["properties"] = _flatten_schema(
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/helpers/_flattening.py", line 244, in _flatten_schema
    _flatten_schema(
  File "[REDACTED]/.meltano/extractors/tap-klaviyo/venv/lib/python3.9/site-packages/singer_sdk/helpers/_flattening.py", line 255, in _flatten_schema
    if list(v.values())[0][0]["type"] == "string":
TypeError: string indices must be integers
Is that a known issue, or maybe something with the singer sdk? (I noticed that there seems to be no code related to flattening in the tap itself).
Let me know if that is a known issue or something to be fixed, then I'll have a look at the code tomorrow, maybe I can finally be of some help here 😛
t
@felix_klein are you able to invoke tap-klaviyo separately in debug mode to see more detail about the error? 
meltano --log-level=debug invoke tap-klaviyo
the 
Catalog discovery failed
is what’s pointing me to recommend taht
e
That error makes me think there’s a mismatch between record and schema messages
f
sure, here you go!
Untitled
So I did some debugging and narrowed the issue down to the "properties" property in the 
profiles.json
schema. See: https://github.com/MeltanoLabs/tap-klaviyo/blob/d049862b4d8d3426481c6739f48ec98a06dda09c/tap_klaviyo/schemas/profiles.json#L175C19-L189
When I remove that whole block flattening works (without that property of course) - however I'm now a bit stuck on how to fix this as I'm not at all familiar with the meltano sdk. Not sure if the bug lies ins tap-klavyio or the sdk.
let me know if you want me to open an gh issue for this - I'm just not sure if it should be in the tap or sdk repo 🙂
t
If you could open an issue in the tap repo that would be helpful! @stephen_sciortino if you have a chance to check this issue out with tap-klaviyo that’d be great. thank you!
s
Hi @felix_klein what method are you using to flatten the data? Just to confirm - does the tap run successfully without the flatten step but with the full schema?
f
@stephen_sciortino Yes, without flattening everything works fine (but the results have jsonb columns with the data in it) When i set 
flattening_enabled: true
in the tap config, the mentioned errors occur. Max Depth does not matter. As I'm using target-postgres, i can use the data with postgres jsonb operators, so this issue is not a blocker, but flattening would be nice of course 🙂 I will open an issue with more detail
Hi there! I'm back with another problem with this tap - sorry 🙈 . I'm trying to import the relationship between lists and profiles. I see that the tap provides the 'listperson' stream, which i selected:
Copy code
- name: tap-klaviyo
            config:
              # flattening_max_depth: 1
              # flattening_enabled: true
              revision: "2023-02-22"
              start_date: "2023-01-01"
            select:
              - lists.*
              - listperson.*
              - profiles.*
And meltano confirms that it is selected: ```> meltano select tap-klaviyo --list --all 150907 2023-06-09T131526.421161Z [info ] Environment 'dev' is active Legend: selected excluded automatic Enabled patterns: lists.* listperson.* profiles.* Selected attributes: [excluded ] campaigns.attributes [excluded ] campaigns.attributes.archived [excluded ] campaigns.attributes.channel [excluded ] campaigns.attributes.created_at [excluded ] campaigns.attributes.message [excluded ] campaigns.attributes.name [excluded ] campaigns.attributes.scheduled_at [excluded ] campaigns.attributes.send_time [excluded ] campaigns.attributes.status [excluded ] campaigns.attributes.type [excluded ] campaigns.attributes.updated_at [excluded ] campaigns.id [excluded ] campaigns.updated_at [excluded ] events.attributes [excluded ] events.attributes.datetime [excluded ] events.attributes.event_properties [excluded ] events.attributes.metric_id [excluded ] events.attributes.profile_id [excluded ] events.attributes.timestamp [excluded ] events.attributes.uuid [excluded ] events.datetime [excluded ] events.id [excluded ] events.name [excluded ] events.type [excluded ] flows.attributes [excluded ] flows.attributes.archived [excluded ] flows.attributes.created [excluded ] flows.attributes.name [excluded ] flows.attributes.status [excluded ] flows.attributes.trigger_type [excluded ] flows.attributes.updated [excluded ] flows.id [excluded ] flows.type [automatic] listperson.id [selected ] listperson.list_id [selected ] listperson.type [selected ] lists.attributes [selected ] lists.attributes.created [selected ] lists.attributes.name [selected ] lists.attributes.updated [automatic] lists.id [automatic] lists.updated [excluded ] metrics.attributes [excluded ] metrics.attributes.created [excluded ] metrics.attributes.integration [excluded ] metrics.attributes.integration.category [excluded ] metrics.attributes.integration.id [excluded ] metrics.attributes.integration.name [excluded ] metrics.attributes.name [excluded ] metrics.attributes.updated [excluded ] metrics.id [excluded ] metrics.updated [selected ] profiles.attributes [selected ] profiles.attributes.created [selected ] profiles.attributes.email [selected ] profiles.attributes.external_id [selected ] profiles.attributes.first_name [selected ] profiles.attributes.image [selected ] profiles.attributes.last_event_date [selected ] profiles.attributes.last_name [selected ] profiles.attributes.location [selected ] profiles.attributes.location.address1 [selected ] profiles.attributes.location.address2 [selected ] profiles.attributes.location.city [selected ] profiles.attribu…
s
Hi Felix, I'll take a look into this today. The 
listperson
table doesn't sync at all?
The 
id
column in the 
listperson
table should join to the id in 
profiles
, making it possible to map profiles to lists.
Copy code
select listperson.list_id, listperson.id as profile_id_from_listperson, profiles.id as profile_id, profiles.attributes
from TAP_KLAVIYO.LISTPERSON
left join TAP_KLAVIYO.PROFILES
    on LISTPERSON.ID = PROFILES.ID
order by 1;
f
hi @stephen_sciortino, yes the listperson table doesnt sync at all
message has been deleted
I also tried clearing caches etc
when i add another stream, like 'events', it gets synced
Untitled
Untitled
Here is some debug output 🙂 : ``````
s
Hi @felix_klein, unfortunately I can't replicate this issue on my end. When I selected those three streams (list, listperson, and profiles) I get output for all three. In the logs you posted it looks like Meltano is recognizing that listperson is selected and it's calling the API endpoint for this list: 
RraK68
. Could you confirm that the list in Klaviyo is populated with profiles? For reference, here's the page for that endpoint in Klaviyo's API docs.
2 Views
Open in Slack
PreviousNext
Powered by