I need to populate my schema based on custom field...
# singer-tap-developmentv
visch
10/27/2021, 12:54 AMI need to populate my schema based on custom field data. Each Project has custom fields for each Item.
Hierarchy of Streams is Teams -> Project -> CustomFields. Issue is that I need to know the list of ProjectIds during discovery. Is there anyway to "hook" into that logic. I really want to say inside of the
schemaItemStreamschemaa
aaronsteers
10/27/2021, 12:59 AMJust so I understand the use case, can you give an example or two of dynamic properties which would come from the parent?
v
visch
10/27/2021, 12:59 AMproject_idvisch
10/27/2021, 1:00 AMSpecefically I need a list of all the
project_ida
aaronsteers
10/27/2021, 1:01 AMSorry, to confirm: are the properties dynamic or the contents?
aaronsteers
10/27/2021, 1:01 AMOr did you want to represent each project in a separate column?
v
visch
10/27/2021, 1:02 AMI'll be specefic š
https://sprints.zoho.com/apidoc.html#Getitemfields
GET request is
/team/{teamId}/projects/{projectId}/visch
10/27/2021, 1:02 AMI need that for the
Itemsa
aaronsteers
10/27/2021, 1:02 AMGot it!!
v
visch
10/27/2021, 1:03 AMParent / Child relationships are all setup, and I have it working, but I need custom fields in there as well. It pairs nicely with the commment I just posted on @josh_lloydās MR š
a
aaronsteers
10/27/2021, 1:04 AMSo, in this example, presumably children of different parents have different schemas? And I guess, the target will merge as well as possible?
v
visch
10/27/2021, 1:04 AMGood question
No in this case all Items are going to have one schema. For Items that don't have that projects custom_field they'll just be null
visch
10/27/2021, 1:05 AMSeems like the best tradeoff
visch
10/27/2021, 1:07 AMBut the trick is the one schema is dynamic due to Custom Fields from each project š
a
aaronsteers
10/27/2021, 1:09 AMSo, before sync, a dag is created internally in the SDK so that everything can be ordered correctly. I don't believe discovery uses the same ordering - but this could be changed to honor the dagified-ordering if it's not doing so already. Then, presumably when Stream.schema is called the first time, it could reference some result from the parent.
aaronsteers
10/27/2021, 1:09 AMSo, "possible, but probably not easy" is the upshot :)
v
visch
10/27/2021, 1:11 AMš
Yeah I was look at
Tapsync_allload_streamsvisch
10/27/2021, 1:11 AMThis combined with having "dummy" streams would be useful in one other context I have with tap-googleads as well
a
aaronsteers
10/27/2021, 1:11 AMI think
tap.discover_streams()sync_all()v
visch
10/27/2021, 1:14 AMMakes sense, now to find the fastest way to get to what I need
visch
10/27/2021, 1:14 AMThanks
a
aaronsteers
10/27/2021, 1:17 AMIt sounds like a fun problem! While you're at it, you might run into this: https://gitlab.com/meltano/sdk/-/issues/93
aaronsteers
10/27/2021, 1:18 AMWe don't have documented patterns for "extra" requests calls - not a barrier for you but just want to call out it's a known area for improvement.
v
visch
10/27/2021, 1:20 AMGot it! Thank you! I try real hard to force everything into streams to avoid that sometimes I think a bit too much about it š
visch
10/27/2021, 1:20 AMGreat issue though!