https://linen.dev logo
Join Slack
Powered by
Is there a way I can use the `select:` config to c...
# singer-tap-development
a
Is there a way I can use the 
select:
config to change what gets passed to my tap's rest API call? For mailchimp, you can choose to exclude certain variables which means a quicker api response. If those are excluded in 
select:
, I will change the api call. More info below:
This is my config.json used for testing:
Copy code
{
  "api_key": "...",
  "start_date": "2001-04-01",
  "dc": "...",
  "select": [
    "*.*",
    "!*.ip",
    "!*.location",
    "!*.merge_fields"
  ]
}
But if I check 
self.metadata
for the field in question:
Copy code
<tap_mailchimp.streams.ListsMembersStream object at 0x7fc64857f700>
self.metadata[('properties', 'merge_fields')]
Metadata(inclusion=<InclusionType.AVAILABLE: 'available'>, selected=None, selected_by_default=None)
it looks like it hasn't been excluded. Does the metadata only apply when sent to a target?
e
select
is a Meltano feature that works as a nice interface to the singer catalog, so outside of Meltano taps don't usually handle a direct 
select
input but instead expect selection to be handled by passing 
--catalog
. The stream class has a 
mask
property that you can use to check if a field is selected, via the passed catalog.
I've done something like this before:
Copy code
def get_url_params(
        self,
        context: dict | None,  # noqa: ARG002
        next_page_token: int,
    ) -> dict[str, t.Any]:
        """Get URL query parameters for Mailchimp streams."""
        <http://self.logger.info|self.logger.info>("Page offset %s", next_page_token)


        fields = self.selected_fields
        <http://self.logger.info|self.logger.info>("Fields %s", fields)
        return {
            "count": self._page_size,
            "offset": next_page_token,
            "fields": fields,
        }


    @property
    def selected_fields(self) -> list[str]:
        """Get selected fields from the input catalog.


        Returns:
            List of selected fields, with nested properties delimited by a dot.
        """
        if self._tap_input_catalog:
            return [
                ".".join(breadcrumb[1::2])
                for breadcrumb, selected in self.mask.items()
                if selected
            ]


        return []
a
Think I understand. But I guess I can't use 
select:
in my config.json like above
e
Yup, 
select
can't be used in config.json. There's an issue suggesting we port that behavior from Meltano: https://github.com/meltano/sdk/issues/1383
Open in Slack
PreviousNext
Powered by