Hi everyone, I am new to it but pretty excited about Meltano and so far it is a blast. I am happy for any direction 🙏 I have created a custom Extractor for a weird external REST api that I try to pull data from. It uses Basic authentication as well as accepting only POST requests to one static url. The json in the post body is a long array of objects that always hold 3 fields to specify what you want to retrieve, looks like this:

{"inputRequests":[{"a":"string","b":"string","c":"string"}]}

Those configuration pairs (a,b,c) are stored in a growing list in a SQL database. My idea was to setup a pipeline where Plugin 1: “Extract those pairs from the SQL and provide them as JSON” and Plugin 2: I take those pairs now as JSON as input to query the API mentioned in the beginning. First - I wonder if this is the intended way to go, and how I can achieve this - handing the data result from one plugin to another one as kind of steps. And Second - how do I configure my local extractor plugin to accept a rather complex json structure as input? For the REST API I have used the cookiecutter template going the REST setting route -

Rows -> Row

contains the actual results, while the rest is just to cope with unsorted results. Currently my files are:

streams.py

class CustomResponseStream(customRestStream):
    """Define custom stream."""

    name = "CustomResponse"
    rest_method = "POST"
    records_jsonpath = "$.[*]"
    path = ""

    def prepare_request_payload(
        self, context: t.Optional[dict], next_page_token: t.Optional[t.Any]
    ) -> t.Optional[dict]:
        return {
            "inputRequests": self.config["inputRequestConfig"],
        }

    schema = th.PropertiesList(
        th.Property("CustomResponse", th.ArrayType(
            th.ObjectType(
                th.Property("a", th.StringType),
                th.Property("b", th.StringType),
                th.Property("c", th.StringType),
                th.Property("Rows", th.PropertiesList(
                    th.Property("Row", th.ArrayType(th.StringType)),
                ))
            )
        )),
    ).to_dict()

tap.py

...
config_jsonschema = th.PropertiesList(
    th.Property(
        "auth_token",
        th.StringType,
        required=True,
        secret=True,  # Flag config as protected.
        description="The token to authenticate against the API service",
    ),
    th.Property(
        "inputRequestConfig",
        th.ArrayType(
            th.ObjectType(
                th.Property("a", th.StringType),
                th.Property("b", th.StringType),
                th.Property("c", th.StringType)
            )
        ),
        required=True,
        description="Project IDs to replicate",
    ),
    th.Property(
        "api_url",
        th.StringType,
        default="<https://whateverapi.com/>....",
        description="The url for the API service",
    ),
).to_dict()
...

First - I wonder if this is the intended way to go, and how I can achieve this - handing the data result from one plugin to another one as kind of steps.

Yes, Meltano doesn't support this natively at the moment so an orchestrator (even as simple as a bash script) that calls Meltano with the output of a previous run is the way to accomplish this.

And Second - how do I configure my local extractor plugin to accept a rather complex json structure as input?

It's probably OK to accept an arbitrary object:

th.Property(
        "inputRequestConfig",
        th.ArrayType(
            th.ObjectType(additional_properties=True),
        ),
        required=True,
        description="Project IDs to replicate",
    ),

@edgar_ramirez_mondragon thx for your help 🙏 For the arbitrary object I guess that makes things a bit slimmer, thx! Now when I execute my scripts above it works just fine with the VS code testing setup and the

.secrets/config.json

input. Now I want to go for setting the config to provide the json input for my POST request. For the Auth details this works just fine via

meltano config tap-customrest set auth_token "...."

but for my JSON input, do I provide it the same way as config? I assume I would need to stringify it for that reason?

If you define the setting in

meltano.yml

as `array`:

plugins:
  extractors:
  - name: tap-customrest
    settings:
    - name: auth_token
      kind: secret
    - name: api_url
    - name: inputRequestConfig
      kind: object

then you should be able to pass a json string

meltano config tap-customrest set inputRequestConfig '[{"a":"string","b":"string","c":"string"}]'

https://docs.meltano.com/reference/plugin-definition-syntax#settingskind

Thank you 🙏 I completely forgot to state it in the settings properly.