https://linen.dev logo
Join Slack
Powered by
Hello, Is it possible for Meltano to store a compl...
# troubleshooting
j
Hello, Is it possible for Meltano to store a complex config settings like this ?
Copy code
{
  "start_date": "2019-01-01T00:00:00Z",
  "surveys": [
    {
      "surname": "MT_NPS_1",
      "name": "MT NPS Feedback - Trees ",
      "id": 4130,
      "data_points": [
        "0.Net_Promotor_Score.74062c60",
        "4130.CHECKBOX_EXTRA.option.76a76681",
        "browser",
        "OS",
        "Form trigger",
        "Page title",
        "Survey",
        "Survey ID",
        "url",
        "0.Comment.34c4bab0",
        "User agent",
        "Viewport",
        "Form completion percentage",
        "4130.CES.9723abe7",
        "4130.TEXTAREA.d0404f22",
        "4130.WEBSITE_DATA.subject.1.3510144d",
        "4130.CHECKBOX.5.50a9a825"
      ]
    },
    {
      "surname": "MM_NPS_1",
      "name": "MM NPS Feedback - Trees",
      "id": 9589,
      "data_points": [
        "0.NPS.a4048779",
        "Viewport",
        "browser",
        "User Agent",
        "9589.CHECKBOX.4.5045dc57",
        "Survey",
        "9589.CHECKBOX_EXTRA.option.04e10d1b",
        "Page title",
        "9589.CHECKBOX.5.da9e7999",
        "Form completion percentage",
        "9589.CHECKBOX.2.635ab671",
        "Form trigger",
        "url",
        "9589.CHECKBOX.3.e812b3d5",
        "OS",
        "Survey ID",
        "9589.WEBSITE_DATA.subject.1.4b334b25"
      ]
    }
  ],
  "env": "dev"
}
The script works on my computer but not with meltano I’m suspecting that the JSON validation is proving to be unsuccessful and hence the variable is passed as a string to the script
a
Hi, @juan_sebastian_suarez_valencia. What approaches have you tried so far? It should be possible to paste this json block directly into “config:” in the meltano.yml file, assuming indentation and syntax is all correct.
j
@aaronsteers This is my config file
Copy code
version: 1
send_anonymous_usage_stats: true
project_id: d1670866-a117-4e45-a4cb-81e3c946f955
plugins:
  extractors:
  - name: tap-mopinion-meister
    namespace: tap_mopinion_meister
    pip_url: git+<https://github.com/MeisterLabs/tap-mopinion-meister>
    executable: tap-mopinion-meister
    capabilities:
    - discover
    - catalog
    - state
    settings:
    - name: start_date
    - name: surveys
    - name: env
    config:
      start_date: '2019-01-01T00:00:00Z'
      surveys: '{"MT_NPS_1":{"name":"MT NPS Feedback - Trees ","id":4130,"data_points":["0.Net_Promotor_Score.74062c60","4130.CHECKBOX_EXTRA.option.76a76681","browser","OS","Form trigger","Page title","Survey","Survey ID","url","0.Comment.34c4bab0","User agent","Viewport","Form completion percentage","4130.CES.9723abe7","4130.TEXTAREA.d0404f22","4130.WEBSITE_DATA.subject.1.3510144d","4130.CHECKBOX.5.50a9a825"]},"MM_NPS_1":{"name":"MM NPS Feedback - Trees","id":9589,"data_points":["0.NPS.a4048779","Viewport","browser","User Agent","9589.CHECKBOX.4.5045dc57","Survey","9589.CHECKBOX_EXTRA.option.04e10d1b","Page title","9589.CHECKBOX.5.da9e7999","Form completion percentage","9589.CHECKBOX.2.635ab671","Form trigger","url","9589.CHECKBOX.3.e812b3d5","OS","Survey ID","9589.WEBSITE_DATA.subject.1.4b334b25"]}}'
      env: 'dev'
  loaders:
  - name: target-bigquery
    variant: adswerve
    pip_url: git+<https://github.com/adswerve/target-bigquery.git@v0.10.2>
    config:
      project_id: meister-bi-prod
      dataset_id: mopinion
      location: europe-west3
As you can see I moved from the array version from it because I saw in the documentation that my use case (array of objects) was not explicitly estipulated
So I decided to use the JSON aproch like in this example
Copy code
meltano config <plugin> set <name> '{"<key>": <value>, ...}' # JSON object
However my code still breaks
So my best guess is that meltano is not parsing the configuration and it’s sending the raw string but that’s not the desired behaviour
a
I noticed the single quotes around the ‘surveys’ json. If I remove those and paste the result in yamllint.com, I get the following:
Copy code
--- 
plugins: 
  extractors: 
    - 
      capabilities: 
        - discover
        - catalog
        - state
      config: 
        env: dev
        start_date: "2019-01-01T00:00:00Z"
        surveys: 
          MM_NPS_1: 
            data_points: 
              - 0.NPS.a4048779
              - Viewport
              - browser
              - "User Agent"
              - 9589.CHECKBOX.4.5045dc57
              - Survey
              - 9589.CHECKBOX_EXTRA.option.04e10d1b
              - "Page title"
              - 9589.CHECKBOX.5.da9e7999
              - "Form completion percentage"
              - 9589.CHECKBOX.2.635ab671
              - "Form trigger"
              - url
              - 9589.CHECKBOX.3.e812b3d5
              - OS
              - "Survey ID"
              - 9589.WEBSITE_DATA.subject.1.4b334b25
            id: 9589
            name: "MM NPS Feedback - Trees"
          MT_NPS_1: 
            data_points: 
              - 0.Net_Promotor_Score.74062c60
              - 4130.CHECKBOX_EXTRA.option.76a76681
              - browser
              - OS
              - "Form trigger"
              - "Page title"
              - Survey
              - "Survey ID"
              - url
              - 0.Comment.34c4bab0
              - "User agent"
              - Viewport
              - "Form completion percentage"
              - 4130.CES.9723abe7
              - 4130.TEXTAREA.d0404f22
              - 4130.WEBSITE_DATA.subject.1.3510144d
              - 4130.CHECKBOX.5.50a9a825
            id: 4130
            name: "MT NPS Feedback - Trees "
      executable: tap-mopinion-meister
      name: tap-mopinion-meister
      namespace: tap_mopinion_meister
      pip_url: "git+<https://github.com/MeisterLabs/tap-mopinion-meister>"
      settings: 
        - 
          name: start_date
        - 
          name: surveys
        - 
          name: env
  loaders: 
    - 
      config: 
        dataset_id: mopinion
        location: europe-west3
        project_id: meister-bi-prod
      name: target-bigquery
      pip_url: "git+<https://github.com/adswerve/target-bigquery.git@v0.10.2>"
      variant: adswerve
project_id: d1670866-a117-4e45-a4cb-81e3c946f955
send_anonymous_usage_stats: true
version: 1
It should work as yaml or json but perhaps the single quotes were causing a problem. It might be worth trying as this yaml-fied version and then also if the direct json-paste will work without single quotes.
j
The single quotes are described in the documentation
message has been deleted
@aaronsteers The YAML version works The direct json-paste also works But the documentation hence should be changed. I can make a pull request if you feel confortable with it
a
Okay, thanks very much for confirming. I don’t know why the cli version did not work as described.
I wonder if there needs to be a ‘kind: json’ (or similar) for 
meltano config
to parse the input as json. An issue or pull request would be much appreciated! Definitely we want this to work as described in the docs.
I found an instance where 
kind: object
is used for a structured key-value json-type object: src/meltano/core/bundle/discovery.yml · master · meltano / Meltano · GitLab I’d have to test but that might be the trigger to get json parsing instead of plain-old string treatment. (Either way, we should make sure this is noted in the docs.)
Copy code
settings:
        - name: surveys
          kind: object
        # ...
j
a
Great - thank you
j
Thank you very much for your help @aaronsteers I would have spent hours trying to figure out if it was my fault or not
a
This is a sticky one, for sure. I’m glad it worked!
Open in Slack
PreviousNext
Powered by