I successfully ran my own Tap just now from the CLI.. woooot I now want to run it via a pipeline in the UI.. it's a tap.. which I want to use with a postgresql target.. the issue is , somehow my postgresql-target is receiving my custom taps configuration and crashing.. Am I to guess this is because I am running perhaps in a single virtual env for python?? (both the meltano server itself AND the taps needed libs are all installed together)

meltano         | Running extract & load...
target-postgres | INFO PostgresTarget created with established connection: `user=juju_meltano password=xxx dbname=mydb host=1.2.3.4 port=5432 sslmode=prefer sslcert=~/.postgresql/postgresql.crt sslkey=~/.postgresql/postgresql.key sslrootcert=~/.postgresql/root.crt sslcrl=~/.postgresql/root.crl`, PostgreSQL schema: `public`
tap-kr        | Installing dependencies from lock file
tap-kr        | 
tap-kr        | Package operations: 0 installs, 0 updates, 1 removal
tap-kr        | 
tap-kr        |   • Removing atomicwrites (1.4.0)
tap-kr        | 
tap-kr        | Installing the current project: tap-kr (0.0.1)
tap-kr        | time=2021-09-06 22:53:20 name=tap-kr level=INFO message=tap-kr v0.0.1, Meltano SDK v0.3.6)
tap-kr        | time=2021-09-06 22:53:20 name=tap-kr level=INFO message=Skipping parse of env var settings...
tap-kr        | time=2021-09-06 22:53:20 name=tap-kr level=INFO message=Config validation passed with 0 errors and 0 warnings.
tap-kr        | time=2021-09-06 22:53:20 name=root level=INFO message=Operator '{MAPPER_ELSE_OPTION}=None' was not found. Unmapped streams will be included in output.
tap-kr        | time=2021-09-06 22:53:20 name=tap-ibkr level=INFO message=Beginning full_table sync of 'stuff'...
tap-kr        | time=2021-09-06 22:53:20 name=tap-ibkr level=INFO message=Tap has custom mapper. Using 1 provided map(s).
target-postgres | ERROR Unable to parse JSON: CONFIG = {MY_CUSTOM_TAPS_CONFIG_CONTENTS}
target-postgres | 
target-postgres | CRITICAL Expecting value: line 1 column 1 (char 0)
target-postgres | Traceback (most recent call last):
target-postgres |   File "/home/emcp/Dev/git/meltano/.meltano/loaders/target-postgres/venv/bin/target-postgres", line 8, in <module>
target-postgres |     sys.exit(cli())
target-postgres |   File "/home/emcp/Dev/git/meltano/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/__init__.py", line 45, in cli
target-postgres |     main(args.config)
target-postgres |   File "/home/emcp/Dev/git/meltano-/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/__init__.py", line 39, in main
target-postgres |     target_tools.main(postgres_target)
target-postgres |   File "/home/emcp/Dev/git/meltano-/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/target_tools.py", line 28, in main
target-postgres |     stream_to_target(input_stream, target, config=config)
target-postgres |   File "/home/emcp/Dev/git/meltano-/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/target_tools.py", line 77, in stream_to_target

here's my meltano.yml snippet

plugins:
  extractors:
  - name: tap-kr
    namespace: tap_kr
    executable: /home/emcp/Dev/git/stonks-tap-kr/tap-kr/tap-kr.sh
    config:
      custom_host: 1.2.3.9
      custom_active_port: 0000
      another_host: 1.2.3.9
      another_port: 0000
  loaders:
  - name: target-postgres
    variant: datamill-co
    pip_url: singer-target-postgres
    config:
      postgres_host: 1.2.3.4
      postgres_username: juju_meltano
      postgres_password: ***
      postgres_database: mydb
      postgres_schema: public
      disable_collection: true

@emcp Do you have any leftover debug print statements in your tap? If you’re writing to stdout, the target will try to parse that output as JSON Singer messages, and it will fail if it sees anything that looks unexpected, like debug output 🙂

ahhh I was unaware of this behavior.. I should finish watching this great video you all just launched I have tons and tons of print statements.. I will drop those immediately! Thank you

Nice!

I set it all up via Juju.. and hoping to open source that soon so.. anyone with LXD and Juju can spin up Meltano in a LXD container.. relate it to a postgresql target.. and infra-as-code sort of getting started using it

Good luck, I hope Meltano, Singer, and dbt can become core components of your data stack 🙂

oh and I also related Apache Superset to this setup

That’s awesome, that’s exactly the vision we have for Meltano-powered data stacks, as we build integrations with more and more data tools including Superset. The idea is that Meltano can serve as the foundation that allows a team to piece together their ideal data stack from different open source components, with the entire thing being managed in a single repo with a

meltano.yml

, and all of the advantages of version control, code review, and CI/CD that web dev teams have gotten used to

Juju and Charmhub do look interesting