I am using `dockerfile` create docker I am using tap mysql a Meltano #troubleshooting

I am using `dockerfile` create docker . I am using...

atif_imam

11/01/2021, 4:49 AM

I am using

dockerfile

create docker . I am using tap-mysql and target-redshift . The issue is when I am adding passwords it is writing to

system-db

but it is unable to retrieve the passwords and fails with catalog discovery error. @aaronsteers . Your help will be appreciated.

Dockerfile

steve_clarke

11/01/2021, 5:50 AM

What does your Meltano.yml file look like. I am working with tap-mssql okay. Here is my entry for tap-mssql.

Copy code

- name: tap-mssql
    namespace: tap_mssql
    pip_url: git+<https://github.com/s7clarke10/pipelinewise-tap-mssql.git>
    executable: tap-mssql
    capabilities:
    - catalog
    - discover
    - properties
    - state
    settings:
    - name: host
      kind: string
    - name: port
      kind: string
    - name: user
      kind: string
    - name: password
      kind: password
    - name: database
      kind: string
    
    select:
    - HumanResources-Department.*
    - HumanResources-Employee.*

I also have some setting in a .env if I am running it locally for my credentials.

Copy code

####################################
# Setting for Extractor MSSQL      #
####################################

TAP_MSSQL_HOST=myhost
TAP_MSSQL_PORT=1433
TAP_MSSQL_USER=my_user
TAP_MSSQL_PASSWORD=My_password
TAP_MSSQL_DATABASE=AdventureWorks2016
TAP_MSSQL_USE_DATE_DATATYPE=true

atif_imam

11/01/2021, 5:52 AM

@steve_clarke , my yml file looks exactly like yours. .env for running locally but for docker sensitive objects are not stored in .env but in system-db and here i am getting issue.

steve_clarke

11/01/2021, 6:13 AM

Okay, I'm running in AWS so I have a SSM Parameter set for when I run in a Production like setting. I have chosen to use Postgres rather than the default sqlite3 database. Actually it is a Aurora Postgres flavour so it is only used as required. I did have to work with my networking team to have the right end points in place to ensure that I could reach this database. Initially I couldn't connect because of network routing. I followed this Meltano link for setting my System Database to use Postgres rather than sqllite3. This is working with a user and password of my choosing. https://meltano.com/docs/settings.html#database-uri

steve_clarke

11/01/2021, 9:26 AM

For storing and retrieving environment variables e.g. MSSQL passwords, there is the option of injecting them from AWS Secrets Manager or AWS SSM Parameter store if you use AWS. I use chamber to retrieve the information like the passwords when I run a pipeline - they only exist for the pipeline execution. https://meltano.com/docs/production.html#managing-configuration I like Chamber because it allows me to have different environments with the right setup. This however may not be what you are wanting to achieve. Apologies if this info isn't helpful.

2 Views

Open in Slack

Previous Next