https://linen.dev logo
Join Slack
Powered by
:wave: I configured my `meltano.yaml` file to read...
# getting-started
b
👋 I configured my 
meltano.yaml
file to read from a local db server .. and worked perfectly, but the issue is when I'm running the same meltano config (but targeting a deployed db instance - using 
tap-mysql
- 100% sure of the db connection string, I'm getting this error:
Copy code
[2m2023-06-30T14:12:20.550169Z[0m [[32m[1minfo     [0m] [1mEnvironment 'dev' is active[0m
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.
Plugin configuration is invalid
No RECORD message received
Here's the yaml file:
Copy code
version: 1
default_environment: dev
project_id: 31c666cb-33b9-43a3-884f-88af7ea504e4
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-mysql
        config:
          host: host
          port: 3306
          user: user
          database: sampledb
          password: password
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
    select:
    - sampledb-table1.*
    metadata:
      sampledb-table1:
        replication-method: INCREMENTAL
        replication-key: id
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
Here's the command I was running:
Copy code
meltano --environment=dev config tap-mysql test
Note: I tried also this select and it worked locally 😄 but not agains the real db I've got on aws
Copy code
select:
    - '*-house_battles.*'
Also, The way I'm running it remotely .. I built a docker image of the meltano project and pushed it to ECR and then ran it on an EC2 instance .. Tested another command 
meltano environment list
and it worked .. thinkspin
u
Hmm I'm not quite sure since theres a bunch of moving parts but 
No RECORD message received
is probably the culprit. The test command you're running is basically starting a sync then stopping once it receives the first record i.e. successfully connected. If you had no access it would probably have raised an exception saying so. To me that means the reason its not passing the test is because its not getting data and not getting data could be due to bad selection criteria. I'd double check your prod db to make sure the db/schema/table name are as expected and mess with the select config, maybe make it less restrictive to start out.
j
Are you sure there is a table 
table1
in schema 
sampledb
Copy code
sampledb.table1
in your database called 
sampledb
?
Copy code
select:
    - sampledb-table1.*
    metadata:
      sampledb-table1:
        replication-method: INCREMENTAL
        replication-key: id
Does running following query on your mysql database work for you?
Copy code
SELECT * FROM sampledb.table1 LIMIT 5
edit: nevermind.. I had not used mysql in years and had forgotten it does not have schema i.e. like in postgresql
b
@janis_puris @pat_nadolny yes .. i downloaded the exact copy of the db / tables from the server and tested the configs against it locally before testing them again (after changing the connection for sure) but had that error 😞
j
give me a moment, I'll try to reproduce this 😕
I can reproduce the problem thinkspin spin up mysql 8 on docker
Copy code
docker run --rm --name some-mysql -p '127.0.0.1:3306:3306' -e MYSQL_ROOT_PASSWORD=password mysql:8 -d
new meltano project
Copy code
mkdir tmp && cd $_
python -m venv venv && activate
pip install meltano
meltano init project && cd project
vim meltano.yml
# Paste the config the OP has provided
# Replace host value with 127.0.0.1
# Replace database value with mysql
# Replace user value with root
# Replace str "sampledb-table1" with "mysql-table1"
create a test table in mysql db
Copy code
docker exec -it some-mysql mysql -e "CREATE TABLE mysql.table1 (id int);" -p
docker exec -it some-mysql mysql -e "INSERT INTO mysql.table1 VALUES (1), (2), (3);" -p
Copy code
❯ docker exec -it some-mysql mysql -e "SELECT * FROM mysql.table1;" -p
Enter password:
+------+
| id   |
+------+
|    1 |
|    2 |
|    3 |
+------+
run meltano cmd
Copy code
❯ meltano --environment=dev config tap-mysql test
2023-06-30T14:56:00.382842Z [info     ] Environment 'dev' is active
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
No RECORD message received
@bassel what is the mysql version you're using?
b
mysql 8
but if you run a meltano elt command to extract data (based on the same config select and metadata) and extract it into a json using target-jsonl extractor .. it works 😄 i mean if you skip the config test command
j
That makes no sense. From what I can tell, the discovery is failing
p
you can try adding 
--log-level=debug
to the config/test command to see if it tells you anything else
b
will try .. @pat_nadolny
@janis_puris yeah .. weird
j
facepalm trek my MRE is invalid, because the discovery excludes 
mysql
schema.. woops
Copy code
SELECT table_schema,
table_name,
table_type,
table_rows
FROM information_schema.tables
WHERE table_schema NOT IN (
'information_schema',
'performance_schema',
'mysql',
'sys'
)
b
i have a similar one ..
Copy code
if dbs:
        filter_dbs_clause = ",".join([f"'{db_name}'" for db_name in dbs.split(",")])
        table_schema_clause = f"WHERE table_schema IN ({filter_dbs_clause})"
    else:
        table_schema_clause = """
        WHERE table_schema NOT IN (
        'information_schema',
        'performance_schema',
        'mysql',
        'sys'
        )"""
should it be changed?
j
Nah.. it was me naively picking 
mysql
. You are using user created db.
b
@pat_nadolny Just ran it with logs and had this
Copy code
2023-06-30T11:38:46.719-04:00	[31m│[0m [31m❱ [0m393 [2m│ │ [0m[94mraise[0m CliError([33m"[0m[33m\n[0m[33m"[0m.join(([33m"[0m[33mPlugin configuration is invalid[0m[33m"[0m, d [31m│[0m
the logs are a bit messed up because I'm running an aws batch job (single node ec2) targeting the docker image in ECR
j
I'm not able to reproduce the problem you're having 😞
Copy code
❯ meltano --environment=dev config tap-mysql test
2023-06-30T15:43:30.283839Z [info     ] Environment 'dev' is active
Plugin configuration is valid
Copy code
❯ meltano --environment=dev select --list --all tap-mysql
2023-06-30T15:44:03.795120Z [info     ] Environment 'dev' is active
Legend:
	SelectionType.SELECTED
	SelectionType.EXCLUDED
	SelectionType.AUTOMATIC

Enabled patterns:
	test_db-table1.*

Selected attributes:
	[SelectionType.SELECTED] test_db-table1.id
b
@janis_puris hmm .. will try to run this 
meltano --environment=dev config tap-mysql test
and see what error it shows 🤞
j
I think I know what is causing this @bassel Please try to move the
Copy code
select:
        - sampledb-table1.*
      metadata:
        sampledb-table1:
          replication-method: INCREMENTAL
          replication-key: id
to the dev environment. Same place you've got the config of the extractor.
b
silly question, if that's the case .. then why it works locally? @janis_puris
j
I do not know. What I do know is that I can replicate the problem with • extractor config in environment • extractor select, metadata in plugins I can not replicate the problem with • extractor config in environment • extractor select, metadata in environment or • extractor config in plugins • extractor select, metadata in plugins
Discovery is failing i.e. the query that returns stream list for the catalog.json has no results
b
i see .. ok, I'll try it now 🤞
j
I need to run now and will be able to assist in couple of hours, if this does not help 🤞
b
@janis_puris Appreciate the help gratitude thank you
@janis_puris Just moved the config to the env section .. still getting the same error 😢
Copy code
root@7c22418a67ed:/project# meltano --environment=dev config tap-mysql test
2023-06-30T17:48:16.399915Z [info     ] Environment 'dev' is active
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
No RECORD message received
root@7c22418a67ed:/project#
j
Something does not add up.. could you please run
Copy code
meltano --environment=dev --log-level=debug config tap-mysql test
Anyhow, here is MySQL 8 running on docker that I can not reproduce the issue after applying the fix I've mentioned. Recipe run mysql in docker
Copy code
docker run --rm --name some-mysql -p '127.0.0.1:3306:3306' -e MYSQL_ROOT_PASSWORD=password mysql:8
docker exec -it some-mysql mysql -e "CREATE DATABASE somedb; CREATE TABLE somedb.table1 (id int); INSERT INTO somedb.table1 VALUES (1), (2), (3); SELECT * FROM somedb.table1;" -p
password is 
password
create meltano project
Copy code
mkdir tmp && $_
python3 -m venv venv && source venv/bin/activate
pip install meltano
meltano init project
vim meltano.yml # Replace contents with the sample
meltano install
test the tap
Copy code
❯ meltano --environment=dev config tap-mysql test
2023-06-30T19:49:38.258688Z [info     ] Environment 'dev' is active
Plugin configuration is valid
🤷
meltano.yaml
Open in Slack
PreviousNext
Powered by