https://linen.dev logo
Join Slack
Powered by
how to create multiple extractors and loaders in m...
# troubleshooting
v
how to create multiple extractors and loaders in meltano for multiple databases [10:13 AM] In order to handle saas project. Need to extract multiple database from postgreesql and load into the starrocks datawarehouse [10:13 AM] Need to configure ELT process separately for all databases
a
v
Using that we can able to use extract. Then how to provide saparate load path for that? @Andy Carter
a
What do you mean by separate load path? A different schema in your database for each extractor?
v
I mean loaders. @Andy Carter
Yes
a
What target are you using? 
postgres
? 
snowflake
?
Copy code
- name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      default_target_schema: raw__${MELTANO_EXTRACTOR_NAMESPACE}
The postgres config has a value called 
default_target_schema
which you can pass the env variable: 
MELTANO_EXTRACTOR_NAMESPACE
So if I have 
tap-mysql
as a extractor, then this data gets loaded into a postgres schema called 
raw__tap-mysql
v
As of now i am using JSONL.Folder path.
a
Can you share your 
meltano.yml
removing any sensitive info?
v
Okay.
Is there any support to load directly in mysql?
version: 1 default_environment: dev project_id: environments: - name: dev - name: staging - name: prod plugins: extractors: - name: tap-postgres variant: meltanolabs pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git config: database: dev-org host: port: user: password: flattening_enabled: true flattening_max_depth: 25666 select: - public-testdb.* metadata: public-testdb: replication-method: INCREMENTAL replication-key: last_modified_on - name: tap-postgress-devorg1002 inherit_from: tap-postgres variant: meltanolabs pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git config: database: host: port: user: password: flattening_enabled: true flattening_max_depth: 25666 select: - public-testdb.* metadata: public-testdb: replication-method: INCREMENTAL replication-key: last_modified_on loaders: - name: target-jsonl variant: andyh1203 pip_url: target-jsonl config: destination_path: devorg
sample.yaml
sample.yaml
a
I can't see a mysql target on the hub, try searching the slack for 
target-mysql
as someone may have a version. I take it your DWH is mysql?
v
Yes. Actually we have an saas application with cloud postgree database. We need to do Flatten,ETL and have to load in starrocks. Can also suggest for this @Andy Carter
a
https://hub.meltano.com/extractors/tap-postgres will get the data out initially. I don't do much transformation in meltano, but it is possible with stream maps. I handle it using dbt.
You can use a 
mapper
plugin between your tap and target: https://docs.meltano.com/guide/mappers#standalone-mapper-plugins
v
Thank you. I will check on DBT andmapper
e
If it's helpful, Meltano populates a few env vars at runtime that might help with your use case
Copy code
loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
    config:
      destination_path: $MELTANO_EXTRACTOR_NAMESPACE_devorg
Open in Slack
PreviousNext
Powered by