Hey guys!! I am new to Meltano, and I am running a pipeline and transforming data it is working well, but I am struggling to make de raw data that I extract persist in my database. I setup target-postgres and after each run my database is cleared. Could you guys help me to understand this part? Thank you in advance.

Hey @david_jose_de_souza! Can you clarify this bit?

after each run my database is cleared

does that mean all the tables and schemas are dropped, or it's only that the tables are truncated? Also, are you by any chance running your database in docker?

Hey @edgar_ramirez_mondragon I think it is truncating and the new data overwrite the old, yes my database is a postgres running on a docker container.

the new data overwrite the old

That's the expected behavior of target-postgres. To upsert the records based on a primary key. If I understand correctly, I think you want the target to append records every time you run

meltano elt

?

Yes that's what I want

One solution would be remove any primary keys the streams may have, so records are only appended. Which tap are you using?

I am using tap-jira

It's possible to override the schema and metadata using meltano: https://meltano.com/docs/plugins.html#how-to-use-3

Hey @edgar_ramirez_mondragon I tried to follow these instructions you send, but I was not able to override the table-key-properties in order to keep a record of each extraction.

Hey @david_jose_de_souza! What does your

meltano.yml

look like?

This is my yml file

did you try something like this?

plugins:
  extractors:
  - name: tap-jira
    variant: singer-io
    pip_url: git+<https://github.com/singer-io/tap-jira.git>
    config:
      username: <mailto:user@example.com|user@example.com>
      password: password
      base_url: "url"
      start_date: 2020-01-01 00:00:00+00:00
      user_agent: tap-jira via Meltano
    metadata:
      issues:
        table-key-properties: []

Hey @edgar_ramirez_mondragon yes I tried, but not succeed. It keeps adding primary key and override records.

Hmm you're right, the tap doesn't seem to let your override metadata and just outputs the hardcoded pks: https://github.com/singer-io/tap-jira/blob/7b2b5ebc546c88917fe9d325a7b0e65513ee5c2d/tap_jira/__init__.py#L77-L79

Hey @edgar_ramirez_mondragon do you mean that using tap-jira there is no way to override table-key-properties? I also tried add a new column in the schema I was trying to use these guys

replication-method

 and 

replication-key

, but I was not able to add a new column with timestamp to use as replication key, and I don't even know if that way would work. What are your thoughts on that?