Hey folks! I have some static

csv

files that I am extracting with

tap-csv

and loading with

target-postgres

. As I mentioned, these files are static and I don't want to append the same data to my tables every time I run

meltano run

. It looks like I could perhaps use

--state-id

if I were using

meltano elt

, but I'm running on Windows and that argument is not supported by

meltano run

. Are there any brilliant ideas for essentially mirroring the CSVs? Right now, I manually empty the tables before loading, but obviously that's not ideal, and I'd rather not automate the emptying of the tables because that will disrupt users who are relying on them.

Have you considered loading to a

raw

area in the database in an append only fashion, then using a dbt model to move the data into a

staging

area where the CSVs are consumed

@alexander_butler - That is what I'm doing for much of the data, but I would really like to avoid the extra dbt "code" and keep simple things simple for some of the small lookup tables. Plus, continuing to append the data will bloat my DB unnecessarily. Though - if I end up switching all of my Loading activity to Meltano (currently using Python with Pandas read_csv() and to_sql()), it's going to become much more important, but that's further off because I have to do some light-transformation and add columns like create_date, update_date and source_file_name, prior to loading the raw data. Due to my limited upstream bandwidth, I don't want to upload anything I don't have to.