Hey folks! I have some static `csv` files that I a...
# plugins-general
c
Hey folks! I have some static
csv
files that I am extracting with
tap-csv
and loading with
target-postgres
. As I mentioned, these files are static and I don't want to append the same data to my tables every time I run
meltano run
. It looks like I could perhaps use
--state-id
if I were using
meltano elt
, but I'm running on Windows and that argument is not supported by
meltano run
. Are there any brilliant ideas for essentially mirroring the CSVs? Right now, I manually empty the tables before loading, but obviously that's not ideal, and I'd rather not automate the emptying of the tables because that will disrupt users who are relying on them.
a
Have you considered loading to a
raw
area in the database in an append only fashion, then using a dbt model to move the data into a
staging
area where the CSVs are consumed
c
@alexander_butler - That is what I'm doing for much of the data, but I would really like to avoid the extra dbt "code" and keep simple things simple for some of the small lookup tables. Plus, continuing to append the data will bloat my DB unnecessarily. Though - if I end up switching all of my Loading activity to Meltano (currently using Python with Pandas read_csv() and to_sql()), it's going to become much more important, but that's further off because I have to do some light-transformation and add columns like create_date, update_date and source_file_name, prior to loading the raw data. Due to my limited upstream bandwidth, I don't want to upload anything I don't have to.