CSV - best of both worlds (tap-csv & tap-sprea...
# plugins-general
CSV - best of both worlds (tap-csv & tap-spreadsheets_anywhere) I have csv files with some fixed columns and some ephemeral columns. This means that some columns exist in some csv files, but not others. However, I want to load all of the csv files into the same table in postgres.
(transferwise), in conjunction with
, let's me do this today using with the
config setting. Each time I extract a new csv, target-postgres will add any columns that are missing. Unfortunately, tap-csv does not seem to support state. If I add a new file to the
folder, tap-csv will extract all files, not just the new one. State support is also called out as missing here: Implement State Capability On the other hand, while
properly supports state, the new columns are not automatically created for some reason. (Not sure why, it seems that capability is part of target-postgres, not the extractor...). I'd like to see one extractor with both features, but I'm wondering which one would be better to improve? tap-csv seems like the best place, but there are some other rich capabilities in tap-spreadsheets-anywhere that myself and others would certainly benefit from. Any thoughts on combining the features of spreadsheets-anywhere into tap-csv?