Hi all, is there any ways that I could extract chu...
# getting-startedl
ly_pham
06/07/2021, 2:36 PMHi all, is there any ways that I could extract chunk of data in the span of 5-10 days each run? That would be very nice
t
taylor
06/07/2021, 2:43 PMIf the tap supports incremental loading like that, then absolutely! Which tap are you working with?
l
ly_pham
06/07/2021, 3:17 PMI am looking for Mysql/Postgres/Mongo
ly_pham
06/07/2021, 3:18 PMdoes the exisiting one support it? And if there is, is there any example on how to config it (In Meltano.yml right)? Thanks alot
t
taylor
06/07/2021, 9:15 PMI misread what you asked originally. Most loaders should support that incremental loading, but it’s a bigger question if the tap supports incremental extraction
l
ly_pham
06/09/2021, 3:59 AMoh, thank you!
e
edward_smith
06/10/2021, 12:30 PM@ly_pham, This comes with no warranty, but I modified tap-postgres to do a TIME_BASED sync. There is a description of how to use it in the README.md: https://github.com/ers81239/pipelinewise-tap-postgres/tree/time_based_sync
k
kat_crichton-seager
09/07/2021, 10:47 AMI'm having trouble running an initial extract of a large table, because the Heroku PostgreSQL instance runs out of temporary disk space. Presumably this would allow me to run it in smaller chunks and avoid that problem?
kat_crichton-seager
09/07/2021, 2:17 PMNope, still seem to be running out of disk space.
e
edward_smith
09/07/2021, 7:11 PMIs Heroku Postgres your target database?
k
kat_crichton-seager
09/10/2021, 8:10 AMNope, it's the tap. The target is Snowflake.
e
edward_smith
09/10/2021, 2:26 PMGotcha... that is tough. I build the above time_based_sync to deal with a similar problem. It might work for you as well
k
kat_crichton-seager
09/10/2021, 3:54 PMHeroku PostgreSQL has a ridiculously small amount of temp file space, it runs out with only 30,000 rows! In order to do my initial import, I've decided to run on my laptop with a downloaded copy of the production database.
kat_crichton-seager
09/10/2021, 3:56 PMIt feels like your branch should have worked for me, although the number of rows per time period grows a lot over the 6 years of data.
kat_crichton-seager
09/10/2021, 3:57 PMI'd still have to run the pipeline about 1000 times to get all the 100 million rows of data in anyway!
kat_crichton-seager
09/10/2021, 3:58 PMWe used to use Stitch to pull this table successfully. I'm wondering how they got around it 🙂
kat_crichton-seager
09/10/2021, 4:01 PMI've made a similar branch that limits the number of rows selected, rather than restricting to a time period - it works better for my use-case and I can set it to one value for Heroku and another for my laptop, where 80 GB spare space still isn't enough.
kat_crichton-seager
09/20/2021, 4:07 PMFor the benefit of anyone who might have found this thread, another probably easier way to reduce the disk space used is to set the
batch_size_rows