tap-duckdb send a chunk of rows in each exactly 1 ...
# troubleshooting
r
tap-duckdb send a chunk of rows in each exactly 1 minute, how can i change this behavior?
e
Hey @Ricardo Celso! That minute between metric messages is not how often chunks are extracted from the tap, only how often the metric is emitted.
Changing that is not possible for end users, and also would not affect the operation of the tap in any way.
Said another way: it's not controlling how fast records are extracted, rather measuring how fast records are extracted.
r
"measuring how fast records are extracted" to sys or to target?
e
Extracted from the source. Records are emitted so stdout and soon as they're generated, and the target reads them from stdin as soon as they're encountered too. The latter may batch records before writing them, though.
👍🏼 1
r
understood, let me explain my problem in more details I'm performing an ingestion with tap-duckdb to target-jsonl, however encountering problems to increase throughput. If i run the query on duckdb isolated i got a million rows in 5s seconds but when i try tap-duckdb to target-jsonl i got 85k per minute any idea of what could be?
e
Could be a few things. The duckdb sqlalchemy adapter slowing things down, backpressure from the target not being able to process records faster.
👍🏼 1