Hi, Just getting started with Meltano. Want to cle...
# troubleshootingc
chintan_patel
06/20/2023, 12:20 PMHi, Just getting started with Meltano. Want to clear couple of doubts.
• How extractors passing data to loaders ?
• Can I configure no of records per batch ?
• Does Meltano works well with large amount of data?
u
user
06/20/2023, 1:47 PM@chintan_patel welcome! Meltano uses taps and targets that implement the Singer spec (see these docs for all the details https://hub.meltano.com/singer/spec). It depends what large amounts of data is but I'd say the answer is yes. The Meltano SDK has also been implementing a new message type called BATCH that tries to avoid passing data via stdout if the source/destination system has a more efficient bulk load capability, see https://meltano.com/blog/6x-more-speed-for-your-data-pipelines-with-batch-messages/ and https://sdk.meltano.com/en/latest/batch.html
u
user
06/20/2023, 1:49 PM• Can I configure no of records per batch ?What do you mean by this question? Meltano uses a plugin architecture that runs Singer taps and targets, so the implementation of each is slightly different depending on the system it interacts with. Do you have a certain use case in mind youre thinking of?
c
chintan_patel
06/21/2023, 5:13 AMIs it possible to configure the no of records in single pass from extractor to loader?
chintan_patel
06/21/2023, 5:14 AMI want to use Shopify as extractor and Redshift as loader in my case
u
user
06/21/2023, 2:24 PMAre you looking to limit the amount of data that gets synced? Theres not really a way to say "sync 100 records" but you can use the start_date config option that most taps implement (depending on if the source has support for filtering) to only sync data from something like yesterday or 4 hours ago. For example in our internal meltano project we run all EL pipelines in CI tests on a small subset of data, anywhere from 2 hrs of data to 1 day, see https://github.com/meltano/squared/blob/f5fafbf680a48dca1df8a41273f85c99b5832213/data/environments/cicd.meltano.yml#L6. Theres also an open issue in the SDK to support syncing a configurable subset of data https://github.com/meltano/sdk/issues/1333 for testing but its not implemented yet
u
user
06/21/2023, 2:25 PMDoes that help answer your question? If not, please share more about what you mean by limiting and what youre trying to achieve.
c
chintan_patel
06/22/2023, 4:46 AMYes, you answered my question. Thank you!