Couple of questions:
1. How do you keep track of t...
# troubleshootingh
haris_rashid
10/08/2023, 5:16 AMCouple of questions:
1. How do you keep track of the progress of a run?
2. How do I know much data is already available that needs to be moved? The job size? Size in GB that needs moving right from Mentalo?
haris_rashid
10/08/2023, 6:46 AM@edgar_ramirez_mondragon Let me know when you're available. Its a follow up to https://meltano.slack.com/archives/C01TCRBBJD7/p1696667903768239
e
edgar_ramirez_mondragon
10/09/2023, 2:37 PMHow do you keep track of the progress of a run?Logs should help with this. Though that will only tell you, for example, what stream is currently being synced and not how many streams or records are pending to be processed.
How do I know much data is already available that needs to be moved? The job size? Size in GB that needs moving right from Mentalo?I'm not sure I understand the question. The size of the data to be moved is an unknown for Meltano (and the Singer spec) because there's no way to ask the source for that information before extracting data, and computing it afterwards is not straightforward because the size depends largely on the serialization format 🤷🏼