Hi everyone 2 questions for ya'll: 1. I'm running into some issues with the dates in my meltano ingestion. My meltano.yml specifies

start_date: '2025-01-01'

but my ingestion continues migrating data from as far back as 2020

meltano.yml
version: 1
default_environment: prod
project_id: 019a0d29-8b5e-7c85-8e86-62fb6716507a
environments:
  - name: prod
plugins:
  extractors:
    - name: tap-freshcaller
      variant: hotgluexyz
      pip_url: git+<https://gitlab.com/hotglue/tap-freshcaller.git>
      config:
        flattening_enabled: true
        flattening_max_depth: 3
        org_name: legrandehealth
        start_date: '2025-01-01'
        timeout: 1200  # seconds
      select:
        # ---------- call_metrics table ----------
        - call_metrics.id
        - call_metrics.call_id
        - call_metrics.created_time
        - call_metrics.updated_time
        - call_metrics.ivr_time
        - call_metrics.ivr_time_unit
        - call_metrics.hold_duration
        - call_metrics.hold_duration_unit
        - call_metrics.call_work_time
        - call_metrics.call_work_time_unit
        - call_metrics.total_ringing_time
        - call_metrics.total_ringing_time_unit
        - call_metrics.talk_time
        - call_metrics.talk_time_unit
        - call_metrics.answering_speed
        - call_metrics.answering_speed_unit
        - call_metrics.recording_duration
        - call_metrics.recording_duration_unit
        - call_metrics.csat
        # ---------- calls table ----------
        - calls.id
        - calls.direction
        - calls.parent_call_id
        - calls.root_call_id
        - calls.phone_number
        - calls.phone_number_id
        - calls.assigned_agent_id
        - calls.assigned_agent_name
        - calls.assigned_team_id
        - calls.assigned_team_name
        - calls.call_notes
        - calls.created_time
        - calls.updated_time
        # - calls.recording
        # - calls.participants
        # ---------- users table ----------
        - users.id
        - users.name
        - users.email
        - users.phone
        - users.status
        - users.preference
        - users.last_call_time
        - users.last_seen_time
        - users.confirmed
        - users.deleted
        - users.role
        # attribute fields can be found here --> <https://developers.freshcaller.com/api/>

  loaders:
    - name: target-bigquery
      variant: z3z1ma
      pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
      config:
        project: ${GCP_PROJECT}
        dataset: ${BQ_DATASET}
        credentials_path: ${GOOGLE_APPLICATION_CREDENTIALS}
        method: storage_write_api
        location: US
        denormalized: true
        upsert: true
        add_record_metadata: true
        schema_evolution: true
        auto_schema_update: true
        timeout: 1200  # seconds

2. I would like both calls.recording and calls.participants migrated over (both currently commented out) but I keep getting schema mismatch type errors, even with

flattening_enabled: true

and

flattening_max_depth: 3

. Here's one of them:

google.protobuf.json_format.ParseError: Failed to parse participants field: repeated field participants must be in [] which is [{"id": 306508795, ...}, {"id": 360339471, ...}] at AnonymousProto_f22be015fa6a30c55da67bd32528945dd8805fd8. 
BrokenPipeError: [Errno 32] Broken pipe

1. from what i can see, this tap does not support start_date or incremental loads and that's because the underlying API doesn't support it. it could perhaps be rewritten to use the asynchronous account/export endpoint which does take date ranges 2. the schema defined in the tap appears to match what the docs say the response is. you could try hitting the API on your own to see if the response is what their docs say, or add a logging step just to see it within meltano like this

on class freshcallerStream

...

@override
def validate_response(self, response: requests.Response):
    self.logger.info(response.json())
    super().validate_response(response)

Ok thank you, I appreciate the response!