trying to understand a basic concept about cutoff ...
# troubleshooting
j
trying to understand a basic concept about cutoff dates - if i want to only pull data starting at a hardcoded value, how do i do that for all streams (using meltano.yml config), and how do i do that for just one (also using the value from meltano.yml)? my yaml looks like this:
Copy code
version: 1
send_anonymous_usage_stats: false
project_id: tap-pardot
default_environment: dev
environments:
  - name: dev
    config:
      plugins:
        extractors:
          - name: tap-pardot
            load_schema: ${SCHEMA_NAME}
            config:
              client_key: ${SFDC_CLIENT_KEY_DEV}     
              client_secret: ${SFDC_CLIENT_SECRET_DEV}
              refresh_token: ${SFDC_REFRESH_TOKEN_DEV}

              API_URL: <https://pi.demo.pardot.com>
              SFDC_REDIRECT_URI: '<https://biz.sandbox.salesforce.com/services/oauth2/callback>'
              SFDC_OAUTH_URI: '<https://test.salesforce.com/services/oauth2/token>'
              pardot_business_unit_id_na: ${PARDOT_BUSINESS_UNIT_ID_NA_DEV}
              start_date: '2023-07-25T07:08:47-07:00'
              SCHEMA_NAME: 'pardot5_dev'
my stream works if i have this:
Copy code
class ProspectsStream(pardotStream):
    """ Custom stream based on v5 pardot api.
    """
    name = "prospects"
    fields_list = [
        "email",
        "id",
        "createdAt",
        "updatedAt",
        "createdById",
        "updatedById",
    ]
    fields = ",".join(fields_list)
    path = f"/{name}?fields={fields}"
    primary_keys = ["id"]
    replication_key = "updatedAt"
    replication_method = "incremental"
    is_sorted = True
    schema_filepath = SCHEMAS_DIR / "prospects.json"
but it gives me data that is before my
meltano.yml
start_date
value, for example
"updatedAt": "2023-05-04T00:53:31-07:00",
i've tried to comment out
replication_method
,
is_sorted
, and have tried to add
orderBy=createdAt desc
and
asc
to the path but i get an error