jake
07/27/2023, 6:39 PMversion: 1
send_anonymous_usage_stats: false
project_id: tap-pardot
default_environment: dev
environments:
- name: dev
config:
plugins:
extractors:
- name: tap-pardot
load_schema: ${SCHEMA_NAME}
config:
client_key: ${SFDC_CLIENT_KEY_DEV}
client_secret: ${SFDC_CLIENT_SECRET_DEV}
refresh_token: ${SFDC_REFRESH_TOKEN_DEV}
API_URL: <https://pi.demo.pardot.com>
SFDC_REDIRECT_URI: '<https://biz.sandbox.salesforce.com/services/oauth2/callback>'
SFDC_OAUTH_URI: '<https://test.salesforce.com/services/oauth2/token>'
pardot_business_unit_id_na: ${PARDOT_BUSINESS_UNIT_ID_NA_DEV}
start_date: '2023-07-25T07:08:47-07:00'
SCHEMA_NAME: 'pardot5_dev'
my stream works if i have this:
class ProspectsStream(pardotStream):
""" Custom stream based on v5 pardot api.
"""
name = "prospects"
fields_list = [
"email",
"id",
"createdAt",
"updatedAt",
"createdById",
"updatedById",
]
fields = ",".join(fields_list)
path = f"/{name}?fields={fields}"
primary_keys = ["id"]
replication_key = "updatedAt"
replication_method = "incremental"
is_sorted = True
schema_filepath = SCHEMAS_DIR / "prospects.json"
but it gives me data that is before my meltano.yml start_date value, for example "updatedAt": "2023-05-04T00:53:31-07:00",
i've tried to comment out replication_method, is_sorted, and have tried to add orderBy=createdAt desc and asc to the path but i get an error