https://linen.dev logo
Join Slack
Powered by
Hello everyone, Anyone has an idea how to get Stat...
# troubleshooting
a
Hello everyone, Anyone has an idea how to get State management running for tap-salesforce ? I have been using the default extractor and using meltano run command like this after selecting some entities.
Copy code
meltano --environment=dev --log-level=debug run tap-salesforce target-jsonl
Log from the command line always complains :
Copy code
2023-07-06T12:07:25.364035Z [debug    ] /etc/timezone found, contents:
 Etc/UTC

2023-07-06T12:07:25.367277Z [debug    ] /etc/localtime found
2023-07-06T12:07:25.373185Z [debug    ] 2 found:
 {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
2023-07-06T12:07:25.388413Z [info     ] Environment 'dev' is active
2023-07-06T12:07:25.628032Z [debug    ] Creating engine '<meltano.core.project.Project object at 0z40021c1434>@sqlite:////project/my-project-meltano/.meltano/meltano.db'
2023-07-06T12:07:25.933064Z [debug    ] Found plugin parent            parent=tap-salesforce plugin=tap-salesforce source=<DefinitionSource.LOCKFILE: 8>
2023-07-06T12:07:25.936263Z [debug    ] found plugin in cli invocation plugin_name=tap-salesforce
2023-07-06T12:07:25.940460Z [debug    ] Found plugin parent            parent=target-jsonl plugin=target-jsonl source=<DefinitionSource.LOCKFILE: 8>
2023-07-06T12:07:25.943068Z [debug    ] found plugin in cli invocation plugin_name=target-jsonl
2023-07-06T12:07:25.943797Z [debug    ] head of set is extractor as expected block=<meltano.core.plugin.project_plugin.ProjectPlugin object at 0x4008497d90>
2023-07-06T12:07:26.680652Z [debug    ] found block                    block_type=loaders index=1
2023-07-06T12:07:26.681400Z [debug    ] blocks                         idx=1 offset=0
2023-07-06T12:07:27.027243Z [debug    ] Variable '$MELTANO_LOAD_SCHEMA' is not set in the provided env dictionary.
2023-07-06T12:07:27.032189Z [debug    ] found ExtractLoadBlocks set    offset=0
2023-07-06T12:07:27.032912Z [debug    ] All ExtractLoadBlocks validated, starting execution.
2023-07-06T12:07:28.015851Z [debug    ] Created configuration at /project/my-project-meltano/.meltano/run/tap-salesforce/tap.109c35b9-db3e-4f7d-9a85-2347cf9e9f62.config.json
2023-07-06T12:07:28.018045Z [debug    ] Could not find tap.properties.json in /project/my-project-meltano/.meltano/extractors/tap-salesforce/tap.properties.json, skipping.
2023-07-06T12:07:28.020021Z [debug    ] Could not find tap.properties.cache_key in /project/my-project-meltano/.meltano/extractors/tap-salesforce/tap.properties.cache_key, skipping.
2023-07-06T12:07:28.021372Z [debug    ] Could not find state.json in /project/my-project-meltano/.meltano/extractors/tap-salesforce/state.json, skipping.
2023-07-06T12:07:28.047809Z [warning  ] No state was found, complete import.
2023-07-06T12:07:28.051140Z [debug    ] Using cached catalog file
Any Idea what I am doing wrong ? Even after rerunning the same export multiple times it shows 
2023-07-06T12:07:28.047809Z [warning  ] No state was found, complete import.
From the documentation states
Meltano stores this pipeline state in its state backend, identified by the 
meltano run
State ID automatically generated based on the extractor name, loader name, and active environment name (see more about incremental state for elt).
Here’s a quick look at my 
meltano.yaml
Copy code
version: 1
default_environment: dev
project_id: my-project-id
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-salesforce
        select:
        - Task.*
- name: staging
- name: prod
send_anonymous_usage_stats: false
plugins:
  extractors:
  - name: tap-salesforce
    variant: meltanolabs
    pip_url: git+<https://github.com/meltanolabs/tap-salesforce.git@v1.5.1>
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: git+<https://github.com/andyhuynh3/target-s3-jsonl.git>
I use environment variables for the configs of both tap-salesforce and target-jsonl
Copy code
TAP_SALESFORCE_PASSWORD=my-password
TAP_SALESFORCE_START_DATE=2014-01-01T00:00:00Z
TAP_SALESFORCE_USERNAME=user@domain.com
TARGET_JSONL_DO_TIMESTAMP_FILE=false
TAP_SALESFORCE_SECURITY_TOKEN=security-token
e
Hey @abhishek_singh! Can you try running
Copy code
meltano state list
to check if any state was produced by the pipeline?
a
@edgar_ramirez_mondragon Yes looks like no state is being produced :
Copy code
>meltano state list

2023-07-10T11:08:46.318980Z [info     ] The default environment 'dev' will be ignored for `meltano state`. To configure a specific environment, please use the option `--environment=<environment name>`.
2023-07-10T11:08:46.639979Z [info     ] No state IDs found.
also tried using the environment but same result :
Copy code
>meltano --environment=dev state list
2023-07-10T11:13:20.879006Z [info     ] Environment 'dev' is active
2023-07-10T11:13:21.427841Z [info     ] No state IDs found.
@edgar_ramirez_mondragon any recommendations or suggestions on what I could be doing wrong?
@edgar_ramirez_mondragon another soft ping in case you have some advice for me
e
I can't see anything obviously wrong with your setup, so perhaps you can try a different combination of tap or target and see if state is still missing. Simple options could be tap-csv and target-sqlite.
Open in Slack
PreviousNext
Powered by