Hello everyone, I have set s3 as state backend uri...
# troubleshooting
a
Hello everyone, I have set s3 as state backend uri. when i run
meltano run
there's no error. However I couldn't see tha state file in s3. I'm using meltano version
3.0.0
. I dockerize the project files based on this using image meltano/meltano:v3.0.0 https://docs.meltano.com/guide/containerization and run. Here's my `meltano.yaml`:
Copy code
version: 1
default_environment: dev
project_id: 82c0c0c9-9249-4fbb-adea-2d3f2c6bb210
environments:
- name: dev
- name: staging
- name: prod
state_backend:
  uri: <s3://my-bucket/meltano/state>
plugins:
  extractors:
  - name: tap-facebook--my
    inherit_from: tap-facebook
    variant: singer-io
    pip_url: git+<https://github.com/singer-io/tap-facebook.git>
    config:
      start_date: '2023-09-18'
      account_id: ''
      include_deleted: true
      insights_buffer_days: 1
    select:
    - ads_insights.*
  loaders:
  - name: target-bigquery--facebook-my
    inherit_from: target-bigquery
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
    config:
      credentials_path: .config/cred.json
      project: project
      dataset: dataset
      denormalized: true
      method: batch_job
      upsert: true
      dedupe_before_upsert: true
and when i see the logs i could see there's no error:
Copy code
2023-09-18T08:40:34.505263Z [info     ] smart_open.s3.MultipartWriter('my-bucket', 'meltano/state/dev:tap-facebook--my-to-target-bigquery--facebook-my/lock'): uploading part_num: 1, 16 bytes (total 0.000GB)
2023-09-18T08:40:34.696251Z [info     ] No state found for dev:tap-facebook--my-to-target-bigquery--facebook-my
This is logs at the end of the run:
Copy code
2023-09-18T08:42:19.250693Z [info     ] 2023-09-18 08:42:19,250 | INFO     | target-bigquery      | Target 'target-bigquery' completed reading 883 lines of input (880 records, (0 batch manifests, 2 state messages). cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-c2b-my
2023-09-18T08:42:19.251212Z [info     ] 2023-09-18 08:42:19,250 | INFO     | target-bigquery      | [f0453f7fe38b4e83957d55dfb937ac56] Loaded 100238 bytes into data-298903.meltano_ingestion.ads_insights__1695026440. cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-my
2023-09-18T08:42:32.423594Z [info     ] Block run completed.           block_type=ExtractLoadBlocks err=None set_number=0 success=True
sometimes i would get this error on the target
Copy code
2023-09-18T09:46:30.678941Z [info     ]   File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/core.py", line 973, in __del__ cmd_type=elb consumer=True name=target-bigquery--facebook-c2b-my producer=False stdio=stderr string_id=target-bigquery--facebook-c2b-my
2023-09-18T09:46:30.680957Z [info     ]     self.close()               cmd_type=elb consumer=True name=target-bigquery--facebook-c2b-my producer=False stdio=stderr string_id=target-bigquery--facebook-c2b-my
2023-09-18T09:46:30.681335Z [info     ]   File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/core.py", line 945, in close cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-my
2023-09-18T09:46:30.681676Z [info     ]     self._buffer.flush()       cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-my
2023-09-18T09:46:30.682110Z [info     ] ValueError: flush of closed file cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-my
and upon running the job with debug mode the state is empty:
Copy code
2023-09-18T14:09:23.623233Z [info     ] {"type": "STATE", "value": {}} cmd_type=elb consumer=False name=tap-facebook--my producer=True stdio=stdout string_id=tap-facebook--my
This was working nicely before. Anyone can help?
It's because i forgot to add base plugins 😅