akmal
09/18/2023, 8:48 AMmeltano run there's no error. However I couldn't see tha state file in s3. I'm using meltano version 3.0.0 . I dockerize the project files based on this using image meltano/meltano:v3.0.0 https://docs.meltano.com/guide/containerization and run. Here's my `meltano.yaml`:
version: 1
default_environment: dev
project_id: 82c0c0c9-9249-4fbb-adea-2d3f2c6bb210
environments:
- name: dev
- name: staging
- name: prod
state_backend:
uri: <s3://my-bucket/meltano/state>
plugins:
extractors:
- name: tap-facebook--my
inherit_from: tap-facebook
variant: singer-io
pip_url: git+<https://github.com/singer-io/tap-facebook.git>
config:
start_date: '2023-09-18'
account_id: ''
include_deleted: true
insights_buffer_days: 1
select:
- ads_insights.*
loaders:
- name: target-bigquery--facebook-my
inherit_from: target-bigquery
variant: z3z1ma
pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
config:
credentials_path: .config/cred.json
project: project
dataset: dataset
denormalized: true
method: batch_job
upsert: true
dedupe_before_upsert: true
and when i see the logs i could see there's no error:
2023-09-18T08:40:34.505263Z [info ] smart_open.s3.MultipartWriter('my-bucket', 'meltano/state/dev:tap-facebook--my-to-target-bigquery--facebook-my/lock'): uploading part_num: 1, 16 bytes (total 0.000GB)
2023-09-18T08:40:34.696251Z [info ] No state found for dev:tap-facebook--my-to-target-bigquery--facebook-my
This is logs at the end of the run:
2023-09-18T08:42:19.250693Z [info ] 2023-09-18 08:42:19,250 | INFO | target-bigquery | Target 'target-bigquery' completed reading 883 lines of input (880 records, (0 batch manifests, 2 state messages). cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-c2b-my
2023-09-18T08:42:19.251212Z [info ] 2023-09-18 08:42:19,250 | INFO | target-bigquery | [f0453f7fe38b4e83957d55dfb937ac56] Loaded 100238 bytes into data-298903.meltano_ingestion.ads_insights__1695026440. cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-my
2023-09-18T08:42:32.423594Z [info ] Block run completed. block_type=ExtractLoadBlocks err=None set_number=0 success=Trueakmal
09/18/2023, 9:48 AM2023-09-18T09:46:30.678941Z [info ] File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/core.py", line 973, in __del__ cmd_type=elb consumer=True name=target-bigquery--facebook-c2b-my producer=False stdio=stderr string_id=target-bigquery--facebook-c2b-my
2023-09-18T09:46:30.680957Z [info ] self.close() cmd_type=elb consumer=True name=target-bigquery--facebook-c2b-my producer=False stdio=stderr string_id=target-bigquery--facebook-c2b-my
2023-09-18T09:46:30.681335Z [info ] File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/core.py", line 945, in close cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-my
2023-09-18T09:46:30.681676Z [info ] self._buffer.flush() cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-my
2023-09-18T09:46:30.682110Z [info ] ValueError: flush of closed file cmd_type=elb consumer=True name=target-bigquery--facebook-my producer=False stdio=stderr string_id=target-bigquery--facebook-my
and upon running the job with debug mode the state is empty:
2023-09-18T14:09:23.623233Z [info ] {"type": "STATE", "value": {}} cmd_type=elb consumer=False name=tap-facebook--my producer=True stdio=stdout string_id=tap-facebook--my
This was working nicely before. Anyone can help?akmal
09/19/2023, 1:04 PM