Hi, I am trying to run a hubspot-tap to bigquery-target. I get an architecture compatibility error as shown below. Has anyone else faced this issue? Could you advise a big-query variant that runs on M1 chip? I have configured the credentials_path, project, dataset, location and batch size in z3z1ma variant. Appreciate any help or information. Thanks

2023-05-12T20:54:15.999725Z [info     ]     from .orjson import *      cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-05-12T20:54:15.999809Z [info     ] ImportError: dlopen(/Users/swetaprabha/Desktop/Analytics/meltano/hubspot-bigquery-poc/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/orjson/orjson.cpython-310-darwin.so, 0x0002): tried: '/Users/swetaprabha/Desktop/Analytics/meltano/hubspot-bigquery-poc/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/orjson/orjson.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/swetaprabha/Desktop/Analytics/meltano/hubspot-bigquery-poc/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/orjson/orjson.cpython-310-darwin.so' (no such file), '/Users/swetaprabha/Desktop/Analytics/meltano/hubspot-bigquery-poc/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/orjson/orjson.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')) cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-05-12T20:54:16.064748Z [error    ] Loader failed
2023-05-12T20:54:16.065048Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Loader failed

@alexander_butler have you seen this error before?

No, this is abnormal. It looks like you are running Python with Rosetta and pip installed stuff and are now running non rosetta python if I had to guess @sweta_garai

I am running python on mac-M1, if that's what you meant by rosetta (not too familiar with this terminology). Is there a workaround? @alexander_butler

Thats okay 🙂 This is it for context https://support.apple.com/en-us/HT211861 For a quicker solution, I would suggest nuking your venv and recreating it in your current environment

Hi @alexander_butler. I installed rosetta on my system but I still get the same error. I am also attaching my tap-target configuration here. Could you take a look? Thanks and appreciate your help! ```(base) swetaprabha@Sweta-Prabha-Garai's-MacBook-Pro hubspot-bigquery-poc % meltano run tap-hubspot target-bigquery 2023-05-15T183507.274928Z [info ] Environment 'dev' is active 2023-05-15T183508.098737Z [warning ] No state was found, complete import. 2023-05-15T183508.641907Z [info ] time=2023-05-15 113508 name=tap-hubspot level=INFO message=tap-hubspot v0.0.1, Meltano SDK v0.4.9) cmd_type=elb consumer=False name=tap-hubspot producer=True stdio=stderr string_id=tap-hubspot 2023-05-15T183508.642365Z [info ] time=2023-05-15 113508 name=tap-hubspot level=INFO message=Skipping parse of env var settings... cmd_type=elb consumer=False name=tap-hubspot producer=True stdio=stderr string_id=tap-hubspot 2023-05-15T183508.642471Z [info ] time=2023-05-15 113508 name=tap-hubspot level=INFO message=Config validation passed with 0 warnings. cmd_type=elb consumer=False name=tap-hubspot producer=True stdio=stderr string_id=tap-hubspot 2023-05-15T183508.653712Z [info ] time=2023-05-15 113508 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. cmd_type=elb consumer=False name=tap-hubspot producer=True stdio=stderr string_id=tap-hubspot 2023-05-15T183508.998475Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery 2023-05-15T183508.998699Z [info ] File "/Users/swetaprabha/Desktop/Analytics/meltano/hubspot-bigquery-poc/.meltano/loaders/target-bigquery/venv/bin/target-bigquery", line 5, in <module> cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery 2023-05-15T183508.998851Z [info ] from target_bigquery.target import TargetBigQuery cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery 2023-05-15T183508.999015Z [info ] File "/Users/swetaprabha/Desktop/Analytics/meltano/hubspot-bigquery-poc/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/target_bigquery/target.py", line 21, in <module> cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery 2023-05-15T183508.999181Z [info ] from target_bigquery.batch_job import BigQueryBatchJobDenormalizedSink, BigQueryBatchJobSink cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery 2023-05-15T183508.999266Z [info ] File "/Users/swetaprabha/Desktop/Analytics/meltano/hubspot-bigquery-poc/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/target_bigquery/batch_job.py", line 22, in <module> cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery 2023-05-15T183508.999525Z [info ] import orjson cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery 2023-05-15T183508.999711Z [info ] File "/Users/swetaprabha/Desktop/Analytics/meltano/hubspot-bigquery-poc/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/orjson/__init__.py", line 3, in <module> cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery 2023-05-15T183508.999798Z [info ] from .orjson import * cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery

Sorry I must have miscommunicated. The original issue seems to be python running in x86_64 arch and pip installing arm64 The fact you are on an M1 mac means you should never have pip installing or creating a x86_64 wheel. So it was my assumption rosetta is getting involved somewhere which is not ideal. I have never had this issue (Also on M1 mac) so I don't think its target bigquery related. So the path I was hoping to steer you down was figuring out if any of your stuff is running in rosetta without you realizing it.

Hi @alexander_butler, I tried running inside docker container. For BQ configuration, does dataset means schema.table or just table? My target table id is in the format project.dataset.table. Can you validate if this is the right configuration?

loaders:
  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
    config:
      credentials_path: /Users/admin/Desktop/Analytics/meltano/hubspot-bigquery-poc/credential.json
      project: myproject
      dataset: properties.contacts
      location: us-west1
      batch_size: 500

Hi! waving wave So 'dataset' in bigquery is just another word for 'schema' in a tradition DBMS. So the table name should not be included. That will be derived automatically from the stream name. For example in Hubspot you may have a Company stream, Lead stream, Property stream. These would get routed to a company, lead, and property table in the dataset defined in the

dataset

config.

Thank you for the info 🙂 I have another problem with docker. I get a ModuleNotFound error. Could you advice? Thanks once again

(base) swetaprabha@Sweta-Prabha-Garai's-MacBook-Pro hubspot-bigquery-poc % docker run hubspot-bigquery-poc run tap-hubspot target-bigquery
2023-05-16T21:30:26.419310Z [info   ] Environment 'dev' is active
2023-05-16T21:30:27.079884Z [warning ] No state was found, complete import.
2023-05-16T21:30:27.354922Z [error  ] Cannot start plugin tap-hubspot: Catalog discovery failed: command ['/project/.meltano/extractors/tap-hubspot/venv/bin/tap-hubspot', '--config', '/project/.meltano/run/tap-hubspot/tap.b9f21d14-9ac6-4ae9-b462-f1b219984d73.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
 File "/project/.meltano/extractors/tap-hubspot/venv/bin/tap-hubspot", line 5, in <module>
  from tap_hubspot.tap import TapHubspot
 File "/project/.meltano/extractors/tap-hubspot/venv/lib/python3.9/site-packages/tap_hubspot/tap.py", line 9, in <module>
  from tap_hubspot.streams import (
 File "/project/.meltano/extractors/tap-hubspot/venv/lib/python3.9/site-packages/tap_hubspot/streams.py", line 4, in <module>
  import singer
ModuleNotFoundError: No module named 'singer'

2023-05-16T21:30:27.355360Z [error  ] Block run completed.       block_type=ExtractLoadBlocks err=RunnerError('Cannot start plugin tap-hubspot: Catalog discovery failed: command [\'/project/.meltano/extractors/tap-hubspot/venv/bin/tap-hubspot\', \'--config\', \'/project/.meltano/run/tap-hubspot/tap.b9f21d14-9ac6-4ae9-b462-f1b219984d73.config.json\', \'--discover\'] returned 1 with stderr:\n Traceback (most recent call last):\n File "/project/.meltano/extractors/tap-hubspot/venv/bin/tap-hubspot", line 5, in <module>\n  from tap_hubspot.tap import TapHubspot\n File "/project/.meltano/extractors/tap-hubspot/venv/lib/python3.9/site-packages/tap_hubspot/tap.py", line 9, in <module>\n  from tap_hubspot.streams import (\n File "/project/.meltano/extractors/tap-hubspot/venv/lib/python3.9/site-packages/tap_hubspot/streams.py", line 4, in <module>\n  import singer\nModuleNotFoundError: No module named \'singer\'\n') exit_codes={} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Cannot start plugin tap-hubspot: Catalog discovery failed: command ['/project/.meltano/extractors/tap-hubspot/venv/bin/tap-hubspot', '--config', '/project/.meltano/run/tap-hubspot/tap.b9f21d14-9ac6-4ae9-b462-f1b219984d73.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
 File "/project/.meltano/extractors/tap-hubspot/venv/bin/tap-hubspot", line 5, in <module>
  from tap_hubspot.tap import TapHubspot
 File "/project/.meltano/extractors/tap-hubspot/venv/lib/python3.9/site-packages/tap_hubspot/tap.py", line 9, in <module>
  from tap_hubspot.streams import (
 File "/project/.meltano/extractors/tap-hubspot/venv/lib/python3.9/site-packages/tap_hubspot/streams.py", line 4, in <module>
  import singer
ModuleNotFoundError: No module named 'singer'

Hey @sweta_garai - we are aware of this bug and will be working to address it. Sorry for the inconvenience