wave from paperless post we recently learned from google th Meltano #announcements

:wave: from paperless post. we recently learned fr...

suraj

05/19/2020, 10:30 PM

👋 from paperless post. we recently learned from google that they are deprecating alooma by end of the year. we currently use bigquery for our data warehouse. our alooma data sources were postgres, mongodb, and zendesk. does meltano support bigquery as a target? but so far i am excited by meltano as a potential option to replace alooma for our data replication needs

douwe_maan

05/19/2020, 11:04 PM

@suraj Welcome! Meltano intends to support all Singer taps and targets, and it looks like 2 different targets for BigQuery have already been built by members of the community: https://github.com/RealSelf/target-bigquery https://github.com/jmriego/pipelinewise-target-bigquery I can't speak to their quality or current support in Meltano (if you add either as a custom plugin), but if you give it a try and run into any trouble, I'll be here to help you out and get it working!

suraj

05/19/2020, 11:09 PM

Thanks for sharing @douwe_maan - I'll take a look at these. I'll def reach out as I start to test if I run into problems

suraj

05/20/2020, 4:47 PM

hey @douwe_maan i am attempting to add a custom extractor and getting the below error. any ideas?

meltano add --custom loader target-bigquery

suraj

05/20/2020, 4:48 PM

Copy code

(meltano) schokshi-2197:paperless_meltano schokshi$ meltano add --custom loader target-bigquery
The `namespace` refers to the data source name.
It is used to infer compatibility between components.

(namespace): bigquery

The `pip URL` refers to the pip installation specification.
You may provide any pip-compatible entry:
	from VCS: git+<https://gitlab.com/meltano/tap-carbon-intensity.git>
	from PyPI: tap-something
	from Source: [-e] /path/to/plugin/source

(pip_url): <https://github.com/jmriego/pipelinewise-target-bigquery.git>

The `executable` refers to the entry point of the plugin.
By convention, plugins should use their package name as executable, but it might be useful to override it to have multiple flavors of the same plugin installed.

(executable) [target-bigquery]: target-bigquery

The `settings` refer to the configuration keys the plugin supports.
In the case of Singer taps and targets, these correspond to supported keys in `config.json`.
Nesting can be represented using the `.` seperator, e.g. `auth.username` for `{ "auth": { "username": value } }`.
Multiple setting names/keys can be specified using a comma-separated string.

(settings) [[]]: config.json
[2020-05-20 12:44:55,802|W|MainThread|root] target-bigquery is already present, use `meltano install` to install it.
The plugin has been added to your `meltano.yml`.
If your plugin requires configuration options, you can add them directly in the created plugin definition under the `settings` or `config` section.
Added 'target-bigquery' to your Meltano project.
Installing 'target-bigquery'...
target-bigquery has an installation issue. Failed to install plugin 'target-bigquery'.
  ERROR: Cannot unpack file /private/var/folders/1l/dmj4h7xj6jb663wnmpjbrr6w6dpk4k/T/pip-unpack-r1stkkyk/pipelinewise-target-bigquery.git (downloaded from /private/var/folders/1l/dmj4h7xj6jb663wnmpjbrr6w6dpk4k/T/pip-req-build-sd7ps0w3, content-type: text/html; charset=utf-8); cannot detect archive format
ERROR: Cannot determine archive format of /private/var/folders/1l/dmj4h7xj6jb663wnmpjbrr6w6dpk4k/T/pip-req-build-sd7ps0w3
WARNING: You are using pip version 19.2.3, however version 20.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Aborted!

douwe_maan

05/20/2020, 4:57 PM

@suraj I think the issue is that you're missing the

git+

prefix on your

pip_url

pip3 install

won't handle the URL correctly unless it knows what VCS tool to use to clone the provided URL 🙂

suraj

05/20/2020, 5:16 PM

perfect that did it.

suraj

05/20/2020, 5:18 PM

for the last prompt, is this the path to config? I tried adding the path as well as the keys within th config.

Copy code

# Specify supported settings (`config.json` keys)
(settings):

when i run

meltano elt tap-zendesk target-bigquery

i get the error:

Copy code

(meltano) schokshi-2197:paperless_meltano schokshi$ meltano elt tap-zendesk target-bigquery
[2020-05-20 13:10:26,205|I|MainThread|alembic.runtime.migration] Context impl SQLiteImpl.
[2020-05-20 13:10:26,205|I|MainThread|alembic.runtime.migration] Will assume non-transactional DDL.
System database up-to-date.
Running extract & load...
No state was found, complete import.
INFO Using API Token authentication.
ERROR Invalid configuration:
   * Required key is missing from config: [project_id]
   * Neither 'default_target_schema' (string) nor 'schema_mapping' (object) keys set in config.
INFO tickets: Starting sync
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 5.647816896438599, "tags": {"status": "succeeded"}}
INFO Syncing ticket_audits per ticket...
INFO Starting metrics capture at 2020-05-20T17:10:35Z

douwe_maan

05/20/2020, 5:20 PM

What did you write on the

(settings):

prompt? Based on https://github.com/jmriego/pipelinewise-target-bigquery#configuration-settingsd, the minimal settings would be

dataset_id,project_id

douwe_maan

05/20/2020, 5:21 PM

If you also provided

temp_schema,default_target_schema

, etc, you'll want to make sure you edit

meltano.yml

to set their default values for these settings by adding

value

properties to the appropriate objects under

settings

douwe_maan

05/20/2020, 5:22 PM

Otherwise Meltano will just pass an empty string as value, instead of whatever default should actually be used

suraj

05/20/2020, 6:00 PM

got it to work @douwe_maan this looks great

suraj

05/20/2020, 6:01 PM

I will continue to test. thanks again

7 Views

Open in Slack

Previous Next