@mani The fact that you're not seeing any log messages coming from

tap-ga360

suggests that the catalog file Meltano is generating doesn't contain any streams marked as

selected

, since we would otherwise see a

Syncing stream: <stream>

message for each selected stream: https://github.com/singer-io/tap-ga360/blob/master/tap_ga360/__init__.py#L20 Do you see any "Selected attributes" when you run

meltano select --list tap-ga360

?

Thanks @douwe_maan, I am new to python,singer,meltano, what should i do to proceed further. Please advise, Here is the o/p

@mani Ah, I think we've found the issue then! The tap does actually support catalog discovery (the

--discover

flag): https://github.com/singer-io/tap-ga360/blob/master/tap_ga360/__init__.py#L77, but Meltano currently thinks it doesn't (as indicated by your error message), so it's not actually discovering and generating a catalog, and not passing a

--catalog

flag to the tap, so

parsed_args.catalog

is unset, and the tap never makes it to this line: https://github.com/singer-io/tap-ga360/blob/master/tap_ga360/__init__.py#L79 To fix this, we should tell Meltano that your custom tap actually does support the

discover

and

catalog

capabilities

, as well as

state

. Can you add them to your plugin definition in

meltano.yml

and try running

meltano select --list tap-ga360

again?

Thanks, I’ve added, but I am getting some other errors`TION_PATH': 'output', 'MELTANO_LOAD_DESTINATION_PATH': 'output', 'TARGET_JSONL_DO_TIMESTAMP_FILE': 'true', 'MELTANO_LOAD_DO_TIMESTAMP_FILE': 'true', 'TARGET_JSONL__DIALECT': 'target_jsonl', 'MELTANO_LOAD__DIALECT': 'target_jsonl'}` tap-ga360     | INFO Syncing stream:ga_sessions tap-ga360 (out)  | {“type”: “SCHEMA”, “stream”: “ga_sessions”, “schema”: {“type”: [“null”, “object”], “properties”: {“visitNumber”: {“type”: [“null”, “number”]}, “customDimensions”: {“type”: [“null”, “array”], “items”: {“type”: [“null”, “object”], “properties”: {“value”: {“type”: [“null”, “string”]}, “index”: {“type”: [“null”, “number”]}}}}, “userId”: {“type”: [“null”, “string”]}, “clientId”: {“type”: [“null”, “string”]}, “visitId”: {“type”: [“null”, “number”]}, “date”: {“type”: [“null”, “string”], “format”: “date-time”}, “visitStartTime”: {“type”: [“null”, “number”]}, “totals”: {“type”: [“null”, “object”], “properties”: {“timeOnSite”: {“type”: [“null”, “number”]}, “newVisits”: {“type”: [“null”, “number”]}, “uniqueScreenViews”: {“type”: [“null”, “number”]}, “transactionRevenue”: {“type”: [“null”, “number”]}, “hits”: {“type”: [“null”, “number”]}, “screenviews”: {“type”: [“null”, “number”]}, “bounces”: {“type”: [“null”, “number”]}, “sessionQualityDim”: {“type”: [“null”, “number”]}, “totalTransactionRevenue”: {“type”: [“null”, “number”]}, “pageviews”: {“type”: [“null”, “number”]}, “timeOnScreen”: {“type”: [“null”, “number”]}, “visits”: {“type”: [“null”, “number”]}, “transactions”: {“type”: [“null”, “number”]}}}, “fullVisitorId”: {“type”: [“null”, “string”]}, “socialEngagementType”: {“type”: [“null”, “string”]}, “visitorId”: {“type”: [“null”, “number”]}, “device”: {“type”: [“null”, “object”], “properties”: {“browser”: {“type”: [“null”, “string”]}, “mobileInputSelector”: {“type”: [“null”, “string”]}, “mobileDeviceBranding”: {“type”: [“null”, “string”]}, “browserSize”: {“type”: [“null”, “string”]}, “language”: {“type”: [“null”, “string”]}, “screenResolution”: {“type”: [“null”, “string”]}, “mobileDeviceMarketingName”: {“type”: [“null”, “string”]}, “operatingSystemVersion”: {“type”: [“null”, “string”]}, “browserVersion”: {“type”: [“null”, “string”]}, “deviceCategory”: {“type”: [“null”, “string”]}, “flashVersion”: {“type”: [“null”, “string”]}, “mobileDeviceModel”: {“type”: [“null”, “string”]}, “javaEnabled”: {“type”: [“null”, “boolean”]}, “screenColors”: {“type”: [“null”, “string”]}, “operatingSystem”: {“type”: [“null”, “string”]}, “mobileDeviceInfo”: {“type”: [“null”, “string”]}}}, “channelGrouping”: {“type”: [“null”, “string”]}, “geoNetwork”: {“type”: [“null”, “object”], “properties”: {“subContinent”: {“type”: [“null”, “string”]}, “metro”: {“type”: [“null”, “string”]}, “region”: {“type”: [“null”, “string”]}, “cityId”: {“type”: [“null”, “string”]}, “city”: {“type”: [“null”, “string”]}, “longitude”: {“type”: [“null”, “string”]}, “networkLocation”: {“type”: [“null”, “string”]}, “continent”: {“type”: [“null”, “string”]}, “networkDomain”: {“type”: [“null”, “string”]}, “latitude…

Great! It definitely looks like some extraction is happening now 🙂

Thanks, it is awesome product..

@mani I'm glad you're liking it! Let me know if I can clear up anything else

@douwe_maan, How can i generate tap.config.json for meltano invoke?

@mani There's no need to manually generate it! Per https://meltano.com/docs/configuration.html:

Meltano is responsible for managing the configuration of all of a project's plugins. It knows what settings are supported by each plugin, and how and when different types of plugins expect to be fed that configuration.

Since this also goes for extractors and loaders, you do not need to manually craft the `config.json` files expected by Singer taps and targets, because Meltano will generate them on the fly whenever an extractor or loader is used through

meltano elt

or

meltano invoke

.

can we have option to store the auto generated file for invoke?

Dumping the contents of the generated config files is actually a new feature I'm just about to release in v1.51.0: https://gitlab.com/meltano/meltano/-/issues/2355

and it is getting deleted when the execution has done rite, i was thinking if i have any option to store generated file and change directly into the file and pass as elt config, which make life easier instead of running command line for changing in the meta data and schemas

@mani The issue is that the unescaped

*

in your command invocation is interpreted by your shell as "all files in this directory", so it's passing all of the filenames to

meltano config

instead of the literal string

*

. If you'd like to actually send a literal

*

, wrap it in quotes:

meltano config tap-ga360 set _metadata trafficSource "*" inclusion available

ooh Got it let me try

Instead of using

meltano config tap-ga360 set _metadata

, you can also directly write your desired rules in

meltano.yml

like in the example here: https://meltano.com/docs/plugins.html#in-meltano-yml-3

the metadata is working fine

Perfect! Then it should also work correctly once you run

meltano elt

but i could see all the fields in the snowflake

If you run

meltano --log-level=debug elt ...

and look at the

SCHEMA

messages tap-ga360 is outputting, do they contain additional fields compared to the

RECORD

messages?

message has been deleted

Can you drop the table in Snowflake and run the extraction again?

okay

I expect that those other fields were created earlier, before the tap was correctly configured to skip the fields you don't want

no luck, i dropped the table

Could it be that the tap is respecting the

selected

metadata when building

RECORD

messages, but ignoring it when building

SCHEMA

messages, which thus always include all attributes/columns?

yes, it is not having any values but fields were created

I just double checked, and the Singer spec doesn't actually state that properties that aren't selected should be excluded from the

SCHEMA

messages, so while this feels like a bug in https://github.com/singer-io/tap-ga360, it isn't technically one, and you'll probably find the same behavior in most if not all other taps: the unselected columns are created in the DB, even though they're never used

ill remove in the schema unused fields

It looks like the tap always loads the full schema from the embedded JSON files, though: https://github.com/singer-io/tap-ga360/blob/master/tap_ga360/streams.py#L49

ill try that,

I think we'll need to modify https://github.com/singer-io/tap-ga360/blob/master/tap_ga360/streams.py#L49 to get the schema from the provided catalog to take priority over the embedded schema

ooh, ill see if i can do it directly the schema folders if u guys piled up with other priorities .

https://github.com/singer-io/tap-ga360 is not maintained by us, so fixing bugs (or improving its behavior) is not a priority for us. The idea is that we can focus fully on making Meltano the best possible Singer runner, so that users can focus on improving the individual Singer taps and targets 🙂

ooh i see, thats ok,ill it do it my local machine

Meltano is a GitLab project, so we've got plenty of investment already 🙂 The plan is to continue to focus solely on the open source version for the foreseeable future (measured in years, not months), and to then eventually figure out a business model around the open source version, just like we did with GitLab itself.

Got it , Thanks and you guys are doing awesome job..

@mani Are you planning to set up Meltano with Airflow or another orchestrator? Notifications are the orchestrator's responsible, in our view, and Airflow (etc) have their own functionality for that 🙂

Perfect,ill check those tools tomorrow

Sounds great!

@douwe_maan, i am getting the below error

@mani The version of Airflow that Meltano currently installs doesn't support Python 3.8 yet: https://gitlab.com/meltano/meltano/-/issues/1956#note_413781207 Would it be an option to use Python 3.6 or 3.7 instead, until we resolve that issue and officially support 3.8?

does meltano runs 3.7?

3.7 is supported, yeah

thanks

@mani Did you add the default Meltano DAG generator (https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py) to your

orchestrate/dags

directory using

meltano add files airflow

?

yes

Have you defined schedules using

meltano schedule

, that will show up in

meltano.yml

under

schedules

?

let me define the schedule i guess i missed it

Per https://meltano.com/docs/orchestration.html#using-an-existing-airflow-installation:

This setup assumes you'll use

meltano schedule

to schedule your

meltano elt

pipelines, as described above, since the DAG generator iterates over the result of

meltano schedule list --format=json

and creates DAGs for each.

meltano schedule loadGA360 tap-ga360 target-snowflake @once

Traceback (most recent call last):  File “/usr/local/bin/meltano”, line 8, in <module>   sys.exit(main())  File “/usr/local/lib/python3.6/site-packages/meltano/cli/__init__.py”, line 38, in main   cli(obj={“project”: None})  File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 764, in call   return self.main(*args, **kwargs)  File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 717, in main   rv = self.invoke(ctx)  File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1137, in invoke   return _process_result(sub_ctx.command.invoke(sub_ctx))  File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1137, in invoke   return _process_result(sub_ctx.command.invoke(sub_ctx))  File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 956, in invoke   return ctx.invoke(self.callback, **ctx.params)  File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 555, in invoke   return callback(*args, **kwargs)  File “/usr/local/lib/python3.6/site-packages/click/decorators.py”, line 17, in new_func   return f(get_current_context(), *args, **kwargs)  File “/usr/local/lib/python3.6/site-packages/meltano/cli/schedule.py”, line 54, in add   session, name, extractor, loader, transform, interval, start_date  File “/usr/local/lib/python3.6/site-packages/meltano/core/schedule_service.py”, line 57, in add   session, extractor  File “/usr/local/lib/python3.6/site-packages/meltano/core/schedule_service.py”, line 93, in default_start_date   return iso8601_datetime(start_date) or datetime.utcnow()  File “/usr/local/lib/python3.6/site-packages/meltano/core/utils/__init__.py”, line 234, in iso8601_datetime   raise ValueError(f”{d} is not a valid UTC date.“) ValueError: 20200911 is not a valid UTC date.

Does formatting the start_date as

2020-09-11

fix this?

@douwe_maan, i need a another help regarding tap mongo and target snowflake

@mani That suggests that the target is receiving an invalid (or unsupported)

SCHEMA

message from the tap. Can you enable debug mode (

meltano --log-level=debug elt ...

), so that we can see the exact

SCHEMA

message that triggered this error?

@mani The problem is that the schema for these two properties is invalid, since they lack a

"type"

key:

"event_population_filters": { "anyOf": [{}] },
"notif_roles": { "anyOf": [{}] },

The other properties also have an empty object at the end of their

anyOf

arrays, which is also invalid, but is ignored by the target since they define at least one type:

"start_date": {"anyOf": [{"type": "string", "format": "date-time"}, {}]}

tap-mongodb==2.0.0

I suggest explicitly specifying a projection (https://docs.mongodb.com/manual/tutorial/project-fields-from-query-results/) as described under https://github.com/singer-io/tap-mongodb#edit-catalog-file, so that only the specific fields you need are queried from MongoDB. Inside the

metadata

object, it'd look like:

"tap-mongodb.projection": "{\"_id\": 1, \"title\": 1}"

(It needs to be an escaped JSON string per https://github.com/singer-io/tap-mongodb/blob/f0fcc01e679acb40689b952d418425500fd17612/tap_mongodb/__init__.py#L239)

yes i tried first and then now i am using passing the catalog file

I've created an issue for this: https://gitlab.com/meltano/meltano/-/issues/2388