https://linen.dev logo
Join Slack
Powered by
I'm trying to correctly add a custom extractor/tap...
# announcements
a
I'm trying to correctly add a custom extractor/tap. I don't think I got it quite right the first time, so I want to remove it and try again. How can I remove an extractor?
I think I have it. Looks like a combination of removing from meltano.yml and removing associated dir from .meltano/extractors
b
Hey Adam. That’s what I was just about to send over
We plan on adding a remove command to simplify the process, but just haven’t gotten around to it just yet
a
thanks!
Is there a ticket created for the remove command?
m
Hi everyone! Is there already a remove command? I wasn't able to find it. I only can do it by doing a manual deletion.
d
@murilo_tavolaro_de_nigris There isn't yet, but I've just created a new issue: https://gitlab.com/meltano/meltano/-/issues/2353 For the moment, manually deleting the plugin from 
meltano.yml
is the way to go, but I'll gladly accept a contribution to add the new 
meltano remove
command 🙂
m
Thank you for the answer @douwe_maan! I'm working on my first meltano project, probably I'll put it in production over the next few days for a company I'm working with :) After that, let's see if I can contribute. I'm so thrilled that this platform exists and is being developed, hard to describe the feeling in words!!
d
@murilo_tavolaro_de_nigris I'm glad you're liking it! Let me know if I can clear up anything else 😄
m
@douwe_maan, Need one more favor, trying to do target-snowflake. I am getting the below error
CRITICAL (snowflake.connector.errors.ProgrammingError) 251001: Account must be specified
d
@mani It sounds like the 
account
setting isn't set correctly. Does it show when you run 
meltano config target-snowflake
?
m
yes it shows the all the config
d
@mani Can you share the 
target-snowflake
plugin definition from 
meltano.yml
? Feel free to remove any sensitive config, of course
m
version: 1
send_anonymous_usage_stats: true project_id: 6ae4b82a-f776-418d-b773-90598b8e3bde plugins: extractors: - name: tap-ga360 namespace: tap_ga360 pip_url: tap-ga360 executable: tap-ga360 capabilities: - catalog - discover - state config: start_date: ‘20200915’ project_id: peupnportal dataset_id: 228768660 service_account_json: ‘{..}’ loaders: - name: target-jsonl pip_url: target-jsonl config: do_timestamp_file: true - name: target-snowflake pip_url: git+https://gitlab.com/meltano/target-snowflake.git config: account: xxxx.us-east-1.snowflakecomputing.com database: xxxx role: LOADER schema: GA360 username: xxxx warehouse: xxxx
i used meltano UI for adding loader and configure the values.
but run the elt using command line
d
OK, that all looks correct, and if the 
account
shows up in 
meltano config target-snowflake
it should be making its way to the target's 
config.json
file as well.
The 
account
setting is making its way to 
snowflake_connector
through this line here: https://gitlab.com/meltano/target-snowflake/-/blob/master/target_snowflake/snowflake_loader.py#L113
@mani You might need a 
https://
on that account URL; can you try adding that?
From https://github.com/snowflakedb/snowflake-sqlalchemy/blob/master/util.py#L16, I'm wondering if the library is having trouble with the 
region
(
us-east-1
) specified as part of the 
account
, but I'd expect that to result in a connection failure rather than 
self._account
not being set at all in src/snowflake/connector/connection.py:730-736
What happens if you only specify your account identifier as 
account
, not the entire hostname?
m
@douwe_maan, Thanks it is working now. i can see data coming into Snowflake, Thanks a lot
d
@mani Awesome, congrats! 😄
m
How do i extract some specific fields from my source
Please share the documentation or demos/examples
d
m
yes, looks like more documentation rather than some examples, since i am new to ETL, it is hard for me to understand
ill go through it and ask you if have any doubts
d
Sounds good, especially the section in the getting started guide should be pretty easy to follow, but let me know if anything is unclear or not working as expected
m
Sure thanks
@douwe_maan, I have made selected fields what to extract from GA, but when i load into Snowflake ,all the fields are loading in to snow flake table
how could i load only selected field from extractor to the destination
d
@mani When you run 
meltano select --list <tap>
and look under "Selected attributes", do the listed attributes match the ones that end up in Snowflake?
m
the select behaves weird
when i do exclude meltano select tap-ga360 --exclude ga_sessions customDimensions
all the selected values are gone
At the same time , when i put only selected values , the loader target-json gets only selected value
but target-snowflake gets all the fields
little confusion
d
@mani That's very odd, the tap (should) output the same records independent of which target it's used with, so I can't explain why target-snowflake is getting more fields than target-jsonl. Can you try running 
meltano --log-level=debug elt
, so that we can see each 
RECORD
message the tap outputs? Do those contain the extra fields?
As for the exclusion weirdness, do you mean that after running 
meltano select tap-ga360 --exclude ga_sessions customDimensions
, 
meltano select --list tap-ga360
doesn't list any attributes anymore? Can you please share your plugin definition with the 
select
configuration, and the full output of 
meltano select --list tap-ga360
?
@mani If you like, we can jump on a Zoom call some time so I can help you get to the bottom of this 🙂
@mani Actually, you may be experiencing this bug in a different way than described in there: https://gitlab.com/meltano/meltano/-/issues/2348 If you delete 
.meltano/run/tap-ga360/tap.properties.json
before running 
meltano select tap-ga360 --list
, does the result change?
m
Sorry @douwe_maan, I was not on the seat.
(.venv) manigovindarajulu@Manis-MBP ga360 % meltano select --list tap-ga360
Legend: selected excluded automatic Enabled patterns: !ga_sessions.customDimensions Selected attributes: (.venv) manigovindarajulu@Manis-MBP ga360 %
After adding meltano select tap-ga360 --exclude ga_sessions customDimensions
I am fine with Zoom, Can we have on MOnday
d
@mani OK, what's happening is that the default selection pattern (
*.*
, to select all entities and attributes) gets overridden by the 
select
list in 
meltano.yml
the moment you add the first rule of your own. And since the only pattern now is 
!ga_sessions.customDimensions
, no entities will be selected at all. If you want to select certain entities, you should have at least one positive (non-
!
) rule under 
select
. If you want to select all entities and properties again, except for 
ga_sessions.customDimensions
, add 
*.*
: 
meltano select tap-ga360 "*" "*"
Then you should end up with the following in `meltano.yml`:
Copy code
select:
- "*.*"
- !ga_sessions.customDimensions
m
let me try
d
I'm happy to call on Monday if this doesn't address it
m
i tried the other way round last ime
- ‘!*.*’ - ga_sessions.visitorId.* - ga_sessions.visitStartTime.* - ga_sessions.userId.* - ga_sessions.date.* - ga_sessions.clientId.* - ga_sessions.fullVisitorId.* - ga_sessions.visitNumber.* - ga_sessions.totals.* - ga_sessions.geoNetwork.* - ga_sessions.device.*
ill try first excluded all and then select whatever i want
and let u know
d
Ah yeah, 
!*.*
won't work because (per https://meltano.com/docs/command-line-interface.html#exclude-parameter):
Exclusion takes precedence over inclusion. If an attribute is excluded, there is no way to include it back without removing the exclusion pattern first.
If you only want those specific patterns, you don't need 
!*.*
at all
Just this should be enough:
Copy code
select:
    - ga_sessions.visitorId.*
    - ga_sessions.visitStartTime.*
    - ga_sessions.userId.*
    - ga_sessions.date.*
    - ga_sessions.clientId.*
    - ga_sessions.fullVisitorId.*
    - ga_sessions.visitNumber.*
    - ga_sessions.totals.*
    - ga_sessions.geoNetwork.*
    - ga_sessions.device.*
Although 
ga_sessions.visitorId.*
doesn't look quite right, with the extra 
.*
at the end
Since the entity is 
ga_sessions
and the attribute 
visitorId
, you just need 
ga_sessions.visitorId
, without the extra 
.*
The patterns you use here should match the ones you see under "Selected attributes" in 
meltano select --list --all tap-ga360
The fact that this wasn't clear from the docs is useful feedback by the way 🙂
m
Got it
thanks
No luck, though i put excluded, still it is showing on selected
PFA
select.json
select.json
d
@mani The entity is called 
ga_session_hits
, but your patterns say 
ga_sessionS_hits
😉
m
omg , silly mistake
d
And the 
ga_sessions.trafficSource
attributes (etc) are included despite your exclusion rule because they are set as 
inclusion: automatic
in the discovered catalog, which means that they are always (automatically) selected per https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md#metadata: 
available
means the field is available for selection, and the tap will only emit values for that field if it is marked with 
"selected": true
. 
automatic
means that the tap will emit values for the field.
Actually, it looks like 
inclusion
isn't set at all in https://github.com/singer-io/tap-ga360/blob/master/tap_ga360/schemas/ga_sessions.json, and Meltano is assuming that means they're supposed to be automatically included: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/catalog.py#L282
To work around that, you may want to use the 
metadata
extra (https://meltano.com/docs/plugins.html#metadata-extra) to explicitly set 
inclusion
to 
available
for all streams and properties, so that selection will work correctly:
Copy code
select:
- ...
metadata:
  "*":
    inclusion: available
    "*":
      inclusion: available
m
yes, i saw that , the parent object is excluded but the children are present
il check metatdata
d
By the way, I'm working on a feature as we speak that would make catalog issues like this one easier to debug: https://gitlab.com/meltano/meltano/-/merge_requests/1875 🙂
m
@douwe_maan, how do we manage different enviroments configuration for example i am having tap-mongodb which is having different server for the enviroments
d
@mani In each environment, you can use environment variables to override certain settings with different values. Would that work for your use case?
m
yes that works, how to use those? any examples please?
d
7 Views
Open in Slack
PreviousNext
Powered by