https://linen.dev logo
Join Slack
Powered by
Just starting with Meltano. I was able to walk th...
# getting-started
s
Just starting with Meltano. I was able to walk through the getting started guide to sync data from gitlab into bigquery. However, I’m having trouble getting the quickbase tap to work. Meltano 
elt
runs without errors, but nothing gets synced to BigQuery. I’ve successfully synced my quickbase data using stitchdata, which apparently uses this same tap. So I’m not clear on what’s going on or how to further troubleshoot this.
Another confusing thing is that it doesn’t seem to be capturing any state after a run….
Copy code
(meltano) ➜  mse-tut git:(master) ✗ meltano --log-level=info elt tap-quickbase target-bigquery --job_id=quickbase-to-bigquery
2022-05-16T18:38:16.818796Z [info     ] Environment 'dev' is active
2022-05-16T18:38:17.486889Z [info     ] Running extract & load...      job_id=quickbase-to-bigquery name=meltano run_id=3250e20e-5e47-437a-80cb-e5b3a017477b
2022-05-16T18:38:49.145222Z [info     ] INFO Pushing state: {}         cmd_type=loader job_id=quickbase-to-bigquery name=target-bigquery run_id=3250e20e-5e47-437a-80cb-e5b3a017477b stdio=stderr
2022-05-16T18:38:49.154500Z [info     ] Incremental state has been updated at 2022-05-16 18:38:49.154307.
2022-05-16T18:38:49.203452Z [info     ] Extract & load complete!       job_id=quickbase-to-bigquery name=meltano run_id=3250e20e-5e47-437a-80cb-e5b3a017477b
2022-05-16T18:38:49.203605Z [info     ] Transformation skipped.        job_id=quickbase-to-bigquery name=meltano run_id=3250e20e-5e47-437a-80cb-e5b3a017477b
(meltano) ➜  mse-tut git:(master) ✗ meltano --log-level=info elt tap-quickbase target-bigquery --job_id=quickbase-to-bigquery --dump=state
2022-05-16T18:38:57.229063Z [info     ] Environment 'dev' is active
Could not find state file for this pipeline
meltano.yml.yaml
meltano.yml.yaml
t
I ran into a similar issue with state data last week. It was because my tap configuration was missing the "state" capability. The page for the quickbase tap on the Meltano Hub doesn't indicate that it supports state, though the readme for the tap does, so... maybe manually add "state" to the list of capabilities and see if that fixes it?
re: nothing getting pushed to BQ - try running 
meltano invoke tap-quickbase
and see what it produces. Often no data getting replicated is caused by the 
select
rules being wrong but, even if that isn't the cause, seeing the output produced by the tap will give you some clues as to what's going on.
s
Ahhh, so adding the “state” capability does make it so that it returns a state, but that state is empty 
{}
. 
invoke
gives me almost nothing except when I use 
debug
log-level. I’m only selecting 
etahub__lgen
and I see a bunch of lines like this:
Copy code
2022-05-16T18:57:41.446208Z [debug    ] Visiting CatalogNode.METADATA at '.streams[21].metadata[0]'.
2022-05-16T18:57:41.446226Z [debug    ] Visiting metadata node for tap_stream_id 'etahub__lgen', breadcrumb '['properties', 'rid']'
2022-05-16T18:57:41.446252Z [debug    ] Setting '.streams[21].metadata[0].metadata.selected' to 'False'
2022-05-16T18:57:41.446269Z [debug    ] Setting '.streams[21].metadata[0].metadata.selected' to 'True'
2022-05-16T18:57:41.446289Z [debug    ] Skipping node at '.streams[21].metadata[0].metadata.inclusion'
2022-05-16T18:57:41.446307Z [debug    ] Skipping node at '.streams[21].metadata[0].metadata.selected'
2022-05-16T18:57:41.446325Z [debug    ] Skipping node at '.streams[21].metadata[0].breadcrumb[0]'
2022-05-16T18:57:41.446342Z [debug    ] Skipping node at '.streams[21].metadata[0].breadcrumb[1]'
So it looks like it’s trying to sync the table, but it’s skipping all the fields?
t
I'm not sure. If you run 
meltano invoke tap-quickbase --discover
what does it produce? The stream names are often not what you'd expect, in my experience, so odds are that the value of the 
select
rule isn't catching the table you want for some non-obvious reason.
s
That command lists a whole lot of json that’s hard to parse. When I run 
meltano select tap-quickbase --list
it does show the tables and fields that I would expect
t
That's a good sign. When you run 
meltano invoke tap-quickbase
what what does the resulting output look like? Are there SCHEMA and RECORD messages? I'm not talking about the debug output (i.e. I'd run it without --log-level=debug for simplicity), I'm talking about the data written to stdout (which is later what gets sent to the target, BTW...)
s
Nope, I get nothing but a message about the environment:
Copy code
(meltano) ➜  mse-tut git:(master) ✗ meltano invoke tap-quickbase
2022-05-16T19:19:59.634105Z [info     ] Environment 'dev' is active
t
🤔
Well at least if the tap isn't producing anything that explains why nothing is getting loaded 😉
Personally, I would grab the source for the tap from github and try to get it working independently of meltano. In my experience these types of issues almost never have anything to do with meltano but it's good to eliminate variables. You can get the config that meltano is sending the tap by running 
meltano invoke --dump config tap-quickbase
and use that to run the tap from source. At least that way you know you're working with the same configuration meltano is.
s
Yeah, I might try that. Thanks for your help
Open in Slack
PreviousNext
Powered by