https://linen.dev logo
Join Slack
Powered by
Hi everyone! I'm attempting to run the `tap-quickb...
# plugins-general
r
Hi everyone! I'm attempting to run the 
tap-quickbooks
tap, and no data is coming out. I see that I'm successfully selecting a certain stream, but just no data is flowing through when I run the pipeline. Anybody ever seen this before?
a
If you run it in debug mode, does anything interesting come up? https://meltano.com/docs/command-line-interface.html#debugging
r
Unfortunately not 😕 it looks like it's just not finding anything in the source
@amanda.folson any other ideas? Do we know of anyone who is actively using this tap in Meltano?
If we hit the Quickbooks API manually in Postman, we are getting data back
Also if we run the singer tap manually we are getting data... it must be something screwed up with our Meltano configuration
@hassan_syyid sorry to bug you, but it looks like you work at Hot Glue. Any chance you would be able to help/know of anyone who could help?
h
@ricky_renner what’s the exact command you’re running
r
meltano elt tap-quickbooks target-bigquery
h
Also are you using the catalog param or properties param?
r
I was just about to ask about that. I believe the catalog param as I think Meltano would be <http://(https://meltano.com/docs/integration.html#extractor-catalog-generation)|generating it on the fly>.
h
It should use properties
r
Got it. Do you have an example of this being used so we have something to base it off of?
Also @amanda.folson not sure how we would want to update the 
meltano.yml
. I'm guessing it'd be the metadata extra somehow.
Thank you for replying so quickly by the way @hassan_syyid
h
It’s listed in the README
r
Thank you for sending that over - I more so was wondering how this works with Meltano.
h
Looks like you’d change the 
capabilities
section
Instead of 
catalog
it’d say 
properties
r
Thank you! We will test that out!
@amanda.folson do we need to update the metadata extra probably to actually specify what the propeties will be?
@hassan_syyid this properties file would specify which tables we want to pull from Quickbooks, right?
h
@ricky_renner Yes, it’s really the same as a 
catalog.json
– you would just pass the same file using 
--properties
instead
r
@hassan_syyid so will we need to manually refresh the 
refresh_token
or will the tap handle that? Meaning - will the tap see that it is expired and just generate a new one?
It looks like it'll refresh it automatically, but I just wanted to be sure.
h
It handles it
r
@hassan_syyid it looks like if the refresh token is invalid, it fails before it gets to that line
h
Yeah it needs the refresh_token to be valid in order to refresh 🙂
r
oh does expired not mean it is invalid? I must've had my refresh token screwed up then. Thank you!
<!subteam^S02A7AJFXQC> we've tried a few different things here. I currently have this set up by adding in the 
tap-quickbooks
tap as a custom plugin as it looks like how it is configured in Meltano is not correct (it includes 
catalog
as a capability as opposed to 
properties
). However, now that I'm implementing this as a custom extractor, it's not generating the 
catalog.json
file. Do I need to add something specific to the custom extractor configuration for it to generate a 
catalog.json
? I know it's confusing - but we basically want the 
catalog.json
to still be generated, but tap-quickbooks requires us to use 
--properties
so it would be like 
--properties catalog.json
as Hassan mentioned above.
d
@taylor Didn't we fix the incorrect capability recently? 🤔
@ricky_renner did you list the discover capability? That’s what determines whether a catalog will be generated. The properties/catalog capability than determines what flag will be used to pass it to the tap in sync mode
r
Yes I did
d
And when you run it with ‘meltano elt’ it’s not generating a catalog.json? If you run with ‘meltano --log-level=debug’, do you see the tap being invoked with ‘—discover’?
r
oof - I had 
discovery
not 
discover
d
I assume you’re not accidentally specifying an explicit catalog.json with the catalog property in meltano.yml, or the —catalog option ‘meltano elt’?
Ah there we go!
Meltano could warn about that
b
Hey <!subteam^S02A7AJFXQC> I am working on this with @ricky_renner . We were able to get the tap-quickbooks working, but now there seems to be an error in the loader. The error seems fairly non-descript and just say "Loader failed." I included an image of my terminal readout. We are currently using target-bigquery as our loader.
r
https://github.com/adswerve/target-bigquery/blob/master/target_bigquery/schema.py it looks like it is failing here because the target bigquery is expecting a 
schema["properties"]
section, so I'm assuming the 
tap-quickbooks
is not sending that.
e
@ricky_renner indeed there is at least one 
object
field without `properties`: https://github.com/hotgluexyz/tap-quickbooks/blob/8cc077bfbd11fa6bda7d45deb6b0cd5609d2bf22/properties.json#L256-L260
r
@edgar_ramirez_mondragon thank you for the response. I am just using the Invoice stream and it looks like that has a properties object...? But maybe you mean the actual fields within the Invoice stream
Would the solution be to make sure that each one of those fields defined in that 
properties.json
has a 
properties
child object
e
I think the actual place where that key is missing is here. Maybe you can try a schema override on that stream? Something like
Copy code
extractors:
- name: tap-quickbooks
  schema:
    Invoice:
      SubTotalLineDetail:
        type: "object"
        properties: {}
r
@hassan_syyid I can confirm that my refresh token was just working, but it expired, and now the tap is telling me that the token is invalid. Is this expected behavior?
h
QBO automatically expires refresh tokens if you generate a new one
So if you or your colleague is generating a new refresh token for the same account it will expire
r
Ah that makes sense. Thank you @hassan_syyid. Then we'll have to capture that and store that refresh token somehow
@edgar_ramirez_mondragon does this seem like an issue with the tap or the target? I can't tell if having this 
properties
section is required by the singer spec
e
@ricky_renner it's not required by the singer spec (which delegates schema validation to JSON Schema) but target-bigquery does seem to require it. Did you try the schema override https://meltano.slack.com/archives/C013EKWA2Q1/p1629314366085900?thread_ts=1629150244.061500&amp;cid=C013EKWA2Q1?
b
@edgar_ramirez_mondragon Hey Edgar, we attempted a schema override but we are still receiving an error. Would a schema override work in this case since SubTotalLineDetail is flagged as Automatic? I know that for 
select
filtering, SubTotalLineDetail has to be included.
e
@benjamin_blankartz hmm, the tap then may be doing some other weird thing in the catalog. Can you share here the contents of 
.meltano/run/tap-quickbooks/tap.properties.json
?
r
<!subteam^S02A7AJFXQC> in this instance, would you expect our team to fix the issues we've found in the tap, target, and Meltano?
@edgar_ramirez_mondragon below is the contents of the specific stream we are looking at
I've tried only selecting 
Id
and 
TotalAmt
but it seems as though it is still throwing the properties error
invoice_stream.json
invoice_stream.json
a
I've tried only selecting 
Id
 and 
TotalAmt
 but it seems as though it is still throwing the properties error
Many taps do not filter their SCHEMA messages as a results of selection rules. Meaning, only the record is filtered and the SCHEMA is passed as-is regardless of filter. Can you use the schema-override method described by @edgar_ramirez_mondragon to remove any property-less object definitions? More context is here in an SDK issue - exactly discussing if we should support variant 
object
schemas. Some targets support them but unfortunately some apparently do not.
r
@aaronsteers how do we specify a schema override when there is an object within an object field?
a
Can you try copy-pasting the same stream definition as in the catalog, but with those property-less object fields removed? This would remove those fields.
Just yesterday @thomas_schmidt ran into the issue of selection filter rules not applying to schema and created an PR for that tap (
tap-stripe
) as described here: https://meltano.slack.com/archives/C01TCRBBJD7/p1629359180216100?thread_ts=1629220362.197000&amp;cid=C01TCRBBJD7
But faster to alter the catalog to not have those offending fields, if you can afford to omit them. (And assuming the tap is correctly using your passed-in catalog schema.)
b
@aaronsteers Just to check that we are using the same terminology, in order to get the tap-quickbooks working, we needed to create a custom tap that uses a properties file instead of a catalog file. Should the modifications be performed in the properties file instead?
a
Sorry - I probably was not using clear terminology... Let me try to restate...
The root cause appears to be that your tap (tap-quickbooks, I think?) is sending schema messages that contain "variant" object definitions, which your target (target-bigquery) is not able to parse. If selection rules were also being applied to schema messages, you could simply deselect those fields that are missing sub-property definitions. Since that doesn't seem to be the case (schema doesn't seem to be filtered by the selection criteria), my recommendation was to use a custom catalog - or use the meltano.yml method of overriding specific schema definitions in order to remove those offending subfields from the catalog definition. Does that make help clarify at all?
This is a pretty long thread, so it's also possible I may have missed or misread some of the context.
Happy to have a quick call if that would be helpful to unblock you on this.
b
That would be super helpful, but I am in meetings until 4:30 EST. I will try the catalog reduction suggestion and see where that gets us.
@aaronsteers --catalog properties.json seems to be still loading everything. I stripped out the offending field and other tables that we were not interested in. This may be due to the custom capabilities for the quickbooks tap.
e
I think the schema override might've not worked because you need a dummy item in the properties object:
Copy code
extractors:
- name: tap-quickbooks
  schema:
    Invoice:
      SubTotalLineDetail:
        type: "object"
        properties:
          dummy:
            type: ["string", "null"]
b
What would happen if a null type was received when the tap was expecting an object?
@edgar_ramirez_mondragon This is the error that I go, similar to before.
e
ah, the field should be nullable then:
Copy code
extractors:
- name: tap-quickbooks
  schema:
    Invoice:
      SubTotalLineDetail:
        type: ["object", "null"]
        properties:
          dummy:
            type: ["string", "null"]
b
@edgar_ramirez_mondragon looks like we are getting the same error.
t
As AJ stated we had the exact same problem with the 
tap-stripe
. As mentioned I created an MR that the selection is also applied for the SCHEMA message -> This way we can deselect the fields which create problems (we don’t need them anyway). What we did additionally was applying a small patch to the catalog which basically removed all schema definitions with empty properties: Turns that
Copy code
metadata": {
    "type": [
      "null",
      "object"
    ],
    "properties": {}
}
into that
Copy code
metadata": {
    "type": [
      "null",
      "string"
    ]
}
as a quickfix we basically dump the catalog first, process it with a regular expression python script and then reuse the processed catalog for the EL job. Maybe this could also be a (hacky) approach for you guys in the meantime
b
@thomas_schmidt thanks for this. We do seem to be running into issues getting tap-quickbooks to import a different catalog because we had to declare it as a custom tap with 
properties
capabilities. Meltano would throw back an error if we call 
elt tap target --properties properties.json
. Would 
--catalog catalog.json
still enable tap-quickbooks to use 
properties
while also consuming a custom 
catalog
?
r
@aaronsteers we will definitely take you up on your offer for a phone call... at this point after 76 messages, that might make the most sense!😆
a
For others on the thread, I think @benjamin_blankartz and team have a path forward and we (Meltano) have taken some notes on how we might streamline the troubleshooting experience in the future, and/or prevent the issue from occurring by logging a new issue on the target-bigquery implementation.
r
@aaronsteers thank you very much for jumping on a call and helping us out with this! The whole Meltano teams dedication has really impressed us and we really appreciate it
a
@ricky_renner - My pleasure. The feedback and findings will be valuable for others as well, I think. 👍
b
<!subteam^S02BCD9FFEF> If a tap updates its configuration file at runtime, how would Meltano track those changes? @hassan_syyid
e
@benjamin_blankartz that behavior is not standard, so Meltano has currently no way to track the changed config file. I'm curious, what is the tap changing in the file? An auth token perhaps?
a
We don't currently support what I'm calling "settings write-back" - meaning the settings are expected to only be read from disk (by the tap at least) and never written back.
b
@edgar_ramirez_mondragon Tap-quickbooks will update a refresh token if it expires. We may look into handling that outside of Meltano.
r
@edgar_ramirez_mondragon and @aaronsteers here is where it is happening in the tap. We could potentially write our own version, but this seems to inhibit this tap working with Meltano out of the box (i.e. we'll have to write a script to get a new refresh token).
I know this whole thread has been a while but my question is related to all of this. <!subteam^S02BCD9FFEF> for a particular tap, where is the actual config file written to? Is it somewhere in the virtual environment created for a tap?
e
Hey @ricky_renner! The config file is (temporarily) saved as 
.meltano/run/<EXTRACTOR>/tap.<SOME UUID>.config.json
. If you run a command in debug mode like 
meltano --log-level=debug invoke <EXTRACTOR>
you should see a log line revealing the config file name 
Created configuration at ...
r
@edgar_ramirez_mondragon it looks like that file is deleted at the end of the run. I see this in the logs:
Copy code
Deleted configuration at /project/.meltano/run/elt/2021-12-10T170813--tap-quickbooks-custom--target-bigquery/c93eb680-e363-4e59-a2c4-6e696ce06cff/tap.3a49254e-2c07-495c-b085-1b132b61dd0c.config.json
Is there a way to maintain that configuration after the run so we could extract some of the properties in it
@edgar_ramirez_mondragon just following up on this
e
@ricky_renner
Is there a way to maintain that configuration after the run so we could extract some of the properties in it
Not really at the moment. Though you can get the config file the tap/target would use with 
meltano config tap-quickbooks-custom
or 
meltano invoke --dump=config tap-quickbooks-custom
. Unless your tap writes back to the config file and you need the updated contents, which is a use case Meltano doesn't currently support.
r
@edgar_ramirez_mondragon well unfortunately, this tap does write back to the config because the Quickbooks API will occasionally switch the refresh token out for a new one and just return it in response. So there needs to be a way to extract this information in order to effectively extract from Quickbooks using Meltano.
t
This is super interesting… so basically Meltano would need to update a secret on the fly. @edgar_ramirez_mondragon can you make an issue about this? Until we have some sort of secrets backend I’m not sure how we could best support this.
a
We have been calling this "settings writeback" and we have some discussion on this here: Add 'write-back' capability to tap and target settings (#2710) · Issues · Meltano / Meltano · GitLab
For this issue, we are currently gaging feedback and asking for developers and users to provide use cases if this is a blocker for them. If this affects you, please post to the comments.
@ricky_renner - Could you help by logging some requirements description there in the issue. Also, if you have available workarounds today or not, would be helpful info.
r
@aaronsteers yeah I can add that information. And it honestly might be an issue with the tap. It looks like it's basically using the sandbox version of Oauth for the Quickbooks API but I'm wondering if there is a way to integrate in a more production grade fashion
Alright just added a comment. Lemme know if that suffices!
h
@ricky_renner If the sandbox flag is flipped to false then you can use production keys without issue
r
@hassan_syyid this is if we use the singer tap standalone - not with Meltano, correct?
h
I think you should be able to edit/create your own yml configuration for Meltano to pass the 
is_sandbox
flag as 
false
But not 100% sure
r
Our configuration is already setup for production. The problem is that Meltano does not save the updated config file with the new refresh token
a
@ricky_renner, @hassan_syyid Sharing back this quick summary back to the channel in a new thread, since this original thread is now quite long and covering multiple topics. As discussed in the thread: 1. Meltano does not support 'settings write-back' by the tap/target - although we do have a ticket open to explore those requirements. 2. Meltano by default will generate a new config file each time the tap and target and cleans up the files after execution for security purposes. 3. @ricky_renner has documented here that 
tap-quickbooks
does support and/or expect this to handle a rotating 
refresh_token
. 4. To my knowledge, we don't (yet) have a way to run meltano 
elt
with a self-managed config file. For past use cases, we generally found there was some workaround in the tap itself. So far, just 
tap-jira
and 
tap-quickbooks
have been identified. I believe 
tap-jira
had a workaround so it dropped off the priority. @hassan_syyid mentioned that the 
tap-quickbooks
setting called 
is_sandbox
may be a workaround but I can't speak to that myself.
Open in Slack
PreviousNext
Powered by