Running

meltano elt

with tap-zuora used to work earlier but now it fails on few objects like ChargeMetricsRun throwing error:

CRITICAL There is a syntax error in one of the queries in the AQuA input (Query = select CreatedById, CreatedDate, EndDate, ErrorFileUrl, Id, ProcessedAccounts, ProcessedCharges, ResultFileUrl, RunNumber, StartDate, TotalAccounts, UpdatedById, UpdatedDate, Entity.Id from ChargeMetricsRun where UpdatedDate >= '1900-01-01T00:00:00 (+00:00)' order by UpdatedDate asc, Error = There is no field named EndDate)

Checking the catalog, I found this field defined as

"EndDate": {
            "type": null
          }

in the Schema. Is Zuora expecting to skip these fields? Anything changed in Meltano around handling such fields?

@rahul_anand Meltano does not currently support making arbitrary changes to discovered catalog files, but this is coming soon: https://gitlab.com/meltano/meltano/-/issues/2145 It's odd that you're seeing the tap generate a catalog with a field that doesn't actually exist, though. I assume you're using https://github.com/singer-io/tap-zuora and its discovery mode?

It’s bit surprising that the data download had worked earlier. May be something changed at Zuora API end. How can I disable the discovery mode and override the catalog for Meltano?

@rahul_anand There doesn't currently exist a native way to do that I'm afraid; Meltano is a little too confident in tap's discovery modes, it appears. However, you can try this workaround: In your custom plugin definition, remove the

discover

capability, but leave

catalog

(or

properties

, if this is an older tap). Then generate your own catalog, and copy it to

.meltano/run/tap-zuora/tap.properties.json

, which is where Meltano will look for a catalog if one wasn't generated using discovery mode.

Sure I will give it a try and confirm if that resolved my issue.

@rahul_anand Ah, great catch! By default, Meltano selects all entities and attributes (streams and properties), but it should obviously be skipping those that are marked as unsupported. Would you mind creating an issue for that, or shall I? As a workaround, you can explicitly tell Meltano not to select these fields:

meltano select tap-zuora --exclude YourTable EndDate

That will add the appropriate selection rule to your meltano.yml, so that Meltano can modify the generated (discovered) catalog file as appropriate before running the tap in sync mode.

Sure @douwe_maan I will create an issue and provide you the issue id. For now I am trying the approach you suggested earlier by disabling the ‘discover’. I am thinking this may be a better approach, giving me more control on executions in production, preventing any scope of errors due to changes in schema.

Yeah, that’s a reasonable workaround in this case. I want to get Meltano to the place where you’ll just trust it to handle this for you, but that will always require a little bit more confidence in Meltano and the specific tap than may be warranted right now :)

Issue created: https://gitlab.com/meltano/meltano/-/issues/2168

@rahul_anand https://gitlab.com/meltano/meltano/-/issues/2168 has been resolved in v1.41.0! 🎉 https://meltano.slack.com/archives/CP8K1MXAN/p1595286519075100