Hi , Does Meltano support getting last updated time , number of records , total table size of various data sources like MySql, Oracle, Snowflake, Postgresql . If yes, can you please point to the documentation on how to get this using Java or Python API

Hi @trkrishnan I'm intrigued by your requirement. Meltano as a project is for moving data from sources to targets - in this respect it does support MySql, Oracle, Snowflake, Postgres and many others (https://hub.meltano.com/) Do you have a requirement to get stats / metadata from those databases, rather than the actual data? In theory, it would be fairly simple to create a plugin that made SQL queries for those stats (using the same tap for that database) and import that as data into another target then simply report on them. Wouldn't be a big effort, and serving those stats as datasets is something many products could do (including ours @Matatika)

Thanks for the quick response

@trkrishnan , I tweaked

tap-oracle

to pull

sys.*

information which pulls all the meta data about tables. Really depends what you're after, if you can get the information out of the information schema you can certainly do it with singer (ie meltano) 🙂

Agreed with all the above. I'll note that the Singer Spec for catalog metadata does have a reserved metadata keyword for

row-count

which some taps do populate. Developers are also free to add any additional metadata they would like into the catalog.json output (such as

table-size

, for instance). This is not a very common practice though due in part to the dev cost of writing the extra logic that scans system tables to populate those row counts. And for some systems, performance cost at runtime could also be a factor in whether the tap populates row counts during discovery or not. As others have noted, you could write custom scripts to capture this info (possibly using the catalog.json file as an input), or if you are the developer of the tap you could add the feature natively during catalog discovery. Hope this helps!