it looks like the singer portion of meltano is foc...
# announcementsl
lance_dacey
04/16/2019, 7:48 PMit looks like the singer portion of meltano is focused on REST APIs - what about processing flat files or Excel files with merged headers, skipped rows, etc?
v
virjinia_alexieva
04/17/2019, 12:14 AMHi Lance, do you mind creating an issue in the meltano project https://gitlab.com/meltano/meltano/issues and we will prioritize accordingly?
virjinia_alexieva
04/17/2019, 12:15 AMWe would also be happy to get on a call and discuss the use cases for you and what you could get from the project
virjinia_alexieva
04/17/2019, 12:15 AMWould you be open to that? (cc: @danielle_morrill)
virjinia_alexieva
04/17/2019, 12:18 AMCurrently you can extract from google sheets + APIs only really
y
yannis_roussos
04/17/2019, 9:04 AM@lance_dacey We are currently working on a Tutorial on how to support this use case 🙂
As long as your excel files have a first line header with the "attribute names", this is currently supported in Meltano:
1. Export your Excel files as CSV
2. Use
tap-csvyannis_roussos
04/17/2019, 9:08 AMYou can check our draft, work in progress tutorial if you want to get an idea on how this works:
https://gitlab.com/meltano/meltano/blob/210ecd8b7816a45d4e09b2d4ff03a2129c0b8fa0/docs/src/docs/tutorial-tap-csv.md
And the
Custom Transformations and Modelsyannis_roussos
04/17/2019, 9:13 AMMeltano follows the ELT paradigm, i.e. you can transform the data if you want beforehand, but you can also Extract the raw data with minimum transformations (to remove for example sensitive data or columns with PII), Load the raw data to a RAW schema (similar to a data lake) and then clean, Transform and move the data to a final analytics schema. We use dbt for that last part.
l
lance_dacey
04/17/2019, 9:38 AMah - cool, I didn't see the tap csv option. right, I need to sometimes format the files (wrong encoding, merged headers or many sheets, lots of skipped rows or summary data) so I generally try to create a standard CSV
lance_dacey
04/17/2019, 9:38 AMand it looks like tap csv would work after that point
y
yannis_roussos
04/17/2019, 9:41 AMExactly 🙂
9 Views