Love the tap-jaffle-shop, and adding a simple dataset is a cool way to get started with any data analysis (e.g. this thread https://meltano.slack.com/archives/C01TCRBBJD7/p1680534309249419) Wondering if there would be much interest in a vast catalog of 'datasets'? Lot's of analysis starts from pre-packaged datasets of course. Option 1: Create a ton of individual plugins for each dataset. e.g. https://www.matatika.com/docs/data-sources/tap-govuk-weekly-road-fuel-prices/ https://github.com/Matatika/matatika-ce/blob/main/plugins/extractors/tap-govuk-weekly-road-fuel-prices--matatika.yml https://github.com/Matatika/matatika-ce/blob/main/plugins/extractors/tap-mad-landscape.yml Option 2: Create a tap that connects to dataset catalogs like Data.world (128k datasets) Thoughts?

I like the idea! I think there’s benefits to both approaches where the dataset is packaged with the tap and just spits it out in Singer format. And the next “level” of complexity could be the ones that actually connect to an external source. This also makes me want to get the templating feature in place to make it easy to import other meltano projects 👀

Hrrm - packaged with the tap is interesting. Hadn't thought of moving the data with the tap - but why not! For option 1, I loved the idea of being able to just add data in a single command, no config, just go! In my mental model, the fetch of the data was delayed until the tap was run.

@aaron_phethean - Yeah, I like both approaches very much. We have considered a connector like tap-socrata or tap-dataworld. However, a generic tap like spreadsheets anywhere doesn't really offer much discoverability - since its intended to use a BYO dataset model. So, for something like the mad landscape or fuel prices, the convenience and discoverability are very nice advantages of adding them as their own entry on the hub. Also, as you point out, its very nice to have zero-config setup, which is only really possible if we declare which dataset the tap is supposed to get.

UK has a postcode address file - non-commercial use permitted. It's a big file though. There are smaller datasets with just postcode to geo we've used - https://github.com/missinglink/uk-postcode-polygons Airports, airlines, currencies, countries all seem worth thinking about for simple datasets. Updating open weather map to new API is a good idea

My use case for this is public IP address files for major cloud providers. I use spread sheets anywhere but theres a variety of weird challenges with using it e.g. no obvious ID, URL includes a date that changes every 14 days eyetwitch , incremental runs could be cached based on content hash, etc. I wanted to write a tap for it but it wasnt worth it for the benefits I'd get vs spreadsheets anywhere.