Hi all, I’ve been writing a blog series about asse...
# announcementsa
andrew_stewart
02/16/2021, 8:36 PMHi all, I’ve been writing a blog series about assembling a modern data stack from open source / open core components for learning and hobby purposes (and on the cheap! $$) - My latest piece is a tutorial that walks through the entire stack using Meltano + dbt + Superset. Any feedback appreciated!
https://towardsdatascience.com/data-stacks-for-fun-nonprofit-part-iii-dcfd46da9f9f
d
douwe_maan
02/16/2021, 8:42 PMGreat introduction of Meltano:
We’re going to use Meltano to manage our ELT (not ETL!) pipelines, each of which will extract data from some source and load that data into a target database. If we were rolling our own ETL solution by hand, we would probably be writing an awful lot of custom python code for each data extraction and loading process, but fortunately the open-source Singer standard composes these processes into interchangeable sets of “taps” (data sources) and “targets” (load destinations). Meltano helps abstract away much of the configuration details, and provides a convenient execution layer. Meltano can also handle the downstream transformation (via dbt) and orchestration (via Airflow) steps, making it convenient top-level controller for your entire project, however for illustrative purposes we’re going to only use Meltano for the extraction and loading steps.
douwe_maan
02/16/2021, 8:43 PMThere are some errors here though:
we only need to add the `tap-spreadsheets-from-anywhere` plugin to our project. This is a very versatile tap that will let us extract CSV and Excel files from many sources including http/s, s3, sftp, and more.
The tap is actually calledmeltano add extractor tap-spreadsheets-from-anywhere
tap-spreadsheets-anywherefromdouwe_maan
02/16/2021, 8:44 PMAnd in the
meltano.ymlvariantcapabilities Copy code
plugins:
extractors:
- name: tap-spreadsheets-anywhere
pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
config: # etcdouwe_maan
02/16/2021, 8:45 PM"Stitch" should be "Singer" in:
While Stitch (via Meltano) loads data into a database from one or more external sources
douwe_maan
02/16/2021, 8:46 PMEither way, solid introduction to all three tools, great job!
a
andrew_stewart
02/17/2021, 8:38 PMAk! Thanks for all those catches @douwe_maan!
andrew_stewart
02/17/2021, 8:48 PMWhat’s really sad is that I proof-read it like 3 times 😛
c
chris_dietrich
02/19/2021, 5:09 PMGreat series. Bookmarked for later review them later.
s
sam_edelstein
02/21/2021, 2:31 PMThis is a great series - appreciate both the overview and the code to make it all work. Lots of smaller orgs who could benefit!
a
andrew_stewart
02/22/2021, 7:17 PMAbsolutely, that’s the hope! As well as enabling folks to learn about modern data stacks w/o access to big enterprise budgets.
9 Views