Hey Everyone :wave: Getting started with ingestio...
# announcements
e
Hey Everyone 👋 Getting started with ingestion is now guided, with Ingestion and Workflow pipelines, automatic stream discovery and multiple backfills. Databricks is now a fully supported data store with faster Arrow loads, every pipeline gets a Streams tab and fixes land across Elementary, Instagram, Snowflake, BigQuery and Google Ads. ☁️ Prefer not to manage pipeline maintenance yourself? Meltano Cloud gives you fully managed connectors and direct access to our team. Here's what's new ✨ Features • From new account to ingested data, in minutes: We’ve been busy streamlining the process for new users to create an account, set up the essentials in a workspace and move data from source to warehouse. Simply head to the website at Meltano.com, hit “TRY MELTANO” and get ingesting your data. • Ingestion as a first-class action: We now formally do what we say in Meltano Cloud: Ingestion. The simplest form of a pipeline, ingestion does what it says on the tin - ingest data from tap to target - fast. We’ve also optimised stream selection, making it super easy to discover streams and use that to populate dbt sources. • Backfill Improvements: Backfill has historically been a pain point for an ingestion pipeline to reload history without affecting its incremental sync. We’ve optimised this process, minimising cognitive overheads so you can focus on the things that matter. An ingestion can now have multiple backfills running at the same time, each listed in a new Backfills tab. Backfills can be configured by stream and start date. • New Streams tab: Each pipeline now has its own Streams tab, so you can review discovered streams and export them in the format you require. • Databricks data store support: Databricks is now a fully supported, first class data store, joining the likes of Postgres, Snowflake and BigQuery. Run full-load and incremental pipelines with target-databricks, including with Apache Arrow bulk loading for much faster loads on big tables, plus dbt transforms with dbt-databricks and pipeline state kept in Databricks. • New tap-onspd connector: Add the Office for National Statistics (ONS) Postcode Directory (ONSPD) as an extractor, providing comprehensive UK postcode data, including latitude, longitude, geographic classifications, and code-to-name lookups for those geographies. • Faster MongoDB extraction with Arrow: tap-mongodb v0.4.0 can now send incremental streams as Arrow batches for faster extraction. It also adds document_layout, which can load each top-level field as its own column, and change_stream_pipeline for change stream syncs. 🐛 Fixes • Bumped target-bigquery to v1.1.3: fixes a failed dedupe before upsert on a table with a float key, and timestamps with an offset like +0000 now load. Fields with a period in their name also load into BigQuery now. • Bumped tap-googleads to v0.15.1: stops duplicate records when a custom query replaces a default stream, and corrects the setting docs. • Bumped dbt-tap-googleads to v0.6.1: the campaign performance by location model now builds on BigQuery. • Bumped tap-instagram to v0.5.4: custom user insight reports no longer fail, and syncs over a long history are much faster because daily and time-series insights are requested in 30-day windows. A new sync writes one demographics snapshot per breakdown, not a copy for every past day. • Bumped target-snowflake to v0.20.4: loaded data can be queried as soon as a run finishes, because the target now waits for its commit before closing. It also no longer pins certifi, which drops the distrusted eMudhra intermediate CAs. • Bumped tap-spreadsheets-anywhere to v0.7.2: skip_empty_rows now works for Excel files, and an Azure container is listed once per run instead of repeatedly. • Bumped tap-tiktok to v1.2.0: now installs on Python 3.10. • Elementary fixes: Adding app-elementary now gives you its Data Quality Dashboard instead of opening Explore, and no longer resets your default data store to Warehouse. Elementary reports also run on Snowflake (files-dbt v1.6.2, app-elementary v0.4.2). Upgrade your files-dbt bundle to pick this up, and if your default data store was reset to Warehouse, set it back. 🚀A guided first pipeline, multiple backfills, full Databricks support and a Streams tab, plus a solid round of fixes, all make it quicker to get your data flowing. Thanks for the reports, and keep them coming! 📚 Curious which setup fits you best? Whether you're leaning Meltano Open or Cloud, the documentation lays out how they stack up against each other. 💬 And if any of these sparked a question (or you've got a connector you're waiting on), grab a slot here, always happy to chat.
melty 2
👍 1