https://linen.dev logo
Join Slack
Powered by
Just wanted to ask if anyone has used meltano targ...
# troubleshooting
a
Just wanted to ask if anyone has used meltano targets as a reverse etl before?
c
v
I do it, simple way to dip your toe in is https://hub.meltano.com/loaders/target-apprise (Slack, teams, any notification system you can imagine, literally of a 100 of them) Just like @christoph mentioned format your data in a realtion. We haven't shared many patterns with each other so I don't go to 
.jsonl
I go 
meltano run tap-postgres target-apprise
(I also have ~5 other targets I write to, active directory, azuread, apprise, sftp, school based api) So in prod it looks more like something like 
meltano run tap-toggl target-postgres dbt:run tap-postgres target-apprise
Target's for saas are a bit too easy in some cases
Copy code
"""Apprise target sink class, which handles writing streams."""
from singer_sdk.sinks import RecordSink
from typing import Optional
import apprise


class AppriseSink(RecordSink):
    """Apprise target sink class."""

    def process_record(self, record: dict, context: dict) -> None:
        """Process the record."""
        a = apprise.Apprise()
        for uri in self.config["uris"]:
            a.add(uri)

        title: Optional[str] = record.get("title")
        body: Optional[str] = record.get("body")
        if title is None and body is None:
            raise Exception("Both the title and body cannot be None")
        a.notify(title=title, body=body)
Trick is in the dbt:run step, how to you be sure you're only sending the data you need to? There's some trickery there but it depends on what you're after
c
Trick is in the dbt:run step, how to you be sure you're only sending the data you need to?
The way how I solve this so far is to manage a (potentially compound) primary key in my dbt model. And I actually just write a sha-256 hash of the primary key values into a dedicated column in dbt (e.g. 
_id
) And then I use that column with the sha-256 hash as the primary key in the target, because the targets I deal with are all mostly datastores which have this concept of primary key exposed as a function. For targets which are not datastores, this approach may of course not work so well, if the target can't easily be persuaded to learn about the concept of an exposed primary key.
v
I do the same except I don't hash the key! Some other tricks like doing a full outer join, and a few dbt macros kind of bakes the rest of the cake!
c
The hashing is indeed optional and is probably a holdover from RDBMS thinking where specifying a size for a varchar column is considered good practice. 😄
v
You're probably right, I probably just don't follow good practices 🙊
c
What I meant was, that I have a habit of declaring varchar column sizes even though the database probably doesn't care (especially when using non-materialized views), since the underlying table is the source table populated by e.g. 
target-postgres
and the string fields in the dbt source table are all of type unlimited varchar ... 😉 Even though in the Postgres case, using a length limited varchar doesn't actually have any benefits ...
There is no performance difference among these three types, apart from increased storage space when using the blank-padded type, and a few extra CPU cycles to check the length when storing into a length-constrained column. While 
character(_n_)
has performance advantages in some other database systems, there is no such advantage in PostgreSQL; in fact 
character(_n_)
is usually the slowest of the three because of its additional storage costs. In most situations 
text
or 
character varying
should be used instead.
https://www.postgresql.org/docs/current/datatype-character.html
2 Views
Open in Slack
PreviousNext
Powered by