The best pipelines:

• Move data reliably.

• Annotate payloads with metadata such as provenance,

collected_at

timestamps, fingerprints, etc.

• Generate stats to provide the operator with feedback.

• Validate and bifurcate payloads, if you’re lucky.

• Know about and act on payload sensitivities - obfuscate, hash, tokenize, redact, redirect, etc.

• Minimize moving pieces.

• Don’t spend all the CEO’s 💸💸💸 so they can afford that house in the Bahamas.