We've been working on open sourcing how we deploy ...
# show-us-what-you-got
v
We've been working on open sourcing how we deploy jobs at AutoIDM, and decided to build a plugin for PDT github.com/AutoIDM/pdt-meltano Still working through how we want the interface to be on the hub, but you can now deploy to Azure Container Jobs AWS Batch Google Cloud Run With our fairly opinionated but light weight infra choices, same way we've been running Meltano ourselves internally for our data integrations for folks. Really easily
meltano --environment=prod invoke pdt-azure deploy
Secrets deployment is kind slick too
Copy code
$ meltano --environment=prod invoke pdt-azure secrets meltano-daily-sync
updated  TAP_SMOKE_TEST_API_KEY  •••• (4 chars)  •••• (7 chars)
Proceed? [y/N]
Other useful pdt commands for meltano
Copy code
pdt logs
pdt runs
pdt health
🔥 2
meltano.yml example
Copy code
version: 1
default_environment: dev
project_id: 01a0f39a-8c93-76f4-ad98-ac380915f9fa
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-smoke-test
    variant: meltano
    pip_url: git+<https://github.com/meltano/tap-smoke-test.git>
    settings:
    - name: api_key
      kind: string
      sensitive: true
    config:
      streams:
      - stream_name: animals
        input_filename: <https://raw.githubusercontent.com/meltano/tap-smoke-test/main/demo-data/animals-data.jsonl>
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
  utilities:
  - name: pdt-azure
    namespace: pdt_meltano
    pip_url: git+<https://github.com/AutoIDM/pdt-meltano.git>
    executable: pdt_meltano
    config:
        provider: google
        region: eastu2
  - name: pdt-gcloud
    namespace: pdt_meltano
    pip_url: git+<https://github.com/AutoIDM/pdt-meltano.git>
    executable: pdt_meltano
    config:
        provider: google-cloud
        region: us-east4
        project: pdt-derek
  - name: pdt-aws
    namespace: pdt_meltano
    pip_url: git+<https://github.com/AutoIDM/pdt-meltano.git>
    executable: pdt_meltano
    config:
        provider: aws
        region: us-east-1
jobs:
- name: sample
  tasks:
  - tap-smoke-test target-jsonl
schedules:
- name: meltano-daily-sync
  interval: '@daily'
  job: sample
Example deploy infromation given back
Copy code
Fetching list prices from the AWS price list...
Plan:
  reconcile shared ECR repository pdt and ECS cluster pdt
  build and push Docker image <http://651629739282.dkr.ecr.us-east-1.amazonaws.com/pdt:meltano-daily-sync-latest|651629739282.dkr.ecr.us-east-1.amazonaws.com/pdt:meltano-daily-sync-latest> (linux/arm64) (from meltano-daily-sync/Dockerfile)
  create Secrets Manager secret pdt-meltano-daily-sync-env
  reconcile the execution, task, and scheduler IAM roles
  allow pdt-meltano-daily-sync-task to update its own secret pdt-meltano-daily-sync-env
  reconcile Fargate task definition pdt-meltano-daily-sync (0.25 vCPU, 512 MiB, no time limit)
  create EventBridge schedule pdt-meltano-daily-sync: cron(0 0 ? * * *) (Etc/UTC)
  use default VPC subnets and security group sg-9560c1a9 with a public IP
  use existing bucket pdt-data-b1dc021115 (kept after destroy)
  grant pdt-meltano-daily-sync-task write access to meltano-daily-sync/ in bucket pdt-data-b1dc021115
Estimated monthly cost (us-east-1 list prices, before free tiers):
  Fargate (arm64): ~30 runs x 5 min assumed x 0.25 vCPU / 0.5 GiB  $   0.03
  Secrets Manager: 1 secret                                        $   0.40
  storage: 0 objects (0.00 GB)                                     $   0.00
  total                                                            $   0.43
  excludes EventBridge Scheduler free tier, ECR storage, and CloudWatch Logs usage
Proceed? [y/N]
Curious what your thoughts are on this, we've found it super useful especially with AI being able to just hook into these things so easily