Anyone using GitHub artifacts (or something similar on gitlab) to store state for meltano runs on GitHub actions

Hi, @visch. I'm interested in exploring this more also. Based on this docs page, I think this would entail a

download-artifacts

action at the beginning of the pipeline and an

upload-artifacts

action at its end. I do want to call out that if two instances run simultaneously, the one that finishes last will clobber the state of the other(s). That said, the impact of this would only be some records being sent multiple times - which is something that can be handled.

Got it! Sounds easy, we shall see 🙂

@visch please let us know how you go. Very interested also

Will do, haven' tackled this quite yet. Odds are by the end of this week or before I'll give it a shot!

Hello, I bookmarked this thread a long time ago. Is there an example of Meltano running via actions in the wild, I'd love to see how it was implemented

Have one in prod (Been working for a month or so), no incremental though, someone was going after incremental a while back not sure who or what the outcome was but here you go as I guess I'm feeling extra nice today 😄 Pretty slick imo (serverless, super simple, logs taken care of, pay per minute)

name: pipeline
on: 
  schedule:
    #Daily at 10am UTC, 2am PST
    - cron:  '0 10 * * *'
  workflow_dispatch: 
jobs:
  pipeline:
    runs-on: ubuntu-latest
    env:
      python-version: 3.9
    steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ env.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ env.python-version }}
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install meltano==1.89.0
          meltano install
      - name: Run tap-clickup to target-stitch
        run: meltano elt tap-clickup target-stitch
        env:
          TAP_CLICKUP_API_TOKEN: ${{ secrets.TAP_CLICKUP_API_TOKEN }}
          TARGET_STITCH_CLIENT_ID: ${{ secrets.TARGET_STITCH_CLIENT_ID }} 
          TARGET_STITCH_TOKEN: ${{ secrets.TARGET_STITCH_TOKEN }}

Awesome, thanks