hi, I’m trying to figure out how I can include a mapper in a schedule? Is there a way to do that?

Mappers currently can’t be included in a schedule. https://gitlab.com/meltano/meltano/-/issues/2924 is the issue that would enable schedules to work with

meltano run

. The best solution right now would be to orchestrate the

meltano run

command via an external orchestrator.

hm…. I’m currently relying on

meltano schedule run “$schedule” --state “$local_state_file”

which works great and is tons less complex than running an airflow instance. So yeah, I can orchestrate pretty much anything I want. But now `meltano run`seems not to offer any support for detailing with state files?

The confusing thing here is that

meltano schedule

calls

meltano elt

and not

meltano run

under the hood. We recently added support for

meltano run

to execute jobs incrementally, but you can’t currently override the state via the command line. The upcoming

meltano state

command https://gitlab.com/meltano/meltano/-/issues/2754 will be the solution for that

Hi is it still the case that mappers can't be added in schedules?

That’s no longer the case! You can define a job with arbitrary tasks and then schedule that job. It will use

meltano run

under the hood

Thanks @taylor! How do I add it to the scheduled tasks in

meltano.yml

? Is there a specific key, e.g

mapper

? or do I just pass a list into

extractors

? e.g

schedules:
- name: dynamodb-to-wh
  interval: '*/50 * * * *'
  extractor: [tap-dynamodb, alias-tables]
  loader: target-postgres
  transform: skip

https://docs.meltano.com/guide/orchestration#scheduling-predefined-jobs is your best reference. You’ll need to define a

job

first with

tasks

and then schedule that job directly. The format you just shared used the

meltano elt

command under the hood which does not support mappers.