Is it possible to make the table name change between the sou Meltano #getting-started

Is it possible to make the table name change betwe...

john_sequeira

06/05/2023, 2:39 PM

Is it possible to make the table name change between the source and target? In other words, I have

src_table

, and I’d like it to sync to a table called

target_table

pat_nadolny

06/05/2023, 2:47 PM

@john_sequeira check out the inline data mapping docs related to renaming streams https://docs.meltano.com/guide/mappers#renaming-streams. That should do what you're looking for.

john_sequeira

06/05/2023, 2:54 PM

Thanks @pat_nadolny — that seems like what I need.

john_sequeira

06/05/2023, 3:19 PM

I’m reading the docs and have a couple follow ups… is this a doc bug? The example on the page you send says

Copy code

meltano run tap-csv hash_email target-sqlite

but the name of the mapper in the config is

lower

? (in the git repo the mapper name is

hash_email

Also I’m testing out an incremental sync and wanted to pass in a custom catalog and state file (with a custom ID to start syncing). Invoke doesn’t seem to support mappers and run doesn’t seem to support tap-specific arguments like ‘--catalog’ . I’m just getting my head around the various commands and am missing something about how to do both things (chain tasks together but pass in args)

user

06/05/2023, 3:31 PM

@john_sequeira ah yeah thanks for calling that out I created https://github.com/meltano/meltano/pull/7744 to fix it.

john_sequeira

06/05/2023, 3:32 PM

Also I think I can pass in needed args (state/catalog) using config?

user

06/05/2023, 3:33 PM

Yeah I think you should be able to pass those in with extractor extras https://docs.meltano.com/concepts/plugins#extras

user

06/05/2023, 3:34 PM

I also wonder if

meltano invoke tap-xyz | meltano invoke mapper-xyz > output.json

would work? I'd have to test that out. UPDATE: no you cant invoke a mapper yet https://github.com/meltano/meltano/issues/3233

john_sequeira

06/05/2023, 3:45 PM

Yeah — I tried dumping my extract to a json file and using the input parameter …

Copy code

meltano invoke meltano-map-transformer  --input output/table.json

It didn’t work for me — I think it’s looking for the json mapper config but I was thinking it should find this config via the meltano.yml.

Copy code

Traceback (most recent call last):
  File "meltano/clean/.meltano/mappers/meltano-map-transformer/venv/bin/meltano-map-transform", line 8, in <module>
    sys.exit(StreamTransform.cli())
  File "meltano/clean/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "meltano/clean/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "meltano/clean/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "meltano/clean/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "meltano/clean/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/singer_sdk/mapper_base.py", line 159, in cli
    mapper = cls(  # type: ignore[operator]
  File "meltano/clean/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/meltano_map_transform/mapper.py", line 63, in __init__
    super().__init__(
  File "meltano/clean/.meltano/mappers/meltano-map-transformer/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 98, in __init__
    config_dict.update(read_json_file(config_path))
TypeError: 'NoneType' object is not iterable

Open in Slack

Previous Next