https://linen.dev logo
Join Slack
Powered by
Hi! I'm using the postgres tap (<https://github.co...
# singer-taps
m
Hi! I'm using the postgres tap (https://github.com/MeltanoLabs/tap-postgres) and I have 2 questions: • is there a way to select only certain records from the DB, perhaps by passing a where clause somewhere: (e.g. 
where custmer_id = 100
) ? • I would like to add a 
stream_maps
-based transformation and use 
faker
in it. The transformation fails because "'fake' is not defined" . The docs say:
Copy code
Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an addtional dependency (through the singer-sdk faker extra or directly).
What does that mean? How can I enable faker? Thank you 🙂
I'm using the following config:
Copy code
config:
           stream_maps:
            data_platform_raw-tbii_daily_gmv_nmv:
              tb_channel_id: fake.pyint()
              __filter__: >
                actual_date == '2024-06-26' 
                and tb_account_number == 1542 
                and tb_channel_abbr == 'zafd'
          faker_config:
            seed: 0
meltano invoke
raises with an error saying that the 
fake
object is undefined. Now it's kind of surprising because the 
tap-postgres
docs mentions 
faker_config
among the settings. Why mentioning it if the plugin doesn't support faker?
a
Could you share the full error stack trace? Might be you need to raise an issue on the tap repo. Do you have 
faker
as an extra dependency in your 
pip_url
for this tap?
m
Untitled
Untitled
Thanks @Andy Carter. Here's the stacktrace
Do you have 
faker
as an extra dependency in your 
pip_url
for this tap? (edited)
how do I do that?
Copy code
- name: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git@v0.0.11>
I tried adding 
[faker]
after 
tap-postgres
, after 
v0.0.11
, after 
postgres.git
. It always failed to install
Ok I tried this way, which works with pip install:
Copy code
pip_url: meltanolabs-tap-postgres[faker] @ git+<https://github.com/MeltanoLabs/tap-postgres.git@v0.0.11>
however 
meltano install
fails with:
Copy code
raise CliError("Failed to install plugin(s)")  # noqa: EM101
Ok this was successful:
Copy code
- name: tap-postgres
    variant: meltanolabs
    pip_url: meltanolabs-tap-postgres[faker] git+<https://github.com/MeltanoLabs/tap-postgres.git@v0.0.11>
(There should be something in the docs that explains how to install plugins with extra dependencies, btw) In any case, even with this, the stream transformations can't find the 
fake
object
e
Hey @Massimiliano Marcon! I'm about to cut a release of the SDK and do all the downstream chores.
(There should be something in the docs that explains how to install plugins with extra dependencies, btw)
I agree! I'm curious what docs you were following. sdk.meltano.com, docs.meltano.com?
🙌 1
m
@Edgar Ramírez (Arch.dev) thank you so much for reacting to this issue so quickly! I will test the new tap-postgres version right away.
I agree! I'm curious what docs you were following. sdk.meltano.com, docs.meltano.com?
I was using both. It's possible that there is a snippet somewhere that shows you how to install plugins with extra dependencies, but if there is one, I missed it
@Edgar Ramírez (Arch.dev) I tried 0.0.12 but I'm unfortunately getting the same error, nothing changed :( Not a meltano or poetry expert by any means, but shouldn't the faker extra dep be listed here perhaps?
I created a PR that fixed the issue: https://github.com/MeltanoLabs/tap-postgres/pull/496
5 Views
Open in Slack
PreviousNext
Powered by