https://linen.dev logo
Join Slack
Powered by
How can I change the logging level for a mapper pl...
# troubleshooting
d
How can I change the logging level for a mapper plugin? In 
my meltano.yml
I have:
Copy code
mappers:
  - name: transform-field
    variant: transferwise
    pip_url: pipelinewise-transform-field
    executable: transform-field
    mappings:
    - name: null-huge-json-blobs
      ...
As soon as I add this mapper into a pipeline (
meltano run tap-postgres null-huge-blobs-in-event-log-json target-snowflake
) it logs every single singer message, include 
RECORD
types which include sensitive data I tried changing the logging level as described here https://sdk.meltano.com/en/latest/implementation/logging.html I added both of these to my Dockerfile, but it’s still logging at level info:
Copy code
ENV TRANSFORM_FIELD_LOGLEVEL warning
ENV NULL_HUGE_BLOBS_IN_EVENT_LOG_JSON_LOGLEVEL warning
Just realized I missed the suffix off of the second env, let me try with that fixed That didn’t help
a
Transferwise mapper might not support that input. Do you know?
They don't mention logging on the main repo, but they do have this in their repo: https://github.com/transferwise/pipelinewise-transform-field/blob/master/sample_logging.conf
I'm not exactly sure how to make use of that though. Someone else with more experience with Python logging configurations might have some better advice.
I tried changing the logging level as described here https://sdk.meltano.com/en/latest/implementation/logging.html
This one is maintained by the very good people at Pipelinewise - that one is not built on the SDK though, although it was the inspiration for our own mapper.
d
Would you recommend using the meltano one instead?
a
Probably so - but can you say a bit more about the use case you had in mind?
d
I’m replicating a table that has a json column. Some of these json values are larger than 16MB, which is too large to fit in snowflake. I want to either filter out those rows, or set the json to NULL or something like that
Being able to do so on values > 16MB would be ideal, but failing that, being able to target any rows with a certain value. So basically one of these options
Copy code
if size(event) > 16MB then remove row
if size(event) > 16MB then set event to NULL
if event_type = "big_event" then remove row
if event_type = "big_event" then set event to NULL
`
4 Views
Open in Slack
PreviousNext
Powered by