https://linen.dev logo
Join Slack
Powered by
I want to use a stream map to filter out rows wher...
# troubleshooting
d
I want to use a stream map to filter out rows where the ID is not a valid UUID. Is this possible?
r
I believe the best you can do with what is currently possible is something like
Copy code
stream_maps:
  <stream_name>:
    __filter__: >-
      len(row_id) == 36
      and row_id[8] == '-'
      and row_id[13] == '-'
      and row_id[18] == '-'
      and row_id[23] == '-'
      and row_id.strip('0123456789abcdef-') == ''
d
Makes sense. Thanks.
r
Please feel free to open issues/PRs if you think this is something that is worth improving, e.g. • exposing the 
re
module
Copy code
stream_maps:
    <stream_name>:
      __filter__: re.fullmatch('[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', row_id, re.IGNORECASE)
• exposing the 
uuid
module and a new exception handler function
Copy code
stream_maps:
    <stream_name>:
      __filter__: try_cast(row_id, uuid.UUID)
@Edgar Ramírez (Arch.dev) unless you have any better suggestions?
d
Nice.
So, stepping back to my use case. I have uuids in varchar fields in my source fields. Like 99.99% of them are valid, there's just a few test fields that weren't. I'm storing them with target-postgres in postgres uuid columns, which works with a schema override like this:
Copy code
schema:
      public-team:
        team_id:
          type: string
          format: uuid
Which is cool.
😎 2
But the whole thing fails on an invalid value. So then I can use your filter expression to just ignore them. I actually ended up just correcting the source data, but this is handy to know for future. And the tickets you raised would be helpful for future cases like this. Cheers.
👍 2
3 Views
Open in Slack
PreviousNext
Powered by