With pipelinewise-target-redshift, I have found that not all columns denest (I am testing using Shopify Tap). Some do perfectly, however others stay fully nested. Has anyone encountered this before? It seems as though it is the columns that are not simply {....}, but [{.....}] Also: Quick note, I found that if you want to adjust the column sizing you can go into the redshift_target folder and in db_sync.py you can adjust the sizing by changing the values for the global variables at the top of the file.

@gunnar Looks like you've already filed an issue about the first point: https://github.com/transferwise/pipelinewise-target-redshift/issues/73 🙂 If you're seeing that behavior consistently, I assume denesting arrays (rather than objects) simply isn't implemented, and you may want to contribute it. As for the sizing, consider filing an issue (and a contribution) to make those configurable, since maintaining your own fork with some changes to

db_sync.py

is not going to be sustainable since you'll want to benefit from any upstream improvements!

Yes I am working on contributing since I believe I located the error. It is due to json column cells being wrapped in [ ] I filed an issue, but was hoping to reach out just in case anyone in this channel has already come across this. All columns not wrapped in [ ] seem to flatten correctly.