Hi Everybody, 👋. i am new to this, and not sure how to progress any further. But I am trying to use the tap-spreadsheets-anywhere to load some online csv's into postgres just for fun. I know the csv as a few null records but not sure how to exclude them. I tried using schema_overrides to set the datatype but that doesn’t seem to work. I also tried not enforcing a primary key at the target but that too doesn’t seem to work. Below is my yaml file

default_environment: dev

project_id: 48bbee3f-b735-48e7-b07b-0f9f88ce4efc

environments:

- name: dev

- name: staging

- name: prod

plugins:

extractors:

- name: tap-spreadsheets-anywhere

namespace: tap_spreadsheets_anywhere

variant: ets

pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>

executable: tap-spreadsheets-anywhere

capabilities:

- catalog

- discover

- state

config:

tables:

- path: <https://open.barnet.gov.uk/download/exm5m/3nf/CityEV%202022_Q1%20data.csv>

name: city_ev

pattern: ""

start_date: '2017-01-01T00:00:00Z'

key_properties: [charging_event]

format: csv

sample_rate: 10

max_sampling_read: 2000

skip_initial: 1

universal_newlines: false

prefer_number_vs_integer: true

schema_overrides:

charging_event:

type: ['null', integer]

loaders:

- name: target-postgres

variant: meltanolabs

pip_url: pipelinewise-target-postgres

config:

host: host.docker.internal

port: 5432

user: postgres

dbname: postgres

add_metadata_columns: 'True'

primary_key_required: 'False'

transformers:

- name: dbt-postgres

variant: dbt-labs

pip_url: dbt-core~=1.3.0 dbt-postgres~=1.3.0

config:

host: host.docker.internal

user: postgres

port: 5432

dbname: postgres

schema: meltup

Hi! You could use mappers to exclude columns (null) or rows (__filter__). @Sven Balnojan have shared recently a few useful examples here

🙇🏾‍♂️ 💐🥂, thank you for 💡.