Does anyone know how to let a column entry not be mandatory in

tap-spreadsheet-anywhere

? ie if a field is missing in a csv file ti should default to null: https://hub.meltano.com/extractors/tap-spreadsheets-anywhere/ I've been looking at "field_names" without luck

Hope this helps (these settings are used with our Matatika variant

pip_url: git+<https://github.com/Matatika/tap-spreadsheets-anywhere@v0.2.0>

, but I don't think that should matter here)

settings:
  - name: tables
    kind: array
    description: A collection of related data organized in rows and columns.
    required: false
    value: >-
      [{
        "path":"<https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1182360/>",
        "name":"weekly_fuel_prices",
        "pattern":"CSV_040923.csv",
        "field_names":["Date","ULSP_per_litre","ULSD_per_litre","ULSP_duty","ULSD_duty","ULSP_vat_pc","ULSD_vat_pc"],
        "skip_initial":18,
        "start_date":"2003-09-06T00:00:00Z",
        "key_properties":["date"],
        "format":"csv"
      }]
    label: Tables

Thanks, so missing rows will be nulls?

No, they won't be in the stream at all.

okay thanks for the example 😄 I am still looking for a way to insert null instead. Maybe I should handle that elsewhere

Yeah, probably custom mapper to add the empty columns you want? https://docs.meltano.com/guide/mappers/ (see ip address in this page)

yeah perfect. Thanls @aaron_phethean 😄