Hi. I'm trying to set up the simpliest meltano project possible. I thought of one that doesn't require credentials, for example parsing a csv file publicly available on the web. tap-csv seems to work only with local files, I've tried tap-sreadsheets-anywhere. As I'm new, I struggled a bit to understand how the configuration works (this plugin requires a json array in one of the settings). Once done I've tried to run it and it gives the following error

"block violates set requirements: Found no end in block set!"

I've googled that without any success. Does this error look familiar to you?

@mehdi_sidi_boumedine if you’re looking for a simple project checkout our jaffle shop one https://meltano.slack.com/archives/C04632W0HT2/p1680301581525659

@mehdi_sidi_boumedine also check out the hub page for some example configs. Can you share the config you're using thats leading to the error?

@pat_nadolny sure, here you go {"path": "https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2021-financial-year-provisional/Download-data/", "name": "financial_table", "pattern": "*.csv", "start_date": "2017-05-01T000000Z"} configured using interactive mode and properly reflected in the yml

@mehdi_sidi_boumedine for your public source I was able to pull a file using this config. You need to configure each CSV file as a table since they have different schemas and since the

path

and

pattern

configs get combined to create the request URL for http then the

*.csv

pattern doesnt work. For context the

*.csv

would work for something like S3 or local files but not for a URL since it has no way of finding those files without the explicit name. Theres also a chance you need to manually clear your

.meltano/run/tap-spreadsheets-anywhere/

directory if its caching an old catalog that was empty from your previous failed attempts.

config:
      tables:
      - path: <https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2021-financial-year-provisional/Download-data/>
        format: csv
        start_date: '2017-05-01T00:00:00Z'
        key_properties: [industry_code_anzsic06]
        name: financial_table
        pattern: "annual-enterprise-survey-2021-financial-year-provisional-csv.csv"

@pat_nadolny thanks so much, this helps understanding the behavior, and that was really my intention to target that specific file. Where do you find the outcome, is it downloaded locally?

No problem! I ran

meltano invoke tap-spreadsheets-anywhere

to test that it worked. It takes ~30 seconds to start up probably because of downloading then its starts writing to the console records like

...
{"type": "RECORD", "stream": "financial_table", "record": {"year": 2015, "industry_aggregation_nzsioc": "Level 3", "industry_code_nzsioc": "CC71", "industry_name_nzsioc": "Primary Metal and Metal Product Manufacturing", "units": "Percentage", "variable_code": "H38", "variable_name": "Margin on sales of goods for resale", "variable_category": "Financial ratios", "value": "9", "industry_code_anzsic06": "ANZSIC06 groups C211, C212, C213, and C214", "_smart_source_bucket": "<https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2021-financial-year-provisional/Download-data/>", "_smart_source_file": "annual-enterprise-survey-2021-financial-year-provisional-csv.csv", "_smart_source_lineno": 29088}}
{"type": "RECORD", "stream": "financial_table", "record": {"year": 2015, "industry_aggregation_nzsioc": "Level 3", "industry_code_nzsioc": "CC71", "industry_name_nzsioc": "Primary Metal and Metal Product Manufacturing", "units": "Percentage", "variable_code": "H39", "variable_name": "Return on equity", "variable_category": "Financial ratios", "value": "11", "industry_code_anzsic06": "ANZSIC06 groups C211, C212, C213, and C214", "_smart_source_bucket": "<https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2021-financial-year-provisional/Download-data/>", "_smart_source_file": "annual-enterprise-survey-2021-financial-year-provisional-csv.csv", "_smart_source_lineno": 29089}}
{"type": "RECORD", "stream": "financial_table", "record": {"year": 2015, "industry_aggregation_nzsioc": "Level 3", "industry_code_nzsioc": "CC71", "industry_name_nzsioc": "Primary Metal and Metal Product Manufacturing", "units": "Percentage", "variable_code": "H40", "variable_name": "Return on total assets", "variable_category": "Financial ratios", "value": "3", "industry_code_anzsic06": "ANZSIC06 groups C211, C212, C213, and C214", "_smart_source_bucket": "<https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2021-financial-year-provisional/Download-data/>", "_smart_source_file": "annual-enterprise-survey-2021-financial-year-provisional-csv.csv", "_smart_source_lineno": 29090}}
...