https://linen.dev logo
Join Slack
Powered by
I am trying to add the pipelinewise-target-s3-csv ...
# troubleshooting
m
I am trying to add the pipelinewise-target-s3-csv to my meltano docker enivornment. I am able to complete the install successfully within my venv (see steps below), but then I get this error message when testing via INVOKE "docker run -v $(pwd):/project -w /project meltano/meltano invoke pipelinewise-target-s3-csv ": Error: "Executable 'pipelinewise-target-s3-csv' could not be found. Loader 'pipelinewise-target-s3-csv' may not have been installed yet using 
meltano install loader pipelinewise-target-s3-csv
, or the executable name may be incorrect." But when I check my meltano.yml, I have the following showing the executable name is the same. - name: pipelinewise-target-s3-csv namespace: pipelinewise_target_s3_csv pip_url: pipelinewise-target-s3-csv executable: pipelinewise-target-s3-csv # settings: # - name: s3_bucket config: s3_bucket: sample_bucket s3_key_prefix: sampley_key_prefix delimiter: ',' quotechar: '"' add_metadata_columns: true Install Steps: docker run -v $(pwd):/project -w /project meltano/meltano add --custom loader pipelinewise-target-s3-csv docker run -v $(pwd):/project -w /project meltano/meltano install loader pipelinewise-target-s3-csv Any ideas on how to resolve this? Thanks!
d
@mike_planting I think the executable name is 
target-s3-csv
, without the 
pipelinewise-
prefix that's in the package name
If you change that in 
meltano.yml
, it may just work 🙂
m
that did it.... thank you. I have been working on this for a few hours 😞. One other thing, with non-custom loaders, the meltano.yml has a config tree with various key pair values... can I use the same structure for settin s3 bucket, delimiters,, etc. Or should I flip to use a settings tree with "-name". Hopefully that made sense
So this:
settings: - name: s3_bucket type: string
or this:
config:       s3_bucket: bucketname (edited)
d
If you're comfortable having all settings in 
meltano.yml
, you can just use 
config
and omit 
settings
. If you'd like to configure some setting through an env var (or 
.env
) , it should have a definition under 
settings
. 
settings
is used to tell Meltano what settings a plugin supports, 
config
is one way of providing values for those settings
m
So helpful. Thanks!
Hi Douwe. Wondering if you can help me again. I am not trying to go from tap-mysql to pipelinewise-target-s3-csv, and then, pipelinewise-tap-s3-csv to target-redshift. tap-mysql to pipelinewise-target-s3-csv is working, but not pipelinewise-tap-s3-csv to target-redshift. Do you know of any examples that I can reference?
I can get to the documentation on git and pipelinewise, but I think my translation into the meltano.yml is the problem
Here is what my meltano.yml looks like so far
Copy code
extractors:

  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
    config:
      host: samplemysqlhost
      user: samplemysqluser
    select:
    - appdb-plans.*
    metadata:
      '*':
        replication-method: INCREMENTAL
        replication-key: id
      appdb-plans:
        replication-method: INCREMENTAL
        replication-key: id

  - name: pipelinewise-tap-s3-csv
    namespace: pipelinewise_tap_s3_csv
    pip_url: pipelinewise-tap-s3-csv
    executable: tap-s3-csv
    config:
      bucket: dashboard
      start_date: "2000-01-01"  
      tables: 
              [{
                search_prefix: "di-meltano",
                search_pattern: "di-meltano/appdb-plans/-appdb-plans.csv",
                table_name: "appdb-plans",
                key_properties: ["id"],
                delimiter: ","
              }]
              
  loaders:

  - name: target-redshift
    variant: transferwise
    pip_url: pipelinewise-target-redshift
    config:
      host: samplehost
      user: sampleuser
      dbname: prod
      s3_bucket: dashboard
      s3_key_prefix: di-meltano
      default_target_schema: meltano
      add_metadata_columns: true
      
  - name: pipelinewise-target-s3-csv
    namespace: pipelinewise_target_s3_csv
    pip_url: pipelinewise-target-s3-csv
    executable: target-s3-csv
    config:
      s3_bucket: dashboard
      s3_key_prefix: di-meltano
      naming_convention: "di-meltano/{stream}/-{stream}*.csv"
      delimiter: ','
      quotechar: '"'
      add_metadata_columns: true

schemas:

  - source_schema: "di-meltano" # This is mandatory, but can be anything in this tap type
    target_schema: "meltano" # Target schema in the destination Data Warehouse

    # List of CSV files to destination tables
    tables:

      # Every file in S3 bucket that matches the search pattern will be loaded into this table
      - table_name: "appdb-plans"
        s3_csv_mapping:
          search_pattern: "di-meltano/appdb-plans/di-meltano-appdb-plans.csv" # Required.
          # search_prefix: "di-meltano"                        # Optional
          # key_properties: ["id"]                   # Optional
          # delimiter: ","                           # Optional. Default: ','

        # OPTIONAL: Load time transformations
        #transformations:
        #  - column: "last_name"            # Column to transform
        #    type: "SET-NULL"               # Transformation type
d
@mike_planting Before we dig into the s3-csv stuff, any reason why you can't go straight from tap-mysql to target-redshift?
m
Ya, i would like to store our bin log results to s3 for replay purposes
... into redshift
So in case we lose our redshift data, then we will have the binlog archived in the CSVs generated by tap-mysql to pipelinewise-target-s3-csv
but it is tricky to load into redshift from pipelinewise-tap-s3-csv and that is where i having the problems
d
@mike_planting Are you seeing a particular error from tap-s3-csv?
m
let me check
d
Note that the 
schemas
stuff you have at the end of 
meltano.yml
is not going to be picked up by Meltano since that's PipelineWise specific
Does 
meltano config pipelinewise-tap-s3-csv
look like what you'd expect from https://github.com/transferwise/pipelinewise-tap-s3-csv#configuration
m
these are teh only things that look suspicious
Copy code
meltano                       | DEBUG Created configuration at /project/.meltano/run/elt/s3_to_redshift_210603_15/fea0bd26-eda4-4c1b-bcc9-2e90ddb48d45/tap.config.json
meltano                       | DEBUG Could not find tap.properties.json in /project/.meltano/extractors/pipelinewise-tap-s3-csv/tap.properties.json, skipping.
meltano                       | DEBUG Could not find tap.properties.cache_key in /project/.meltano/extractors/pipelinewise-tap-s3-csv/tap.properties.cache_key, skipping.
meltano                       | DEBUG Could not find state.json in /project/.meltano/extractors/pipelinewise-tap-s3-csv/state.json, skipping.
then i have debug logging results... but nothing jumps out
it is similar, the are minor formatting issues, but generally the same
it doesn't say "json" though
d
Copy code
capabilities:
- discover
- properties
- state
Can you add this to your 
tap-s3-csv
definition?
m
Copy code
{
  "bucket": "samplebucket",
  "start_date": "2000-01-01",
  "tables": [
    {
      "search_prefix": "di-meltano",
      "search_pattern": "di-meltano/app-plans/di-meltano-capp-plans.csv",
      "table_name": "app-plans",
      "key_properties": [
        "id"
      ],
      "delimiter": ","
    }
  ]
}
i dont have any capabilities
d
You should add some 🙂
The tap has those capabilities, but Meltano doesn't currently know it does
Which may be why the tap isn't getting called as it expects to, and ends up doing nothing
m
so should it be nested under config?
d
No, at the top-level of the plugin
So 
capabilities
key at the same level as 
config
m
and do i need to call out any specifics for these 3 configs?
d
No, just add the capabilities and see if that changes the behavior you're seeing
Meltano will know what to do, if it knows the tap supports those features
m
ok, so now that allowed the table to be created, but no data imported
d
OK, that's progress 😄
I wonder if the search pattern isn't matching anything
m
ya, i guess there are errors during transform, trying to stuff decimal into an integer
it should match that data type from source when created the destination table right?
d
Yeah, but that will depend on how good the tap is at deriving types from what it's seeing in the (untyped) CSV
Are you seeing new error messages now?
m
i see that meltano force integer when it encounters float, is there a way to globally map float to float, or float to decinal?
d
I think that's the tap's behavior, not Meltano's
m
i see, the csv does declare a datatype
d
Meltano doesn't have much say over how the tap decides to read the source and determine the schema. But you may be able to override the schema using https://meltano.com/docs/plugins.html#schema-extra
m
ok, thanks for the help!
d
I wonder if you could use that knowledge to have some influence on the detected types
Looks like the overridden schemas are respected, since this reads directly from the catalog: https://github.com/transferwise/pipelinewise-tap-s3-csv/blob/69f94e81ebe26386a80cf04076671c8bad1ff53b/tap_s3_csv/__init__.py#L66
Let me know if you're having trouble figuring out the right syntax 🙂
m
So would this go under the extractor or the loader? If under the extractor, then it seems like we would still run into the same problem.
i put this under the pipelinewise-tap-s3-csv and that worked, thanks again
d
That's great to hear!
5 Views
Open in Slack
PreviousNext
Powered by