https://meltano.com/ logo
Channels
Powered by
#announcements
Title
# announcements
m

many-farmer-43903

12/22/2020, 3:20 PM
currently working through containerization of my meltano project using custom plugins (everything ran fine there), and running into a failure during the 
install
when it is trying to install my custom plugin, any general guidance on troubleshooting? I feel like I'm going in circles
Copy code
Installing loader 'target-s3-jsonl'...
ERROR: ./target-s3-jsonl is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with svn+, git+, hg+, or bzr+).
✅ 1
general project structure
Copy code
data-integration/
  meltano.yml
  DockerFile
  ...
  target-s3-jsonl/
     target_s3_jsonl/__init__.py
meltano.yml snipped:
Copy code
loaders:
  - name: target-s3-jsonl
    namespace: target_s3_jsonl
    pip_url: -e ./target-s3-jsonl
    executable: target-s3-jsonl
    capabilities:
    - catalog
    - discover
    - properties
    settings:
    - name: aws_access_key_id
    - name: aws_secret_access_key
    - name: s3_bucket
    - name: s3_key_prefix
everything worked file when I ran
Copy code
meltano install loader target-s3-jsonl
so part of me thinks my 
pip_url
is not defined correctly
r

ripe-musician-59933

12/22/2020, 5:15 PM
@many-farmer-43903 This appears to indicate that the 
target-s3-jsonl
directory is missing when 
meltano install
is run as part of the Dockerfile: https://gitlab.com/meltano/files-docker/-/blob/master/bundle/Dockerfile#L12. As you can see there, only 
meltano.yml
is copied before running `meltano install`: https://gitlab.com/meltano/files-docker/-/blob/master/bundle/Dockerfile#L11, and all other files are copied later: https://gitlab.com/meltano/files-docker/-/blob/master/bundle/Dockerfile#L21 Since you want 
target-s3-jsonl
to already be available when 
meltano install
is run, you'll have to add a new 
COPY
directive ahead of it:
Copy code
COPY ./target-s3-jsonl ./target-s3-jsonl/
I think that's the correct syntax for copying an entire directory's contents, but https://docs.docker.com/engine/reference/builder/#copy is a little confusing:
If 
<src>
is a directory, the entire contents of the directory are copied, including filesystem metadata.
Note
The directory itself is not copied, just its contents.
m

many-farmer-43903

12/22/2020, 5:54 PM
ahh, I think you're right
@ripe-musician-59933 that's exactly what it was, the auto-generated docker file needed to be updated to include those directories before the install ran, thanks for the help!
👍 1
Powered by