Hi all I m trying to containerize a project that runs tap hu Meltano #troubleshooting

Hi all! I'm trying to containerize a project that ...

claire_b.

07/19/2023, 6:18 PM

Hi all! I'm trying to containerize a project that runs tap-hubspot and target-bigquery with gcs as a state backend. I keep getting an error to the effect of " '' is not a valid state backend." I've tried passing state backend information via an env file with no luck. Could this be because the base meltano docker image doesn't contain the extras needed to use state backends? Or am I not passing in my state backend information correctly?

luke_rodgers

07/19/2023, 6:25 PM

can you share your relevant

meltano.yml

config? also what does

requirements.txt

look like?

claire_b.

07/19/2023, 6:26 PM

Will do in just a moment!

claire_b.

07/19/2023, 6:28 PM

version: 1

default_environment: dev

project_id: 4a612e70-f840-4fbd-bcef-ed4d0cfc8ca4

environments:

- name: dev

- name: staging

- name: prod

plugins:

extractors:

- name: tap-hubspot

variant: potloc

pip_url: git+<https://github.com/potloc/tap-hubspot.git>

select:

- '!list_contacts_v1.*'

- workflows_v3*.*

- web_analytics_v3.*

- web_analytics_deals_v3*.*

- web_analytics_contacts_v3*.*

- properties*.*

- associations*.*

- analytics*.*

- calls*.*

- campaign*.*

- companies*.*

- contacts*.*

- deals*.*

- forms_v3*.*

- lists_v1*.*

- marketing_emails_v1*.*

- meetings*.*

- owners*.*

state_backend: '[cloud storage bucket]'

loaders:

- name: target-bigquery

variant: z3z1ma

pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>

config:

credentials_path:

[credential path]      project: [proj]

dataset: pre_container_test

location: US

fail_fast: false

timeout: 300

column_name_transforms:

lower: true

quote: false

denormalized: false

generate_view: true

flattening_enabled: false

flattening_max_depth: 0

claire_b.

07/19/2023, 6:28 PM

requirements.txt is empty...should I put in meltano[gcs]?

claire_b.

07/19/2023, 6:29 PM

I had previously had trouble with pip vs pipx installs re: requirements.txt

luke_rodgers

07/19/2023, 6:30 PM

i think you will need that, yes also - maybe it’s just a YAML indentation problem? removing the leading tabs/spaces from the

state_backend

line (could also just be slack formatting that makes it appear that way)

claire_b.

07/19/2023, 6:30 PM

It's just slack (and my slapdash redacting of some stuff)

claire_b.

07/19/2023, 6:30 PM

I'll try again with meltano[gcs] in requirements.txt

luke_rodgers

07/19/2023, 6:31 PM

do you have the actual bucket uri under a

uri

key?

luke_rodgers

07/19/2023, 6:31 PM

that is required, i believe

claire_b.

07/19/2023, 6:31 PM

Yes I do

claire_b.

07/19/2023, 6:39 PM

Copy code

Run invocation could not be completed as block failed: Cannot start plugin tap-hubspot: '' is not a valid StateBackend

claire_b.

07/19/2023, 6:39 PM

I added meltano[gcs] to my requirements file and got the same error

luke_rodgers

07/19/2023, 6:41 PM

i still think your YAML format seems off it appears that you have the state backend nested under the plugins key

extractors

and

loaders

should be nested under the

plugins

key try moving

state_backend

to the top of the file

claire_b.

07/19/2023, 6:42 PM

Thank you so much for your help! I'm giving that a go now.

claire_b.

07/19/2023, 6:49 PM

Same error, unfortunately

luke_rodgers

07/19/2023, 6:57 PM

is it possible that you have a unicode zero-width space char or some other undesirable char at the start of your gcs bucket url?

edgar_ramirez_mondragon

07/19/2023, 7:07 PM

Have you perhaps set the

MELTANO_DATABASE_URI

claire_b.

07/19/2023, 7:07 PM

I don't think I have...should I?

claire_b.

07/19/2023, 7:08 PM

I retyped the url, and it didn't seem to help

edgar_ramirez_mondragon

07/19/2023, 7:21 PM

I don't think I have...should I?

No, not really. So, the

state_backend

struct should be at the same level as

environments

and

plugins

and that doesn't seem to be the case in your

meltano.yml

. Should look something like this:

Copy code

version: 1
default_environment: dev
project_id: 4a612e70-f840-4fbd-bcef-ed4d0cfc8ca4
environments:
- name: dev
- name: staging
- name: prod
state_backend:
  uri: ...
plugins:
  ...

Also, make sure to remove any

MELTANO_STATE_*

entries from your

.env

file

claire_b.

07/19/2023, 7:29 PM

Thank you both! The docker container takes a minute to rebuild

claire_b.

07/19/2023, 7:46 PM

This is perhaps a related question: how should I get my api key into the container without hard coding it? I'm not super familiar with docker.

claire_b.

07/19/2023, 7:46 PM

Update: The app seems to recognize my state backend! Thank you so much!

edgar_ramirez_mondragon

07/19/2023, 7:58 PM

This is perhaps a related question: how should I get my api key into the container without hard coding it? I'm not super familiar with docker.

I'm not sure how you're running your container, but

docker run

has a couple of options: •

--env

-e

one for each env var you want to set. For example

docker run -e TAP_HUBSPOT_CLIENT_ID=... -e TAP_HUBSPOT_CLIENT_SECRET=...

•

--env-file

you point it to a dotenv file. For example

docker run --env-file .env

claire_b.

07/19/2023, 7:58 PM

Solid thank you!

2 Views

Open in Slack

Previous Next