Hi all quick question about adding sqlfluff to a Meltano pro Meltano #troubleshooting

Hi all, quick question about adding sqlfluff to a ...

vadiem_janssens

08/15/2022, 2:19 PM

Hi all, quick question about adding sqlfluff to a Meltano project. I've added the following code to my

meltano.yml

file, howerver I can't get sqlfluff to work.

Copy code

utilities:
    - name: sqlfluff
      variant: sqlfluff
      pip_url: sqlfluff[dbt]==1.2.1 sqlfluff-templater-dbt==1.2.1
      executable: sqlfluff

I've also created a

.sqlfluff

file with the following contents:

Copy code

[sqlfluff]
templater = dbt
dialect = postgres

After that, I run

meltano install

and

meltano invoke sqlfluff lint -v

, but I keep getting the following error

Copy code

Starting in sqlfluff version 0.7.0 the dbt templater is distributed as a separate python package. Please pip install sqlfluff-templater-dbt to use it.
Error loading config: Requested templater 'dbt' which is not currently available. Try one of raw, jinja, python, placeholder

Anyone has an idea what might cause this? There is no mention of adding the

dbt

templater separately in https://hub.meltano.com/utilities/sqlfluff, so I don't understand why it doesn't work out of the box.

vadiem_janssens

08/15/2022, 2:43 PM

Apparantly, you need to add

dbt-core

to the list as well, like

Copy code

utilities:
    - name: sqlfluff
      variant: sqlfluff
      pip_url: dbt-core==1.2.0 dbt-postgres==1.2.0 sqlfluff[dbt]==1.2.1 sqlfluff-templater-dbt==1.2.1
      executable: sqlfluff

Now I'm getting the following error:

Copy code

Could not find profile named 'meltano'

What does 'profile' in this context mean?

vadiem_janssens

08/15/2022, 3:01 PM

I've ended up copy'ing the exact same setup as described here (https://github.com/meltano/squared/blob/main/data/.sqlfluff) and now it seems to work.

vadiem_janssens

08/15/2022, 3:02 PM

I do still get a new error though:

Copy code

Unable to do partial parsing because config vars, config profile, or config target have changed
Unable to do partial parsing because profile has changed
Unable to do partial parsing because env vars used in profiles.yml have changed

Not a clue what this means or how I can resolve it.

aaronsteers

08/15/2022, 5:15 PM

@pat_nadolny - Have you run into this specific issue before? "Unable to do partial parsing..."

pat_nadolny

08/15/2022, 5:43 PM

Theres an open issue related to this https://github.com/meltano/hub/issues/726 and a PR https://github.com/meltano/hub/pull/633 that looks like it will end up be a docs update for now, potentially with a smoother long term solution.

pat_nadolny

08/15/2022, 5:46 PM

@aaronsteers yes I get those warnings related to partial parsing but everything works so I never looked into it 🤷

aaronsteers

08/15/2022, 5:57 PM

@vadiem_janssens - Can you confirm if they are errors or warnings on your side? I know dbt will try to do partial parsing (faster) instead of full parsing, but partial parsing should not be strictly required.

aaronsteers

08/15/2022, 5:58 PM

Also - after running successfully once, I'd be interested to know if those warnings go away if no values have changed between invocations. https://docs.getdbt.com/reference/parsing#partial-parsing

pat_nadolny

08/15/2022, 7:03 PM

@aaronsteers I just tested mine and it gives the same message even after a successful run

aaronsteers

08/16/2022, 12:07 AM

Sounds like we may need to dive deeper into how dbt is attempting to perform this check. But important for this thread, it sounds like this is just a warning for you, yes? @vadiem_janssens - Let us know if you are experiencing this as an error, and if so, do you see any other messages that might be a clue to the failure cause/explanation?

vadiem_janssens

08/16/2022, 8:38 AM

Yes, it indeed looks like 'just' a warning and all files were linted properly.

vadiem_janssens

08/16/2022, 8:43 AM

Looking back at what I had to do yesterday, either the installation instructions need to be updated or the

install sqlfluff

command should do everything to get it in a working state. It was quite a painful experience to be honest, as I had to combine so many different resources to get it to work.

vadiem_janssens

08/16/2022, 8:48 AM

Also - after running successfully once, I'd be interested to know if those warnings go away if no values have changed between invocations.

No, the warning persists after runs.

vadiem_janssens

08/16/2022, 8:59 AM

Looking back at what I had to do yesterday, either the installation instructions need to be updated or the
install sqlfluff
command should do everything to get it in a working state.

I've added my recommendations to update the docs here: https://github.com/meltano/hub/pull/633#issuecomment-1216340315. It seems to work right now, thank you all for the help 🙂.

8 Views

Open in Slack

Previous Next