https://linen.dev logo
Join Slack
Powered by
Hey, does anyone have experience running FAL insid...
# troubleshooting
f
Hey, does anyone have experience running FAL inside dbt inside meltano?
v
Thought about this a while back didn't run with it, but if you want to go ahead! From https://meltano.com/discovery.yml
Copy code
transformers:
  - name: dbt
    namespace: dbt
    docs: <https://meltano.com/docs/transforms.html>
    repo: <https://github.com/dbt-labs/dbt-core>
    pip_url: >
      dbt-core~=1.0.0
      dbt-postgres~=1.0.0
      dbt-redshift~=1.0.0
      dbt-snowflake~=1.0.0
      dbt-bigquery~=1.0.0
    settings:
      - name: project_dir
        value: $MELTANO_PROJECT_ROOT/transform
      - name: profiles_dir
        env: DBT_PROFILES_DIR
        value: $MELTANO_PROJECT_ROOT/transform/profile
      - name: target
        value: $MELTANO_LOAD__DIALECT
      - name: source_schema
        value: $MELTANO_LOAD__TARGET_SCHEMA
      - name: target_schema
        value: analytics
      - name: models
        value: $MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project
    commands:
      clean:
        args: clean
        description: Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)
      compile:
        args: compile --models $DBT_MODELS
        description: Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory.
      deps:
        args: deps
        description: Pull the most recent version of the dependencies listed in packages.yml
      run:
        args: run --models $DBT_MODELS
        description: Compile SQL and execute against the current target database.
      seed:
        args: seed
        description: Load data from csv files into your data warehouse.
      snapshot:
        args: snapshot
        description: Execute snapshots defined in your project.
      test:
        args: test
        description: Runs tests on data in deployed models.
Just copy this replace the dbt specefic stuff with fal, and you should be off to the races
f
will try now
t
@finn_frotscher curious what you’re wanting to do with fal that meltano + dbt alone can’t do. You can add any python script as a utility and just run dbt together using 
meltano run
- something like 
meltano run dbt:my_models my_python_script dbt:my_test
should work
f
@k_s
@taylor FAL atop DBT scales better with complexity, in our estimation. We can have various smaller scripts at specific points in our dbt dag and dont need to call them explicitly
@visch I adapted your script but i must have done it wrong. when running fal from the CLI i need both project-dir and profiles-dir.
Copy code
utilities:
  - name: fal
    namespace: fal
    pip_url: dbt-bigquery==1.0.0 fal
    executable: fal
    settings:
      - name: project_dir
        value: $MELTANO_PROJECT_ROOT/transform
      - name: profiles_dir
        env: DBT_PROFILES_DIR
        value: $MELTANO_PROJECT_ROOT/transform/profile
    commands:
      run:
        args: run --before
        description: wip
When i run 
meltano invoke fal run
, the settings dont get forwarded. i havent used this feature of meltano yet. am i doing it wrong?
v
I just copy pasted meltano's setup so not mine 😄 settings describe the config that's needed, config sets the configuration. See https://docs.meltano.com/guide/configuration You can make those defaults or use a 
config
object so something like
Copy code
utilities:
  - name: fal
    namespace: fal
    pip_url: dbt-bigquery==1.0.0 fal
    executable: fal
    settings:
      - name: project_dir
        value: $MELTANO_PROJECT_ROOT/transform
      - name: profiles_dir
        env: DBT_PROFILES_DIR
        value: $MELTANO_PROJECT_ROOT/transform/profile
    config: 
      project_dir: abc
      profiles_dir: blash
    commands:
      run:
        args: run --before
        description: wip
f
and i invoke the 
commands: run
with 
meltano invoke fal:run
?
v
🤷 I think so yeah, I haven't done it exactly but that seems right.
f
Copy code
(bi) ⟩meltano invoke fal:run                                                                                                                                                                                                                                                                                                                                                                                          .../elt/meltano
Could not read dbt run_results artifact
14:51:21  Found 0 models, 0 tests, 0 snapshots, 0 analyses, 396 macros, 0 operations, 1 seed file, 0 sources, 0 exposures, 0 metrics
16:51:22 | Starting fal run for following models and scripts:
its not picking up any of the scripts.
v
getting closer at least! it's running 😄
I don't know fal at all so I'd have to dive into to look at the setup
f
hahha honestly not. i bypassed the issue of missing config by hardcoding it in the 
run
command
Copy code
utilities:
  - name: fal
    namespace: fal
    pip_url: dbt-bigquery==1.0.0 fal
    executable: fal
    settings:
      - name: project-dir
        value: $MELTANO_PROJECT_ROOT/transform
      - name: profiles-dir
        env: DBT_PROFILES_DIR
        value: $MELTANO_PROJECT_ROOT/transform/profile
    config: 
      project-dir: $MELTANO_PROJECT_ROOT/transform
      profiles-dir: $MELTANO_PROJECT_ROOT/transform/profile
    commands:
      run:
        args: run --before --profiles-dir /Users/finn/code/data-cofenster/elt/meltano/transform/profile --project-dir /Users/finn/code/data-cofenster/elt/meltano/transform
        description: wip
but i wanted to test the whole system e2e before optimizing individual steps. but thank you, you have opened a new area for me. i’ll check it out
so the missing piece was
Copy code
// dbt_project.yml

vars: 
  "fal-scripts-path": ./fal_scripts
Open in Slack
PreviousNext
Powered by