tada Wrote my first <https dev to sburwash data analytics a Meltano #show-us-what-you-got

:tada: Wrote my first <blog post> :tada: to expl...

Stéphane Burwash

01/06/2023, 3:33 AM

🎉 Wrote my first blog post 🎉 to explain how to integrate Elementary into your meltano project (and why you should do it). Elementary is a dbt plugin that allows you to bring data integrity to the next level with anomaly detection and report generation. Would love any comments / feedback ! I expect to write more of these blog posts (as the title suggests) as we include more and more plugins into the meltano utilities 😄

josh_lloyd

03/22/2023, 5:29 PM

@Stéphane Burwash I tried following the instructions in your post. I’ve installed package version 0.7.3 and cli version 0.7.3 but when I try to run

meltano run elementary:monitor-report

I get and error that says,

Could not find adapter type snowflake!

When I added

elementary-data[snowflake]==0.7.3

to the

pip_url

line in the

meltano.yml

file it worked fine. Just FYI

Stéphane Burwash

03/22/2023, 6:05 PM

Thanks for notifying me @josh_lloyd! I'll look into it. Please let me know how you like the extension 😄

josh_lloyd

03/22/2023, 6:19 PM

So far yes! I’m really only just coming to know elementary so I’ve barely scratched the surface, but so far I’ve discovered that poetry’s dependency resolver won’t allow

meltano

and

elementary-data

to be installed in the same environment due to dependency conflicts, so installing it via meltano’s plugin seems to be the most optimal way to run it.

josh_lloyd

03/22/2023, 6:20 PM

The one thing I’d like to see support for is hosting a static website on S3 with the report. Currently I’m trying to find work arounds, which isn’t necessarily a deal breaker but takes a bit more time.

Stéphane Burwash

03/22/2023, 6:21 PM

Could this help? https://docs.elementary-data.com/guides/share-observability-report/host-on-s3

josh_lloyd

03/22/2023, 6:22 PM

yeah that’s what I’ve been looking at. Theres just not settings in the plugin natively for providing the

s3_bucket_name

and what have you

josh_lloyd

03/22/2023, 6:23 PM

at least not that’s obvious in the documentation …

josh_lloyd

03/22/2023, 6:23 PM

I’d like this all to run on a regular schedule from an ECS container that’s running this from a dagster implementation

Stéphane Burwash

03/22/2023, 6:24 PM

No there isin't - I didn't want to support everything in the plugin (too many combinations), so in our own implementation I specified a command directly in the meltano.yml

josh_lloyd

03/22/2023, 6:24 PM

yeah, that’s how I’m going about it too 🙂

Stéphane Burwash

03/22/2023, 6:24 PM

Copy code

commands:

          initialize:
            args: initialize
            executable: elementary_extension
          describe:
            args: describe
            executable: elementary_extension
          monitor-report:
            args: monitor-report
            executable: elementary_extension
          monitor-send-report:
            args: monitor-send-report
            executable: elementary_extension
          send-report-gcs:
            args: send-report
                  --google-service-account-path ${ELEMENTARY_GOOGLE_SERVICE_ACCOUNT_PATH}
                  --gcs-bucket-name ${ELEMENTARY_GCS_BUCKET_NAME}
                  --update-bucket-website true
                  --executions-limit 5

Stéphane Burwash

03/22/2023, 6:25 PM

Yeah it's not the cleanest, but at least that allows the plugin to remain "pure" and gives the user freedom to implement different features themselves

Stéphane Burwash

03/22/2023, 6:25 PM

(Great idea on the static hosting btw - this is a feature I now use on the daily)

6 Views

Open in Slack

Previous Next