https://linen.dev logo
Join Slack
Powered by
Hey <@U06CASRQ0H0> were you ever able to figure ou...
# troubleshooting
m
Hey @Edgar Ramírez (Arch.dev) were you ever able to figure out how to get meltano logs sent to gcp cloud logging? I can do it from a local machine using a regular .py script but it does not work with meltano (or meltano running inside a docker container). Any ideas how we can translate this simple .py script to persist across all meltano logs?
Copy code
import google.cloud.logging
client = google.cloud.logging.Client()
client.setup_logging()

<http://logging.info|logging.info>("*** INFO ***")
logging.critical("*** CRITICAL ***")
I like that gcp cloud logging captures the severity level, which I haven’t figured out how to do using meltano / docker -> gcp.
e
Hey @matt_elgazar 👋! The gist https://gist.github.com/edgarrmondragon/feb148d50339dbbc2a6251707e13882f still works for me. The severity level seems to be translated automatically:
m
Hey @Edgar Ramírez (Arch.dev) OK a couple of things: • I’m able to get the basics working in that doc on Ubuntu but not on Mac (m1 chip). It fails at 
meltano test
with the error 
Unable to configure handler 'google'
• On Ubuntu, all steps in that doc work. I am able to see logs from running 
meltano test
but not when running an actual tap. I assume this may have something to do with the log messages being too big / long? Or something to do with google cloud storage? Take my 
tap-yfinance
tap. Running 
meltano el tap-yfinance target-jsonl --select futures_tickers
will actually complete the tap but the logs don’t get sent to GCP cloud logging. I get this error:
Copy code
Program shutting down now, attempting to send 12724 queued log entries to Cloud Logging...
Waiting up to 5 seconds.
Failed to send 12203 pending logs.
I thought maybe this could have something to do with invalid credentials or issue reading .env variables, so I hard coded every env variable to rule that out. Got the same error 😞
Actually, in addition to this on Ubuntu, when I uncomment the 
state-backend
section, I get the following error:
Copy code
meltano el tap-yfinance target-jsonl --select futures_tickers
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

ELT could not be completed: Cannot start extractor: module 'google.cloud' has no attribute 'storage'.
For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag.
Note that you can also check the generated log file at '/home/oem/scripts/dev/ds-mono-repo/projects/financial_elt/tap-yfinance/.meltano/logs/elt/2024-02-15T052410--tap-yfinance--target-jsonl/75cff2c3-81cc-4fb8-9717-e0b361aa91fa/elt.log'.
For more information on debugging and logging: <https://docs.meltano.com/reference/command-line-interface#debugging>
Program shutting down, attempting to send 14 queued log entries to Cloud Logging...
Waiting up to 5 seconds.
Sent all pending logs
@Edgar Ramírez (Arch.dev) Ok after trying multiple things, can you try running any tap with a longer message and see if it works with gcp cloud logging?
e
I ran 
meltano run tap-getpocket target-jsonl --full-refresh
from my machine:
Copy code
Program shutting down, attempting to send 3 queued log entries to Cloud Logging...
Waiting up to 5 seconds.
Sent all pending logs.
It's not as many logs as you were trying to send but 
12724
still doesn't look like that large a number 🤔. That said, I'm pretty sure there's an even easier way to send logs to cloud logging with the Ops Agent, 
logging.FileHandler
and 
meltano.core.logging.json_formatter
. I've yet to try that though 😅
when I uncomment the 
state-backend
section, I get the following error:
That means you probably need to install 
meltano[gcs]
FYI I updated the gist to use Ops Agent. Much cleaner approach :)
m
Hey @Edgar Ramírez (Arch.dev) awesome! A couple of things: • The original steps ended up working for me for most of the logs. It only worked when I changed the name from 
google.yaml
to 
logging.yaml
. I guess there was an issue with the name 
google.yaml
, which is weird because it’s specified in meltano.yml. • The new steps aren’t working for me (I’m on a local ubuntu machine, not a GCP VM). Are you running this on a gcp vm? Do you think this would work on any machine (not necessarily a GCP vm)?
The initial steps still returned failed logs at times. Most of the logs went through but I still occasionally saw messages like the screenshot below
Only thing is from the meltano logs every once in a while I’ll get this error because the log messages is too big. Not the end of the world but a solid start 👍
5 Views
Open in Slack
PreviousNext
Powered by