https://linen.dev logo
Join Slack
Powered by
Hey does anyone know how to integrate running melt...
# troubleshooting
m
Hey does anyone know how to integrate running meltano via terminal to get the logs sent to GCP cloud logging? Example: 
meltano el tap-mongodb target-jsonl
logging.yaml
Copy code
version: 1

disable_existing_loggers: false

formatters:
  json:
    (): meltano.core.logging.json_formatter

root:
  level: INFO
  propagate: yes
  handlers: [console]


handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: json
    stream: "<ext://sys.stderr>"
I’m a bit confused how meltano is generating logs. Is it with python logging library? In python I can see logs using 
google-cloud-logging
library:
Copy code
import google.cloud.logging

client = google.cloud.logging.Client()
client.setup_logging()

<http://logging.info|logging.info>("*** INFO ***")
e
Hi @matt_elgazar! This is probably worth writing a short guide if it works: 1. Install 
google-cloud-logging
in the same virtualenv as Meltano (they should be compatible) 2. Write a custom python module that contains your logging handler callable 3. Set 
PYTHONPATH
to the directory containing the logging module 4. Check that Meltano is sending logs to cloud logging? https://gist.github.com/edgarrmondragon/feb148d50339dbbc2a6251707e13882f
m
@edgar_ramirez_mondragon Wow that looks amazing! I just tried it on two different machines and I’m getting stuck here 
meltano lock --all --update
Copy code
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Unable to configure handler 'google'
e
I see. You can probably get more details if you print any exception raised when trying to create the client:
Copy code
import sys

import google.cloud.logging
from google.cloud.logging.handlers import CloudLoggingHandler


def get_handler():
    """Return a handler based on the environment."""
    try:
        client = google.cloud.logging.Client()
     except Exception as e:
        print(f"Failed to create logging client: {e}", file=sys.stderr)
        raise
    return CloudLoggingHandler(client)
m
hey @edgar_ramirez_mondragon I don’t see any other error than 
Unable to configure handler 'google'
. If I run the python script 
python gcp_logging.py
it runs successfully
running 
meltano test
returns the google handler error. Are you not having this issue? I re-cloned a fresh meltano repo and ran the steps one by one on two different machines getting the same error
e
Ok, maybe your error's not on 
google.cloud.logging.Client()
, so you could try moving the handler creating to the try block:
Copy code
def get_handler():
    """Return a handler based on the environment."""
    try:
        client = google.cloud.logging.Client()
        return CloudLoggingHandler(client)
    except Exception as e:
        print(f"Failed to create logging client: {e}", file=sys.stderr)
        raise
With that I still get
Copy code
PYTHONPATH=$PWD meltano test                              
Failed to create logging client: Your default credentials were not found. To set up Application Default Credentials, see <https://cloud.google.com/docs/authentication/external/set-up-adc> for more information.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Unable to configure handler 'google'
because I haven't configured cloud logging
m
Are you sure it’s a problem with the python handler? I’ve tried a few different ways and still getting 
Unable to configure handler 'google'
. My gut is telling me it’s something that needs to be configured in logging.yaml
e
Do you have 
PYTHONPATH
set?
Could also be an import error
m
yep
Copy code
echo $PYTHONPATH
/home/melgazar9/meltano_projects/tap-yfinance
I also tried setting the PYTHONPATH to /usr/bin/python3
e
Ok, you try wrapping everything in the try block:
Copy code
def get_handler():
    """Return a handler based on the environment."""
    try:
        import google.cloud.logging
        from google.cloud.logging.handlers import CloudLoggingHandler

        client = google.cloud.logging.Client(project="<your project id>")
    except Exception as e:
        print(f"Failed to create logging client: {e}", file=sys.stderr)
        raise

    return CloudLoggingHandler(client)
m
same error. I don’t think it’s something with the python code? Reading this I see a setup for google logging https://docs.meltano.com/guide/logging/
“By default, meltano will look for this in a 
logging.yaml
” Google Cloud logging config
Copy code
version: 1
disable_existing_loggers: false

formatters:
  json:
    (): meltano.core.logging.json_formatter

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: json
    stream: "<ext://sys.stderr>"

root:
  level: INFO
  propagate: yes
  handlers: [console]
do I need to set the handler here to 
google.cloud.logging.handlers.CloudLoggingHandler
?
when I set it up like this:
Copy code
handlers:
  console:
    class: google.cloud.logging.handlers.CloudLoggingHandler
    level: INFO
    formatter: json
    stream: "<ext://sys.stderr>"
I get this error now: 
Unable to configure handler 'console'
e
Oh, I think 
CloudRun
captures stderr automatically so yeah that basic config should work there but if you want to explicitly use "Cloud Logging for Python", then you do need something like the gist. I got it working btw: • 
gcp_logging.py
Copy code
import sys


def get_handler():
    """Return a handler based on the environment."""
    try:
        import google.cloud.logging
        from google.cloud.logging.handlers import CloudLoggingHandler

        client = google.cloud.logging.Client(project="personal-warehouse-349804")
    except Exception as e:
        print(f"Failed to create logging client: {e}", file=sys.stderr)
        raise

    return CloudLoggingHandler(client)
• 
google.yaml
Copy code
version: 1
disable_existing_loggers: false

handlers:
  google:
    (): gcp_logging.get_handler

root:
  level: INFO
  handlers: [google]
message has been deleted
(working == sends logs from my machine to gcp)
m
Ok awesome trying now! When you say 
logging/google.yaml
you created a logging directory and put google.yaml in there?
Oh nvm saw you edited it!
e
Yeah, that's cause I'm using my dogfood project for which I have a dedicated folder with logging configs: https://github.com/edgarrmondragon/meltano-dogfood/tree/main/logging That filename should be whatever you set with 
meltano config meltano set cli.log_config
m
Ok, so I have my tap in 
meltano_projects/tap-yfinance
--- did you set your 
PYTHONPATH=meltano_projects/tap-yfinance
?
e
Ok so my project looks like this
Copy code
.
├── gcp_logging.py
├── logging
│   └── google.yaml
└── meltano.yml
And I run 
PYTHONPATH=$PWD meltano ...
from the root of the project. 
gcp_logging.py
should be in 
meltano_projects/tap-yfinance
in your case
m
dang I am still getting this! 
Unable to configure handler 'google'
I started from scratch and copied everything verbatim
If you try running it on a fresh gcp VM and a fresh meltano project do you get the same error as me? Here are my steps (please let me know if I’m missing anything): 1. git clone https://github.com/melgazar9/tap-yfinance.git 2. 
cd tap-yfinance
--- Create 
gcp_logging.py
and 
google.yaml
copying your above code snippets 3. 
pipx ensurepath
4. 
pipx inject meltano google-cloud-logging
5. 
meltano config meltano set cli.log_config google.yaml
6. 
export PYTHONPATH=$PWD meltano test
7. 
meltano lock --all --update
Copy code
Unable to configure handler 'google'
• Repeat the above but try putting 
google.yaml
in the 
logging/
directory. • Try the same steps as above on both MacOS and Ubuntu linux.
Maybe you have something pre-configured on your local machine?
e
What's the output of 
echo $PYTHONPATH
?
m
/Users/melgazar9/meltano_projects/tap-yfinance
e
If you wrap the entire thing in a try/except do you see anything printed?
Copy code
import sys


def get_handler():
    """Return a handler based on the environment."""
    try:
        import google.cloud.logging
        from google.cloud.logging.handlers import CloudLoggingHandler

        client = google.cloud.logging.Client(project="<your-project-name>")
        return CloudLoggingHandler(client)
    except Exception as e:
        print(f"Failed to create logging client: {e}", file=sys.stderr)
        raise
m
@edgar_ramirez_mondragon no it’s the exact same message 
Unable to configure handler 'google'
. I started a branch new VM on gcp (and also locally on MacOS), installed git, meltano, pip, pipx, and followed these steps directly getting the same error above: https://meltano.slack.com/archives/C01TCRBBJD7/p1701890466285389?thread_ts=1701828771.518699&amp;cid=C01TCRBBJD7
I posted this question on stack overflow, paid someone on fiverr, and now it's in meltano troubleshooting with no solution. It's nobody's fault but I don't think something as simple as cloud logging should be this complicated 😂
e
I got it working on a gcp vm 🙂:
m
Hey @edgar_ramirez_mondragon awesome! Is there anything different in here that you did differently other than the code in your above screenshot? https://gist.github.com/edgarrmondragon/feb148d50339dbbc2a6251707e13882f I am still getting 
Unable to configure handler 'google'
from this line:
Copy code
cli:
  log_config: google.yaml
e
m
Ok awesome thanks. Trying again now
@edgar_ramirez_mondragon I am also getting 
sent all pending logs
but I don't see any logs in GCP cloud logging. Are you able to see the actual logs in gcp cloud logging?
e
No, logs didn't reach cloud logging. Grabbing lunch atm but probably worth taking a look at
For an application to write logs by using the Cloud Logging library for Python, the service account for the underlying resource must have the Logs Writer (roles/logging.logWriter) IAM role. Most Google Cloud environments automatically configure the default service account to have this role.
https://cloud.google.com/logging/docs/setup/python#run-gce
m
Sure np. I read through that many times and have it working for pure python apps. I just don't know how to integrate that documentation with meltano
If you add something like this it should 100% work as expected. Is there a place somewhere we can call 
setup_logging()
using meltano? Maybe it's something under the hood in singer?
Copy code
import google.cloud.logging
 
client = google.cloud.logging.Client()
client.setup_logging()
 
<http://logging.info|logging.info>("*** INFO ***")
e
I don't think calling 
setup_logging()
is necessary if you're using python's logging library. I just followed the instructions in the link
Use the Compute Engine default service account or another service account of your choice, and select Allow full access to all Cloud APIs in the Identity and API access section.
I also changed the level in 
google.yaml
to DEBUG to get more logs:
Copy code
version: 1
disable_existing_loggers: false

handlers:
  google:
    (): gcp_logging.get_handler
    project: "personal-warehouse-349804"
    logger_name: meltano
root:
  level: DEBUG
  handlers: [google]
And it worked!
When I add 
setup_logging()
to the handler module
Copy code
import sys


def get_handler(*, project=None, logger_name=None):
    """Return a handler based on the environment."""
    try:
        import google.cloud.logging
        from google.cloud.logging.handlers import CloudLoggingHandler

        client = google.cloud.logging.Client(project=project)
        client.setup_logging()  # <- HERE
        return CloudLoggingHandler(client, name=logger_name)
    except Exception as e:
        print(f"Failed to create logging client: {e}", file=sys.stderr)
        raise
I just got duplicate init messages:
Copy code
Waiting up to 5 seconds.
Sent all pending logs.
Waiting up to 5 seconds.
Sent all pending logs.
So I think 
CloudLoggingHandler
calls 
setup_logging()
under the hood.
m
Yep getting the exact same thing, still no logs 😞 Been playing with it for the last few hours
3 Views
Open in Slack
PreviousNext
Powered by