Hello fellow Meltano s users I m trying to extract data from Meltano #troubleshooting

Hello fellow Meltano's users! I'm trying to extra...

andrea_radaelli

03/15/2023, 11:10 AM

Hello fellow Meltano's users! I'm trying to extract data from SalesForce Marketing Cloud (with tap-exacttarget) and bring them on Google Cloud Storage (with target-gcs). I have selected only a few attributes on SFMC that I want to move, unfortunately I keep having this error: [error ] 'utf-8' codec can't decode byte 0xe0 in position 41: invalid continuation byte UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 41: invalid continuation byte I'm really out of my depth here 😅 but I think there's a problem with the format of some of my data. Since I can't modify my data source, is there a way to resolve the issue?

pat_nadolny

03/15/2023, 2:09 PM

Hey @andrea_radaelli! Can you share more info on what youre doing/seeing? Commands you ran, more snippets of the error, your meltano.yml (minus any sensitive credentials or anything).

pat_nadolny

03/15/2023, 2:12 PM

Some helpful common troubleshooting notes are in https://docs.meltano.com/guide/integration#troubleshooting

andrea_radaelli

03/15/2023, 2:19 PM

hello @pat_nadolny and thakyou for your help! this is my meltano.yml

Copy code

version: 1
default_environment: dev
project_id: <>
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-exacttarget
    variant: singer-io
    pip_url: tap-exacttarget
    config:
      batch_size: 99999999
      tenant_subdomain: <>
      start_date: '2023-02-01T00:00:00Z'
    select:
    - 'campaign.*'
    - 'content_area.*'
    - 'data_extension.*'
    - 'email.*'
    - 'event.*'
    - 'folder.*'
    - 'list.*'
    - 'list_send.*'
    - 'list_subscriber.*'
    - 'send.*'
    - 'subscriber.*'
  loaders:
  - name: target-gcs
    variant: datateer
    pip_url: git+<https://github.com/Datateer/target-gcs.git>
    config:
      bucket_name: meltano_sfmc
      credentials_file: <>

andrea_radaelli

03/15/2023, 2:24 PM

it runs for a while, skipping all the

data_extension.*

table as I want it to do. But then this error appears

andrea_radaelli

03/15/2023, 2:24 PM

```2023-03-15T142219.910010Z [info ] INFO 'data_extension.Lista email premium_Promo BTS_03092020' is not marked selected, skipping. cmd_type=elb consumer=False name=tap-exacttarget producer=True stdio=stderr string_id=tap-exacttarget 2023-03-15T142219.910010Z [info ] INFO 'data_extension.Store promo BTS IT_03092020' is not marked selected, skipping. cmd_type=elb consumer=False name=tap-exacttarget producer=True stdio=stderr string_id=tap-exacttarget 2023-03-15T142219.928991Z [info ] INFO 'data_extension.Prova card store adereni BTS' is not marked selected, skipping. cmd_type=elb consumer=False name=tap-exacttarget producer=True stdio=stderr string_id=tap-exacttarget 2023-03-15T142219.941500Z [error ] 'utf-8' codec can't decode byte 0xe0 in position 41: invalid continuation byte ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ C:\Users\AndreaRadaelli\Desktop\Development\Venvs\benetton_SFMC\lib\site-packages\meltano\core\l │ │ ogging\output_logger.py:203 in redirect_logging │ │ │ │ 200 │ │ │ *ignore_errors, │ │ 201 │ │ ) │ │ 202 │ │ try: │ │ ❱ 203 │ │ │ yield │ │ 204 │ │ except ignored_errors: # noqa: WPS329 │ │ 205 │ │ │ raise │ │ 206 │ │ except RunnerError as err: │ │ │ │ C:\Users\AndreaRadaelli\Desktop\Development\Venvs\benetton_SFMC\lib\site-packages\meltano\core\b │ │ lock\extract_load.py:435 in run │ │ │ │ 432 │ │ │ # TODO: legacy

meltano elt

style logging should be deprecated │ │ 433 │ │ │ legacy_log_handler = self.output_logger.out("meltano", logger) │ │ 434 │ │ │ with legacy_log_handler.redirect_logging(): │ │ ❱ 435 │ │ │ │ await self.run_with_job() │ │ 436 │ │ │ │ return │ │ 437 │ │ else: │ │ 438 │ │ │ logger.warning( │ │ │ │ C:\Users\AndreaRadaelli\Desktop\Development\Venvs\benetton_SFMC\lib\site-packages\meltano\core\b │ │ lock\extract_load.py:461 in run_with_job │ │ │ │ 458 │ │ │ │ 459 │ │ with closing(self.context.session) as session: │ │ 460 │ │ │ asyn…

andrea_radaelli

03/15/2023, 2:25 PM

```'utf-8' codec can't decode byte 0xe0 in position 41: invalid continuation byte Exception ignored in: <function BaseSubprocessTransport.del at 0x000002A44A5971F0> Traceback (most recent call last): File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\base_subprocess.py", line 125, in del _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\base_subprocess.py", line 78, in repr info.append(f'stdout={stdout.pipe}') File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 79, in repr info.append(f'fd={self._sock.fileno()}') File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\windows_utils.py", line 102, in fileno raise ValueError("I/O operation on closed pipe") ValueError: I/O operation on closed pipe Exception ignored in: <function _ProactorBasePipeTransport.del at 0x000002A44A596AF0> Traceback (most recent call last): File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 115, in del _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 79, in repr info.append(f'fd={self._sock.fileno()}') File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\windows_utils.py", line 102, in fileno raise ValueError("I/O operation on closed pipe") ValueError: I/O operation on closed pipe Exception ignored in: <function _ProactorBasePipeTransport.del at 0x000002A44A596AF0> Traceback (most recent call last): File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 116, in del self.close() File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 108, in close self._loop.call_soon(self._call_connection_lost, None) File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 746, in call_soon self._check_closed() File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 510, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed Exception ignored in: <function _ProactorBasePipeTransport.del at 0x000002A44A596AF0> Traceback (most recent call last): File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 115, in del _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 79, in repr info.append(f'fd={self._sock.fileno()}') File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\windows_utils.py", line 102, in fileno raise ValueError("I/O operation on closed pipe") ValueError: I/O operation on closed pipe Exception ignored in: <function BaseSubprocessTransport.del at 0x000002A44A5971F0> Traceback (most recent call last): File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\base_subprocess.py", line 125, in del _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\base_subprocess.py", line 70, in repr info.append(f'stdin={stdin.pipe}') File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\proactor_events.py", line 79, in repr info.append(f'fd={self._sock.fileno()}') File "C:\Users\AndreaRadaelli\AppData\Local\Programs\Python\Python39\lib\asyncio\windows_utils.py", line 102, in fileno raise ValueError…

pat_nadolny

03/15/2023, 2:41 PM

It seems like the tap is likely having a problem with some of SFMC data but I'm not 100% sure from those log messages. I'd recommend running

meltano invoke tap-exacttarget > output.json

to isolate the tap and see if you can get a successful extract before bringing the target back in

andrea_radaelli

03/15/2023, 4:19 PM

I ran

meltano invoke tap-exacttarget > output.json

as you suggested, and I see that the

output.json

is beign populated. While I wait for the process to end I have a side question: I know that SFMC APIs offer a wide variety of attributes (like

ClickEvent

BounceEvent

and more), but I don't see them when I run

meltano select tap-exacttarget --list --all

. Is there a reason for that? Am I doing something wrong?

pat_nadolny

03/20/2023, 3:03 PM

@andrea_radaelli any luck? I'm noticing that the uptilab2 variant https://hub.meltano.com/extractors/tap-exacttarget--uptilab2/ (fork of singer-io) seems to be the most active, maybe give that one a try. It also looks like the click/bounce events your asking about are all part of the events stream

Events: Each of BounceEvent, ClickEvent, OpenEvent, SentEvent, UnsubEvent go into a single event table

(From the repo README)

andrea_radaelli

03/23/2023, 1:22 PM

@pat_nadolny the

invoke

comand populated the

output.json

correctly, I can now move the file in Cloud Storage manually. I will try again in the future and see if I can do both in one run (with the

uptilab2

variant as well). Thanks for the additional explanantion about

Events

table, I really appreciate the insight and the patience!

pat_nadolny

03/23/2023, 2:05 PM

@andrea_radaelli glad to hear you were able to extract your data! I think that confirms that its a problem with loading your data and not with extracting it. Now that you have that file you can run

cat output.json | meltano invoke target-gcs

to independently test the loader.

Open in Slack

Previous Next