Hello I just started exploring Meltano today I m trying to c Meltano #troubleshooting

Hello! I just started exploring Meltano today. I'm...

dalyn_newby

03/19/2023, 1:23 PM

Hello! I just started exploring Meltano today. I'm trying to configure and setup the Postmark tap. Does anyone know how to resolve this error?

Copy code

CRITICAL tap_postmark.postmark.Postmark.messages_opens() argument after ** must be a mapping, not NoneType cmd_type=elb consumer=False name=tap-postmark producer=True stdio=stderr string_id=tap-postmark

Or can at least point me in the right direction?

Reuben (Matatika)

03/19/2023, 4:06 PM

Can you share your

meltano.yml

dalyn_newby

03/19/2023, 4:06 PM

Yes.

dalyn_newby

03/19/2023, 4:06 PM

Copy code

version: 1
default_environment: dev
environments:
  - name: dev
    config:
      plugins:
        extractors:
          - name: tap-postmark
            select:
              - stats_outbound_overview.date
  - name: staging
  - name: prod
project_id: d1720607-0c9b-4d8e-bfe2-30a7c26647c0
plugins:
  extractors:
    - name: tap-postmark
      variant: yoast
      pip_url: git+<https://github.com/Yoast/singer-tap-postmark.git>
      config:
        start_date: "2023-03-01"
      select:
        - stats_outbound_overview.date

Reuben (Matatika)

03/19/2023, 4:11 PM

Wonder if it's the empty
select
statements... Have you tried removing them before running?

Reuben (Matatika)

03/19/2023, 4:12 PM

Sorry, phone formatting is weird, I realise they're not empty.

dalyn_newby

03/19/2023, 4:12 PM

So completely remove this?

dalyn_newby

03/19/2023, 4:12 PM

Copy code

select:
        - stats_outbound_overview.date

dalyn_newby

03/19/2023, 4:13 PM

I tried, still getting this message.

dalyn_newby

03/19/2023, 4:13 PM

Copy code

CRITICAL tap_postmark.postmark.Postmark.messages_opens() argument after ** must be a mapping, not NoneType cmd_type=elb consumer=False name=tap-postmark producer=True stdio=stderr string_id=tap-postmark

dalyn_newby

03/19/2023, 4:17 PM

Ah, ok.

Reuben (Matatika)

03/19/2023, 4:34 PM

Have you tried running with

--log-level debug

to see if the tap outputs anything more helpful? Even with your

select

, from the screenshot it looks like it's trying to sync the

messages_opens

stream.

dalyn_newby

03/19/2023, 4:37 PM

Does this seem relevant?

Copy code

Variable '$TARGET_POSTGRES_SCHEMA' is not set in the provided env dictionary.

dalyn_newby

03/19/2023, 4:38 PM

Copy code

2023-03-19T16:38:12.664538Z [debug    ] Could not find tap.properties.json in /Users/dnewby/meltano/iation/.meltano/extractors/tap-postmark/tap.properties.json, skipping.
2023-03-19T16:38:12.664690Z [debug    ] Could not find tap.properties.cache_key in /Users/dnewby/meltano/iation/.meltano/extractors/tap-postmark/tap.properties.cache_key, skipping.
2023-03-19T16:38:12.664845Z [debug    ] Could not find state.json in /Users/dnewby/meltano/iation/.meltano/extractors/tap-postmark/state.json, skipping.

dalyn_newby

03/19/2023, 4:40 PM

That is what I found in the debug other than the CRITICAL message I shared earlier and a lot of tracebacks.

dalyn_newby

03/19/2023, 4:41 PM

I found this but I'm not sure if it is applicable. https://github.com/meltano/meltano/issues/6189

dalyn_newby

03/19/2023, 4:42 PM

Has to do with setting up mappers and a missing mapper configuration.

dalyn_newby

03/19/2023, 4:42 PM

Based on what I have read about mappers I don't think it is applicable but I am very new to Meltano.

Reuben (Matatika)

03/19/2023, 5:03 PM

Do you see a

Stream state: None

log (from here)? I expect the tap is not finding the stream state for some reason, causing it to error out trying to unpack

None

. FWIW, your configuration looks fine to me.

dalyn_newby

03/19/2023, 5:05 PM

Stream state: None

does not exist in the debug.

dalyn_newby

03/19/2023, 5:19 PM

Thank you for your assistance on this Reuben. I need to step away for a bit.

Reuben (Matatika)

03/19/2023, 5:30 PM

No problem, I'm curious to try this out for myself later. 🙂

dalyn_newby

03/19/2023, 5:30 PM

Ah, well, please ping me later if you figure it out. Trying to see if I can make Meltano work for my use cases and Postmark is one of the data sources I needed.

dalyn_newby

03/19/2023, 5:31 PM

(I will do the same to you if I figure it out.)

Reuben (Matatika)

03/19/2023, 5:51 PM

Will do!

Reuben (Matatika)

03/20/2023, 12:05 AM

Ok, so I did a little research this evening... After testing the tap myself and running into the exact same issue you were having, I came across this PR raised by @pat_nadolny a couple of months ago. As far as I can tell, this addresses a couple of things: •

stream_state

can sometimes be

None

if no state was set by a previous sync - this is the root cause of the

argument after ** must be a mapping

error (i.e. trying to

**None

) • A simple local change to coalesce

stream_state

to an empty dictionary quickly surfaces another error:

The parameter start_date is required

- this happens because the tap is trying to obtain

start_date

from state only (i.e. not taking into account the config you provided in your

meltano.yml

) So in summary, as-is, the tap is broken unless you are willing to manually set state before a sync (you can actually do this with the

meltano state

command if you really wanted to) - not a great user experience... I tried out the PR changes which fix the errors, but wasn't able to proceed much further as I don't have a Postmark account (

Client error '401 Unauthorized'

as expected, since I set

TAP_POSTMARK_POSTMARK_SERVER_TOKEN=test

in a

.env

file). As for how you might proceed, looks like the tap hasn't been updated since it was originally developed so I don't know how feasible it is to expect that PR to be merged anytime soon (especially since the PR itself has been open for over 4 months now). So you could either: • Fork the tap repository, merge in the PR changes and point the

pip_url

for

tap-postmark

to the URL of your fork in your

meltano.yml

- you could even submit the fork to the Meltano Hub as a new variant, once you're done • Try out the `airbyte` variant of the tap - the caveat here is this is a wrapped Airbyte Connector and so requires use of Docker (you can read more about this here) • If you have time, you could build a new

tap-postmark

with the Meltano SDK - speaking from experience, the SDK is very developer-friendly, the docs are great for reference and there are always people in this Slack willing to help if you get stuck Hope this helps! --- Side note: even with

--log-level debug

set, the tap doesn't seem to output debug logs, which explains why you didn't see the

Stream state: None

message before. Not sure why this is the case, but this and the above issues are good examples of why I prefer to use SDK-based taps when available, since the SDK ensures a standard of quality and behaviour (especially with Meltano).

Reuben (Matatika)

03/20/2023, 1:10 AM

Update: just found a tap-postmark built with the SDK that is not on the Hub yet. It says under development, so no idea what working state it is in, but definitely worth checking out at least.

dalyn_newby

03/20/2023, 1:11 AM

Ah, thank you Reuben! I will look into all of this some more. Really appreciate the details.

pat_nadolny

03/20/2023, 2:34 PM

I'm getting

Copy code

tap_postmark/streams.py", line 105, in OutboundMessageStream
    def get_url_params(self, context: Optional[dict], next_page_token: Optional[_TToken]) -> dict[str, Any]:
TypeError: 'type' object is not subscriptable

When I tried to run

tap-postmark --help

with that SDK version. From a skim it looks like it covers most of the yoast version stream but not all.

pat_nadolny

03/20/2023, 2:36 PM

If you wanted to test my bug fix fork in your project you can simply update your

pip_url

in the meltano.yml to

git+<https://github.com/pnadolny13/singer-tap-postmark.git@development>

and then run

meltano install extractor tap-postmark --clean

to reinstall it

4 Views

Open in Slack

Previous Next