Hello, I'm trying to find the

state.json

file in order to see that the state is being written. Unfortunately, I have the feeling that nothing is accomplished. I'm using the following code:

singer.bookmarks.write_bookmark(self.state, self.name+"_"+str(id), self.replication_key, lastupdatetime)
singer.write_state(self.state)

but nothin is written in the end. Can you help me out?

I might be wrong but the state is saved in the database AFAIK, not a state.json file

@pablo_seibelt is right, while meltano runs, the state is maintained in

.meltano/meltano.db

. After a run completes, a state.json file is written to

.meltano/run/elt/LOADER_NAME/RUN_ID/state.json

Thanks @pablo_seibelt and @edward_smith. Unfortunately, when checking the state.json file, it is empty. Also when logging the bookmark, it's empty

Have you had a successful ELT run?

Oh one thing to note is you may want to make sure you have set a job id and reuse it among runs

And how does it work when using the u, then?

What do you mean, the "u"?

UI, yes!

How about if you open your sqlite database?

Hello @pablo_seibelt, I tried browsing the `.meltano/meltano.be`the was nothing about bookmarks unfortunately

@charley_guillaume the bookmarks are a column on the jobs if i'm not mistaken

The payload column is empty for every successfull job

That's weird, might be that the tap is not reporting the state correctly

Thanks for your suggestion, I'll try that!

That's weird, does anyone know why the state would go out and not be captured by meltano?

Hi, @charley_guillaume. Are you also declaring a job id during invocation? For Meltano to capture and reuse state, you'll need to provide the optional --job_id argument during calls to

meltano elt

. https://meltano.com/docs/command-line-interface.html#elt

Hello @aaronsteers, I've been doing that. I keeep having the statement

No state was found, complete import

May be a dumb suggestion, but since this is a custom plugin, are you declaring the

state

capability?

extractors:
  - name: tap-whise
    pip_url: 'git+<https://github.com/we-invest-real-estate/tap-whise.git>'
    capabilities:
      - catalog
      - discover
      - state

Thanks @edgar_ramirez_mondragon for your suggestion! The state capability is indeed declared in the meltano.yml file:

name: tap-whise
    namespace: tap_whise
    pip_url: git+<https://github.com/we-invest-real-estate/tap-whise.git>
    executable: tap-whise
    capabilities:
    - catalog
    - discover
    - state

@charley_guillaume If you run with

meltano --log-level=debug elt …

, do you see the state output both with a

tap-whise (out)

prefix, and a

target-<foo> (out)

prefix? The tap writes the state onto stdout, which is picked up by the target on stdin, which is then expected to write it on its own stdout where it can be picked up by Meltano.

I'm not sure I see the target-postgres (out)

If you’re not seeing any

target-postgres (out)

lines, then target-postgres isn’t generating output, which would explain why Meltano isn’t seeing the state. But that’d be odd — target-postgres definitely supports state.

Is there a way to save the output of debug in a file?

Definitely, it should already be in a file under https://meltano.com/docs/project.html#meltano-directory:

.meltano/logs/elt/<job_id>/<run_id>/elt.log

🙂

I definitely do not have to target-postgres (out) in thelog

Which target-postgres are you using?

Meltano variant: https://github.com/meltano/target-postgres.git

OK, that should be fine, but can you try with the datamill-co variant instead? https://hub.meltano.com/loaders/postgres

Just to confirm, is the data being loaded successfully into Postgres? You should have log lines from it if so

The data is loaded for each stream from the tap

@charley_guillaume Are you seeing regular

target-postgres

log lines about the data being loaded, even if you’re not seeing any

target-postgres (out)

? I assume you are seeing the

tap-whise (out)

lines for all the messages including

STATE

?

I do see the information about the data being loaded. I'm having an issue now with the datamill-co variant regarding one of my schema. Debugging it now. I do see all the

tap-whise (out)

information with the

STATE

message

I don't think that the state will be written unless you successfully load data into the database. Remember that the state tracks data that was successfully loaded, so it only is saved when your TARGET has processed the records successfully, regardless of how many rows and state messages your TAP has sent.

As Edward says, if the target is failing to load any of your records, it will not output the state, because this would indicate that all records up to that point were successfully loaded and don’t need to be considered on the next run anymore, which is not the case if loading failed. If two of the variants are saying there’s a

null

value in your column, and all three are failing to forward the state message, I think we can assume that the targets are behaving correctly and that there is some issue with your tap. Can you share a snippet of out tap output with the full SCHEMA, RECORD, and STATE messages?

Thanks @edward_smith and @douwe_maan for your support! It's really appreciated as I find myself stuck in my development. Here is the snippets: `tap-whise (out) | {"type": "SCHEMA", "stream": "contacts", "schema": {"properties": {"address1": {"type": ["null", "string"]}, "address2": {"type": ["null", "string"]}, "agreementMail": {"type": ["null", "boolean"]}, "agreementMailingCampaign": {"type": ["null", "boolean"]}, "agreementSms": {"type": ["null", "boolean"]}, "box": {"type": ["null", "string"]}, "city": {"type": ["null", "string"]}, "comments": {"type": ["null", "string"]}, "companies": {"items": {"properties": {"Id": {"type": ["null", "integer"]}, "Name": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "type": ["null", "array"]}, "contactOrigin": {"properties": {"Id": {"type": ["null", "integer"]}, "Name": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "contactStatus": {"properties": {"Id": {"type": ["null", "integer"]}, "Name": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "contactTitle": {"properties": {"Id": {"type": ["null", "integer"]}, "Name": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "contactTypes": {"items": {"properties": {"Id": {"type": ["null", "integer"]}, "Name": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "type": ["null", "array"]}, "Country": {"properties": {"Id": {"type": ["null", "integer"]}, "Name": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "createDateTime": {"format": "date-time", "type": ["null", "string"]}, "firstName": {"type": ["null", "string"]}, "id": {"type": ["null", "integer"]}, "languageId": {"type": ["null", "string"]}, "name": {"type": ["null", "string"]}, "number": {"type": ["null", "string"]}, "officeIds": {"items": {"type": ["null", "integer"]}, "type": ["null", "array"]}, "privateEmail": {"type": ["null", "string"]}, "privateMobile": {"type": ["null", "string"]}, "privateTel": {"type": ["null", "string"]}, "prospectionStatusId": {"type": ["null", "integer"]}, "Reference": {"type": ["null", "string"]}, "representativesIn": {"items": {"properties": {"commission": {"type": ["null", "number"]}, "commissionUnit": {"type": ["null", "string"]}, "directLine": {"type": ["null", "string"]}, "email": {"type": ["null", "string"]}, "firstName": {"type": ["null", "string"]}, "id": {"type": ["null", "integer"]}, "mobile": {"type": ["null", "string"]}, "name": {"type": ["null", "string"]}, "officeId": {"type": ["null", "integer"]}, "pictureUrl": {"type": ["null", "string"]}, "updateDateTime": {"format": "date-time", "type": ["null", "string"]}}, "type": ["null", "object"]}, "type": ["null", "array"]}, "searchCriteria": {"items": {"properties": {"areaMax": {"type": ["null", "number"]}, "areaMin": {"type": ["null", "number"]}, "automaticMatching": {"type": ["null", "boolean"]}, "bathrooms": {"type": ["null", "integer"]}, "categoryId": {"type": ["null", "integer"]}, "countryId": {"type": ["null", "integer"]}, "customCriteria": {"items": {"properties": {"SubdetailId": {"type": ["null", "integer"]}, "numericValue": {"type": ["null", "number"]}, "enumId": {"type": ["null", "integer"]}}, "type": ["null", "object"]}, "type": ["null", "array"]}, "floor": {"type": ["null", "boolean"]}, "fronts": {"type": ["null", "integer"]}, "furnished": {"type": ["null", "boolean"]}, "garage": {"type": ["null", "boolean"]}, "gardenAreaMax": {"type": ["null", "number"]}, "gardenAreaMin": {"type": ["null", "number"]}, "groundAreaMax": {"type": ["null", "number"]}, "groundAreaMin": {"type": ["null", "number"]}, "id": {"type": ["null", "integer"]}, "investmentEstate": {"type": ["null", "boolean"]}, "parking": {"type": ["null", "boolean"]}, "pool": {"type": ["null", "boolean"]}, "priceMax": {"type": ["null", "number"]}, "priceMin": {"type": ["null", "number"]}, "purposeId": {"type": ["null", "integer"]}, "regionIds": {"items": {"type": ["null", "integer"]}, "type": ["null", "array"]}, "rooms": {"type": ["null", "integer"]}, "state":…

@aaronsteers Can you please help out here?

The only thing that jumps out to me is that when you call

write_bookmark

you should be passing a stream name, but you are passing the name

+ str(id)

. Is it possible that this mismatch between stream names caueses the state message to be ignored?

like

"contacts"

and

"contacts_6534"

?

I'm doing so because I need to bookmark on different contacts id. I am however taking the bookmark using the same method: singer.bookmarks.get_bookmark(self.state, self.name+"_"+str(id), self.replication_keys[0])

Hi, @charley_guillaume - I'm just catching up on these latest comments and I also dug into the code a little bit for context. It looks like you have some components from a non-SDK tap combined with the SDK-based framework, and there may be a disconnect somewhere that's causing the confusion. Admittedly, our porting guide is not very robust yet so I do apologize for the inconvenience if some things are not well documented. For instance, I notice you are overriding the `sync()` method directly and doing some direct state management which we intended would be internal. Your approach might still work fine, but I haven't looked deep enough to know if there are issues with this approach. Correction: I see this is not actually an SDK-port so strikethrough my above comment. The README.md is based on SDK, but the code is not using SDK classes.. Just so I'm clear on the symptoms you are seeing - is it correct that you see STATE being emitted as expected but it isn't being loaded and leveraged correctly when the tap runs the second time? Do we know if records are being received and processed by the target?

@aaronsteers Yes, the README.md as not been adapted yet 😔 I see the STATE being emitted in my tap-whise (out). At the end of the log file, I receive the following: ```target-postgres | INFO Table 'representatives' does not exist. Creating... CREATE TABLE tap_whise.representatives ("commission" numeric, "commission_unit" character varying, "direct_line" character varying, "email" character varying, "first_name" character varying, "id" bigint, "mobile" character varying, "name" character varying, "office_id" bigint, "picture_url" character varying, "synchronization_date" timestamp without time zone, "update_date_time" timestamp without time zone, PRIMARY KEY ("id")) target-postgres | INFO Loading 16 rows into 'offices' target-postgres | INFO COPY offices_temp ("address1", "address2", "box", "city", "client_id", "client_name", "country_id", "country_name", "email", "fax", "id", "mobile", "name", "number", "status_id", "synchronization_date", "telephone", "update_date_time", "vat", "website", "zip") FROM STDIN WITH (FORMAT CSV, ESCAPE '\') target-postgres | INFO UPDATE 0 target-postgres | INFO INSERT 0 16 target-postgres | INFO Loading 32 rows into 'regions' target-postgres | INFO COPY regions_temp ("id", "name", "synchronization_date", "zips") FROM STDIN WITH (FORMAT CSV, ESCAPE '\') target-postgres | INFO UPDATE 0 target-postgres | INFO INSERT 0 32 target-postgres | INFO Loading 328 rows into 'contacts' target-postgres | INFO COPY contacts_temp ("address1", "address2", "agreement_mail", "agreement_mailing_campaign", "agreement_sms", "box", "city", "comments", "companies", "contact_origin__id", "contact_origin__name", "contact_status__id", "contact_status__name", "contact_title__id", "contact_title__name", "contact_types", "country__id", "country__name", "create_date_time", "first_name", "id", "language_id", "name", "number", "office_id", "office_ids", "private_email", "private_mobile", "private_tel", "prospection_status_id", "reference", "representatives_in", "search_criteria", "synchronization_date", "update_date_time", "zip") FROM STDIN WITH (FORMAT CSV, ESCAPE '\') target-postgres | INFO UPDATE 297 target-postgres | INFO INSERT 0 31 target-postgres | INFO Loading 22 rows into 'contacts_origin' target-postgres | INFO COPY contacts_origin_temp ("id", "name", "synchronization_date") FROM STDIN WITH (FORMAT CSV, ESCAPE '\') target-postgres | INFO UPDATE 0 target-postgres | INFO INSERT 0 22 target-postgres | INFO Loading 5 rows into 'contacts_title' target-postgres | INFO COPY contacts_title_temp ("id", "long_description", "short_description", "synchronization_date") FROM STDIN WITH (FORMAT CSV, ESCAPE '\') target-postgres | INFO UPDATE 0 target-postgres | INFO INSERT 0 5 target-postgres | INFO Loading 14 rows into 'contacts_type' target-postgres | INFO COPY contacts_type_temp ("base_contact_type_ids", "id", "name", "synchronization_date") FROM STDIN WITH (FORMAT CSV, ESCAPE '\') target-postgres | INFO UPDATE 0 target-postgres | INFO INSERT 0 14 target-postgres | INFO Loading 9765 rows into 'estates' target-postgres | INFO COPY estates_temp ("address", "annuity_monthly", "area", "availability__id", "availability__name", "availibility_date_time", "bathrooms", "box", "can_have_children", "category__id", "category__name", "city", "client", "clientid", "comments", "commission", "commission_incl_vat", "commission_unit", "compromis_date", "contacts", "contract_day_date", "contract_type__id", "contract_type__name", "country__id", "country__name", "create_date_time", "currency", "details", "display_address", "display_price", "display_status_id", "documents", "end_mandate_date", "energy_class", "energy_value", "estate_order", "evaluation", "export_texts", "floor", "fronts", "furnished", "garage", "garden", "garden_area", "ground_area", "id", "investment_estate", "is_ty…

@charley_guillaume - Do the logs show anything happening with that stream?

tap-whise (out) is showing data coming from the API

target-postgres       | INFO Loading 90 rows into 'representatives'

target-postgres       | INFO COPY representatives_temp ("commission", "commission_unit", "direct_line", "email", "first_name", "id", "mobile", "name", "office_id", "picture_url", "synchronization_date", "update_date_time") FROM STDIN WITH (FORMAT CSV, ESCAPE '\')

target-postgres       | INFO UPDATE 0

target-postgres       | INFO INSERT 0 90

Here are the 2 messages I get about that stream

@charley_guillaume I think the issue here is that a stream is fundamentally a tap/target pair that can sync multiple tables. State is saved on a per-stream basis. So, because your tap is crashing, the final, successful state message is never produced.

Hello @edward_smith, Indeed the issue is that that particular stream is making the whole process fail. So far, I've been only selecting the other streams. But I can't seem to find the issue regarding that particular stream. That stream has the very same architecture the other have and apart from the endpoint used, there is not much that differ.

@charley_guillaume - Sometimes APIs differ from their docs and differ from sibling endpoints in surprising ways. It might be worth testing your assumptions about pagination, record location, data formats, etc. using a tool like Postman, or Thunder Client for VS Code.

@aaronsteers Excellent, thank you again for your time and recommandations. I'll keep you up to date. I'm using Postman so far to see the results of my requests but maybe there is something up down there I've overlooked!

Good luck!

Hello @aaronsteers! Hope you're doing good. I've rechecked everything and the data received corresponded to the schema I wrote. However: In the way my tap works, the last stream, for which I do not save the state, is the last to be run. I renamed it so it was run in the beginning and so that the last tap was one using a state. For some reason this solution works! But I'm now wondering if I should use the write_state anyway even when the stream is not using it. I don't really understand the reason this "workaround" is working. I'm currently running the whole tap to see if just using the write_state is working. I'll keep you posted on the outcome.