Hi Everyone, I'm trying an incremental load of a table from mssql to postgres. The second run always fails after info: table exists. Is there something I'm doing wrong here

My hunch would be that you don't have a replication method explicitly configured for

target-postgres

(assuming it's the pipelinewise variant)

Thanks alot, I think the problem was that my source was a view. Is it possible to get an incremental update on a view source?

I think that would depend on the view. E.g. Let's say that the view has some datetime column that you can use as the replication key (e.g. 'updated_at') If you use that 'updated_at' column as the replication key, then meltano will keep setting the bookmark value in the meltano state database for that stream, whenever a new record has appeared that has a newer 'updated_at' value than what was previously stored in the meltano state db as the bookmark value. If the view is built in such a way that the view will NEVER make records appear unexpectedly with an 'updated_at' value that is "in the past" (i.e. that is older than what meltano has recorded as the latest bookmark value for 'updated_at'), then you're probably OK.

That's very helpful advice. Thanks alot

It's due to the nature of what a View is - a View's intent can be a very simple thing that is almost a carbon copy of a single underlying table and it can be a very complex thing joining multiple tables and applying complex business logic. Unless you have analysed the inner workings of the view in question and know for a fact that you can use the view with an INCREMENTAL replication scheme with a stable time-based replication key, FULL replication is the way to go.

I doubled checked by trying out Incremental replication on a table instead of a view. I noticed that the failure is not happing on the target but on the Source. The error is not descriptive, only saying status error. As shown below time=2023-01-14 190941 name=target_postgres level=INFO message=Table '"snmptrap"' exists cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres 2023-01-14T190941.162356Z [info ] 2023-01-14 190941,162 | INFO | singer_sdk.metrics | INFO METRIC: {"metric_type": "timer", "metric": "sync_duration", "value": 0.04533576965332031, "tags": {"stream": "dbo-SnmpTrap", "context": {}, "status": "failed"}} cmd_type=elb consumer=False name=tap-mssql producer=True stdio=stderr string_id=tap-mssql Is there a way to see a more descriptive error log?