https://linen.dev logo
Join Slack
Powered by
Hi. I'm running meltano 1.100. I'm extracting data...
# troubleshooting
d
Hi. I'm running meltano 1.100. I'm extracting data from postgres and adding to redshift. Meltano is running on AWS Batch. I added some new tables to the 
select
attribute, but my elt run isn't ingesting the new tables. Any idea why this might be?
meltano.yml
meltano.yml
e.g. the "subdomain" table isn't being ingested.
I discovered that my meltano postgres user didn't have access to the table in question. I can easily fix that. I'm wondering, though, why there were no errors about this in the elt run.
v
I'm not sure if there's an issue in for this in Meltano. Select iterates over the 
catalog
generated by the tap during 
discovery
, in this case, 
tap-postgres
Currently the select doesn't fail if something doesn't exist. I'm not clear about exactly how this would be implemented as there's a lot of factors to think about as there's glob syntax available for selects. -- What people do today: Generally since you're parsing the data with DBT anyways you'll see that the data doesn't exist in the DB when DBT blows up yelling at you for source data not existing.
d
That makes sense. Thanks. Yeah, I got DBT blowing up.
v
Could you put in an issue for this to Meltano though? It does seem interesting to me
To expand on
Generally since you're parsing the data with DBT anyways you'll see that the data doesn't exist in the DB when DBT blows up yelling at you for source data not existing.
A bit, to be sure your data is up to date, folks write tests in DBT on the source data as well (using freshness normally) just to be sure Meltano keeps doing what it promised for you
d
I've logged https://github.com/meltano/meltano/issues/6147 with a suggested algorithm that may result in my condition being caught and an error thrown.
Open in Slack
PreviousNext
Powered by