Hey all, I've been trying to set a up a new meltan...
# troubleshooting
d
Hey all, I've been trying to set a up a new meltano project for some testing of a plugin im making, and have been running into a few issues. I've been trying to add tap-google-analytics and target-postgres, running the elt, build some reports in the UI and then use my plugin to convert those reports to our own dataset format. I install the defaults of the tap-google-analytics, target-postgres, and the transform and model for tap-google-analytics. This is all in a clean venv only wheel and the most recent meltano installed. It seems when you use the default (datamill) target-postgres the transforms for the tap-google-analytics errors out. The data from the tap seems to have different column names, (missing an "_" in a few names), than what its expecting. The same error happens with the transferwise variant of target-postgres, but with the meltano target-postgres variant everything works. Then if you go to the default reports in the UI using the meltano target-postgres, you get an psycopg2 error saying the table doesnt exist. If you run the transforms with the meltano post-gres variant then try to view reports with the transferwise target-postrgres, you get the same psycopg2 error, and if you try to view the reports with the default (datamill) variant it errors out with "The dialect 'postgres' is not yet supported by meltano analyze". Has anyone else run into anything like this / Am I missing something? My plan at the moment is to fork the transforms for tap-google-analytics first, trying to get at least the transforms to work with the default postgres variant first. Then I guess I'll work on the report errors, but just wanted to check this isn't something known or if anyone knows of any other ideas?
I ended up forking the transforms, installing meltano locally to edit the discovery.yml and changing the forked tap-google-analytics transforms to work with the default (datamill) target-postgres, which also makes it work for the transferwise variant. (It breaks the previously working meltano variant though). Now when you go to the explore>reports or reports builder using either the transferwise or meltano target-postgres variant, the data loads, you can make and save reports. Still running into the issue of the default (datamill) target-postgres just showing "The dialect 'postgres' is not yet supported by meltano analyze" on all reports pages. Gonna make a start on working out whats going on with it. Should I make some of my issues into actual issues? Feel like this would be nicer to break down into a few issues, though I'm not sure if UI stuff is super relevant at the moment?
d
@daniel_walker As you've correctly identified, the transforms and models were made with the
meltano
variant of
target-postgres
in mind, and when we deprioritized everything related to the UI/Analysis and made the
datamill-co
variant the default instead, we didn't go back to update the transforms and models. While not a priority for us right now, it's definitely worth tracking in issues, so if you could create those that'd be appreciated. I wonder if it would be possible to have the transforms work with all variants of the loader, by branching based on the actual loader used, which I think should be available through environment variables Meltano inserts into dbt's execution environment. As for
The dialect 'postgres' is not yet supported by meltano analyze
, that sounds like an actual bug we should also track in an issue and resolve. I think the error originates here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/connection_service.py#L63, since the template
"postgresql://{user}:{password}@{host}:{port}/{dbname}"
expects a
user
key, while the config for
datamill-co
uses
postgres_user
instead. We'd need to make that not dependent on the actual setting names used.