<!channel> Office Hours start in just under an hou...
# office-hours
a
<!channel> Office Hours start in just under an hour! Join us today to chat about SDK updates or anything else that's on your mind. We'll drop a link to the Zoom call when we go live.
v
Missed this! Sorry 😄 here's my 2 cents fwiw DB Types streams - Couple points, binlog is nice but not every DB has it so don't fret too much! @edgar_ramirez_mondragon - Pydantic / SQLAlchemy stuff. Great hearing you talk about how you're thinking about things!!! @aaronsteers / @edgar_ramirez_mondragon listened deeply to the SQL Alchemy stuff vs direct/odbc. How painful would ripping SQL Alchemy out be if it turns out to be the wrong choice is really where my brain goes. I think I get what you are after. The proprietary type issues are a pain especially depending on which version of JSON Schema you can use. Where are the type hints going to be placed? Having access to the catalog if you're a target can be pretty useful, but at least having some way of mapping types between DBs is useful in some DB -> DB data moves. Looks like the rest of the office were cut off
e
Where are the type hints going to be placed? Having access to the catalog if you're a target can be pretty useful, but at least having some way of mapping types between DBs is useful in some DB -> DB data moves.
The singer spec supports a
sql-datatype
metadata field (https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md#metadata). Do you think something like would help in keeping data types more faithful when doing db->db EL?
a
There’s a second video on YouTube with the next section. They’re still processing but I’m going to try and combine them
v
https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md#metadata yes, and the SCHEMA row that's passed to the target doesn't contain that information. I believe what you are showing is just about the catalog. Maybe I"m mistaken? @edgar_ramirez_mondragon
Do you think something like would help in keeping data types more faithful when doing db->db EL?
Yeah I had to do some nasty stuff (Maybe there's a better way I'm very open to chucking out these ideas) to get VARBINARY and date vs datetime passed properly from oracle to mssql https://gitlab.com/autoidm/autoidm-target-mssql/-/blob/main/target_mssql/streams.py#L196-197
It turns into a nasty matrix really, where ideally you'd just define the type in the json schema and have everyone use that instead but 🤷 it's not clear to me what the right route is