Just merged an update to <mdsinabox.com> - now off...
# show-us-what-you-gotj
jacob_matson
08/16/2023, 1:42 AMJust merged an update to mdsinabox.com - now officially has evidence in as a util and some other goodness like an NFL š model if that floats your boat!
t
taylor
08/16/2023, 1:55 AMwhen Formula 1 though? šļø
j
jacob_matson
08/16/2023, 1:56 AMš¤ š¤
jacob_matson
08/16/2023, 1:56 AMIf there are published odds I can build a model haha.
t
taylor
08/16/2023, 2:00 AMpretty sure itās Redbull 100% for drivers and constructors championship. but p2 is looking interesting
v
visch
08/16/2023, 2:10 AMProject (meltano.yml layout) is getting so good @jacob_matson any tweaks I'd recommend are so small they wouldn't make a difference great job!
e
edward_ryan
08/16/2023, 8:32 PMThis is great @jacob_matson! Who is your NFL team? We're gearing up for the season
j
jacob_matson
08/16/2023, 8:36 PMi'm a seahawks fan! excited to see where it goes
j
john_cassil
08/21/2023, 7:07 PM@jacob_matson maybe I am a bit dense -- or maybe I overlooked something -- but I am having trouble building a mental model of a useful mdsinabox system that doesn't include a storage layer.
Like, sure if the source of your data is always available and you can meltano tap it in to a duckdb database on the fly and analyse stuff that's cool and all, and maybe it's purposely simple so you can show it off with the NBA/NFL data.
But is there a way to actually build a warehouse this way for a team -- with persistent data?
I don't want to knock this because I enjoyed reading it and really appreciate your work here -- but it duckdb seems like a shiny bobble for data analysis workflows right now... rather than having good integrated data engineering use cases.
j
jacob_matson
08/21/2023, 7:11 PM@john_cassil hey no problem, you aren't being dense! I'm starting to contemplate how to do this going forward. It was an intentional trade off i made in order to ship the project back in october 2022. Welcome some thoughts over on this github issue. https://github.com/matsonj/nba-monte-carlo/issues/102
jacob_matson
08/21/2023, 7:15 PMAlso see this PR - https://github.com/matsonj/nba-monte-carlo/pull/29 which did add S3 support via localstack but it was SO HEAVY. There were also a lot of moving parts with how materialization was supported with duckdb, and that is much more stable today.
jacob_matson
08/21/2023, 7:19 PMI think with the #C05CNUF699B it might really streamline how this works (cc @visch), but i haven't had time to explore it yet
j
john_cassil
08/21/2023, 7:37 PMDesigning this in my mind -- granted I haven't used meltano before, just been watching from afar and know very little
john_cassil
08/21/2023, 7:38 PMfloating numbers
john_cassil
08/21/2023, 7:40 PMOf note: At CARFAX we had an ELTL system that I have never forgotten. It was way better than ETL or ELT because you could go back to the raw file and reload it if you found an issue or needed to bring in a new field -- but you didn't have to transform the same data every time -- and you didn't have to build complex incremental models for literally everything -- it was the assumed and easy default.
john_cassil
08/21/2023, 7:41 PMI dream of the day that DBT + something else is as beautiful as this handrolled system that people built with hadoop in like 2007
j
jacob_matson
08/21/2023, 10:33 PMFwiw I think you can go even simpler than what you have drawn up and use S3 for storage and duckdb for compute. just write back to different buckets for different stages( raw, bronze, silver, gold etc). You only need to keep what you need for transactional loads in a transactional db, hold the rest in s3. (or thats how i'm thinking about this)
t
taylor
08/22/2023, 3:28 PM@ken_payne has some ideas around this idea called āSingerlakeā - https://blog.kenpayne.co.uk/blog/2023-02-15.html weāre looking at a pattern like this for an upcoming product effort
j
jacob_matson
08/22/2023, 5:41 PMYeah, I love the idea of keeping everything in the lake (incl meltano state)
13 Views