Related Posts
Mentor
How much would you give to this child 😏😉😊
Don’t make me go back to the office… 😩😩😩
35 / F / SF / 115k+
Additional Posts in Data & Analytics Consultants
Bowl Leader
Thought this was interesting. Across 160 teams of researchers, just about all failed to make good life outcome predictions on things like GPA, evictions, layoffs, and others. Data followed 4.5k families across 15 years, with 13k features (varied over time). Haven't looked at it directly yet, but will be turning the docs and data inside out... In the meantime, authors claim this as showing the limits of ML. Oh, and it's published in PNAS, so you know there's some big publication energy there.
https://www.pnas.org/content/117/15/8398
Mentor
New to Fishbowl?
Download the Fishbowl app to
unlock all discussions on Fishbowl.
unlock all discussions on Fishbowl.
Subject Expert
These are very valid questions. If you are trying to migrate database records as batch process, you generally would need to stage in cloud buckets and then update your database tables. These might differ across cloud vendor.
However, assuming once it’s setup, you should be able to query and update the database directly using an endpoint. There is no need to first load the data to storage and subsequent update to database. You could maintain replicas and cache as well as perform snapshots for faster reads and reliability.
This brings us to the question as how many hits one can make to the database per second and associated scalability constraints. For small and moderate scale workloads, sql are more than fine if you have a defined schema and query patterns do not include network like joins.
Not a database design expert but just my two cents on the topic.
Well metadata of media files. Media files would be stored in an S3 bucket, but trying to understand best way to store and analyze the metadata of those media files
Bowl Leader
Listing three to five words that describe your requirements may help you organize. Try picking from the below list, and then look up tools based on your priorities:
https://en.m.wikipedia.org/wiki/List_of_system_quality_attributes
Subject Expert
Is the metadata schema enforced or could contain free fields?
Yes user info would be a separate DB - assuming something like AWS RDS or Google cloud SQL. I guess my confusion lies on whether that data needs to first be stored somewhere like an S3 bucket beforehand? And if this is ideally real time info, would it make sense to have things cached after/if being stored before querying info from a DB. Again apologies if I'm just totally off here, just started looking at cloud architecture and really trying to learn some different scenarios
Use cloud data store aka fire store to store metadata