Hi all, working on a project from scratch and i have to search and fetch huge number of rows of data and show on UI. Also need to export data using pdf/xls. Data can be million and billion records. What is the best possible approach.
Related Posts
Hi Fishes
I would like to know about the client interviews (especially in data science or machine learning roles).
Are these interviews tough?
Are these interviews meant to assess technical knowledge? For example, what is binomial distribution, what are different data structures in python, ml algorithms etc.
I would request each of the nagarrians to share their thoughts irrespective of the technologies you are working.
I am thankful to you for taking time and helping me out.
Nagarro
Anyone working on Raymond James(RJ) client?
More Posts
Best way to get cheap park city lift tickets?
Additional Posts in Financial Services - American Express, Mastercard, JP morgan, Citi, Barclays, Goldman sachs, capital one and similar Fintech/Companies
New to Fishbowl?
unlock all discussions on Fishbowl.



Use pagination or slice concept
Yes... pagination is what i am thinking..with offset and limit, but shall i use queue or kafka in between...will it help?
I am thinking of following architecture:
client-> gateway-> Service A -> db
Service A checks count puts request in a queue, returns a general response to user that ur file processing has started with this count.
Service B picks up request from queue-> fetch records from DB using pagination (limit and offset) getting few thousands records at once, and writing in a file.
Once done, a db table maintains file is created status.
Polling from UI every 1 min, if any file is completed, ui shows file completed status
User clicks get file-> gateway-> service B
Service B streams file back which gets downloaded in chrome browser.
Please suggest any flaw/improvement in above design^