How do you prevent a data warehouse from being overloaded by queries?

Do you spin up multiple instances of the same databases? Does AWS/Azure/Goog auto-scale, or is this all manual?

like
Posting as :
works at
You are currently posting as works at

It depends. There's a lot of ways to architect something that can scale horizontally (and in real-time)

If you're using a managed service like Redshift, you don't have to manage this yourself.

But if you're building it on your own and you're wondering about scaling up, then you'd want to look into load balancing and the likes.

You'll also want to think about views and temp tables, so every user is querying different "instances" of the database.

Are you on a cloud service with enterprise account? Call your solutions architect and ask them.

like

Thanks! This isn’t for a project, just trying to educate myself On how BI architecture best practices are set up

I’ve been doing a lot of Googling, but am interested to hear if you have a more focused suggestion

Not really. I'm just a hobbyist as well. I'd suggest starting with aws or gcp websites. They need to create educational materials so they have more business. They are motivated to educate you. Lol.

like

You can also look into various architecture patterns - caches, message queuing, circuit breaker, load management/shedding

Make your warehouse redundant and implement a load balancer

Related Posts

I’m looking for a creative lesson to go with this standard. Any suggestions?

Post Photo
like

What is the latest on Microsoft return to work strategy? I’m due to start in January and know I’m going to start remote but want to know the latest RTW plan. TYIA!

like

Has anyone ever worked with reproductive science center of NJ?

like

Anyone else moving out of the city/GTA? Seems like this isn't going anywhere until early/mid 2021...

like

Just got a controller offer with pretty expanded duties given no CFO at the company. Well funded tech startup, no shortage of cash. I only have 3 years experience at Big 4, so probably a little early for me to get this role BUT they offered me a salary way below what is reasonable for this role.

A bit conflicted on if I should take it and run, or if its worth fighting for a comp that matches the duties

likesmart

Is KPMG callling people to Work from Office and i am about to join KPMG and no one mentioned anything . Pleasr advise

Affordable places near LA/in CA to learn to snowboard?

like

How are the health tech companies doing? Starting to see some layoffs in the space.

like

Anyone else required to put together completely unrealistic timelines?

Upper management's direction is to put together the most aggressive deadlines we can. If we think a project will take 16 months, make the plan for 10 months then there's a chance we may launch in 13 months.

I understand that time to market is important in consumer product goods, but being constantly behind schedule and rushing everything is terrible for the team's morale.

like

Looking to learn what other people’s jobs are like, what’s your role and what’s your day to day like?

like

Is TW calling employees back to office?

like

Buddy , how to apply for h1b from india is it possible i heard some consultancy does that

EY Dallas folks - Anyone get the golden email yet?

like

Have made 16k from options sales this year, such easy money, is it dumb to think that this is a good passive income stream? Worst case scenario, I get stuck-bag holding FAANG during a bad downturn.

like

Which company should I join, Mindtree or Persistent as a Java developer with 8 years of experience?

Curious if DE Shaw requires a drug test in employment screening?

funnylike

Question about EHE.... is this only for a PwC employee and not spouse on the same work health insurance ? Isn’t your partner covered??

like

Anyone have side effects from Estrace? I thought it would be a piece of cake compared to Lupron microdose, but it kind of knocked me off my feet (started taking yesterday) - didn’t see it coming.

like

Additional Posts in Data & Analytics Consultants

Is it even worth applying for new jobs? Are companies still hiring?

like

How would you rank these master programs?business analytics MIT sloan, new data analytics program at UC Berkeley, analytics and business intelligence by CMU, analytics program at UCLA, business analytics at Duke Fuqua, data science program at Columbia, ds in Northwestern?

like

Any 🐠 able to speak to the differences between Accenture AI and Deloitte for data science and exit opportunities to tech?

like

How do you all start preparing for interviews while working 45-50 hrs. a week? I'm at a point where I hate my current job but I don't know how to even begin the whole interviewing process. I am quite good at sql, power bi, and worked with tableau a bit too. Looking to find BI roles

likehelpful

Does anyone have a template they don’t mind sharing for their performance review? TIA

like

How would you prepare for a SQL or any coding live interview? Do you solve hackerank or leet code problems every day?

like

I have stumbled my way into being a Data Management “SME”. I am leading projects performing source to target mappings, data lineage, building data controls like data quality rules, designing database schemes, building ETLs, etc.

But, I have no formal training in data. I remember when I got thrown into this googling to figure out the difference between a field and an attribute. It’s been years and I still think my foundation is shaky.

Any recommendations on good resources for data management?

like

Would you give a referral to someone you haven't talked to in years and don't know much about their ability?

like

Hi Fishes,
IQVIA - Want to understand what is pay band for grade 140 position at IQVIA India
Also can someone through light at consultant position at IQVIA for Patient Analytics team, what is the kind of work being done and is there chance of opportunity abroad from here on.

like

Book recommendations for Experimentation (for a noob)

like

Isn’t Pascal’s triangle just a visual tool for seeing combinations of items in sets?

like

Is there a path for a 45yr old male to move into Machine learning space at this stage of life? I am good at learning, understand data modeling quite well. If you have done it, can you please share your approach in making this change and how are you doing it after the switch.

like

How good are PwC and EY around advanced analytics compared to K Lighthouse and Deloitte A&C? General feeling at the latter two is they only compete with themselves and PwC and EY are a few yrs behind.

like

Do y’all use CRISP-DM?

like

Question to Independant consultants. What apps do you use for running your business ? Accounting, Marketing, SocialMedia and Proposal/Contracting ?

like

Client wants to "integrate" ERP with various systems. In charge of requirements.

What's the difference between interface and integration (SI)?...

like

What’s the salary band at the director level for KPMG in their D&A practice?

like

What are some technical skills that I should develop to land a Business Intelligence role in the tech industry? SQL and a data vis tool are a must I’m assuming. What else?

like

How MBA can help my career in Data science assuming I have mature experience in the field of DS/AI? What is that you can’t do in DS without MBA?

like

Would you all interview for a role you’re not sure you’re qualified for? I initially thought the role was more BI, but it’s a true DS position at a well known Tech company. I’m worried about the Python tech screen and if they ask more theoretical Machine Learning questions. I’ve got the Tech undergrad degree, but my day to day consists of pandas and pyomo

likefunny

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal