What is “acceptable” with regards to query runtime. We’re on snowflake and have some queries that take seconds and some that take hours (multi-billion row tables). For the long running queries, what’s the balance between run time (worker efficiency loss) vs processing cost (cloud compute fees) to upgrade?

like
Posting as :
works at
You are currently posting as works at

It really depends upon the process details and the teams that interact with the specific data pipelines. Some questions to consider could be:
1) Is it possible to shift the large processing time from business hours to non-business hours (with the help of schedulers)?
2) Can you cache intermediate results? This would require a review but could pay off massively.
3) Are several long running queries doing the same intermediate calculations? This may involve refactoring queries.
4) How is the delivery of results affected from long processing times?
5) Does your current setup hinder teams from developing future use cases?

like

Use a hashmap bruh

like

Related Posts

31 F based in Chicago. Does anyone want to talk?

like

Siemens technology is product based company?

like

Young solo here w/ a question about fees: 1. A local solo criminal defense attorney has offered to pay me to go and make appearances before the Court on some of her cases to make announcements. (For example, “ready when reached”). I’m open to doing this and this attorney is asking what my fee would be? Does anyone have thoughts or opinions on what a reasonable fee would be to do something like this? Hourly? Flat Fee? 2. I want to ensure this is ethical. I don’t see an issue with this. Thoughts?

like

Think Covid is gonna get out of hand in the few months. Any stock recommendations in light of this?

likesmart

Hi Fishes,
My total experience is 5 in QA, I attended Deloitte interview. First round was technical and it went quite well, they asked lot of theory questions. I answered few and quite I was not sure of answers. She gave feedback like my communication and confidence level is good and project wise I'm having good knowledge and coming to concept wise it's not upto the mark for experience. So my query is will they call me for next round?I really wanna get selected since Deloitte is mydream company

like

Why companies asking pan card details for L1 discussion...... Is that fine to share the same....... Do any one can use our pan number to any fraud purpose......?

Post Photo
like
like

Anyway to get GSAP approved before your eligibility? They changed the eligibility to 2 years of working prior GSAP to 3...

like

How does a new campus hire best set themselves up for GSAP success?

like

Anyone else having a hard time focusing today? Knowing that I might find out today has me so anxious

like

Can anyone provide more detail into the GSAP retroactive app process?

like

“The copy needs more modern slang young millennials can relate to. It isn’t jiggy enough right now."

likefunnyuplifting

Who is hiring?

like

Hi all,

I’ve been applying to a ton of Account manager and CSM roles and have not been getting any traction with my resume. I do have 3 yoe in sales. Would anyone be able to critique my resume? I’ve cropped out the personal details at the top. looking for some guidance 🙏🏾

Post Photo
likeupliftinghelpful

Will the forensics and cyber merger lead to a reorg of the IFS groups in advisory? Heard there’s a chance of a reorg for finance, ops, comma, and Hr.

Bain & Company McKinsey & Company Bain & Company Boston Consulting Group fishes, in job descriptions it is usually written that knowledge of local language is mandatory. Is it true? It would be helpful if you mention your country as well. Thanks :)

like

I want to buy my husband some underwear for Valentine’s Day. What brand do you guys like?

What are the best consulting firms operating out of London known for / working on sustainability and energy transition topics? Many thanks!

like

I feel like a lot of my fellow producer friends are getting absolutely destroyed with their workload this summer (myself included). How are you guys surviving?

Post Photo
likefunny

What’s the ratio of IT support staff (excluding Directors, managers, etc) to users at your shop?

Additional Posts in Data & Analytics Consultants

If you were to hang your own shingle as a consultant what BI tool would you use to deliver dashboarding to clients?

Assume that you aren’t plugging in to source systems as I primarily work in CDD.

like

Anyone in the KPMG Lighthouse practice that would be willing to take a look at my husband’s resume and put in a referral? He’s interested in the Associate, Data Scientist position.

Docket + R - I have an container running a rocker image and I’m trying to launch a bash terminal and the RStudio server (/init) at the same time. Is this possible? I tried adding CMD [“/init”] too.

like

Have you ever failed to deliver? What was it and why?

like

Anyone have recommendations on the best app/site to learn Python? 🐍
I’m a complete beginner.

like

Anyone here work or has interviewed for a data science role at DataRobot? Would love to get some insight. Third interview coming up.

like

Big 4 Data Scientists, how did you exit to big tech firms? I’ve been unlucky with online applications and even referrals. I think it might be my resume or how I present myself. Can any free birds please give guidance?

like

McKinsey & Company Does somebody have experience regarding
compensation differences between
QuantumBlack & McKinsey? I am in the process at
QB and disappointed with the salary that's
proposed. (New grad Data science from top
university applying for London) McKinsey & Company QuantumBlack

like

Partner: Let’s take the data in the spreadsheet and put it in PowerBI so we can cut it up.

Me: What? It’s already cut up in the spreadsheet. I don’t get it.

Also me: Yeah! Analytics!

likefunny

It’s been ages since I’ve taken college statistics. Can anyone recommend an online course where I can review and brush up?

like

Newish to data analytics. Have some experience in power BI, sql, and R. I wanted to get some exposure to aws and saw the AWS cloud practitioner as an available course at my job. Is this cert worth it? Or should I try one of the more advanced ones ? The data analytics one for example but wasn’t sure if it would be difficult to get through as a beginner. Will it look good on my resume?

like

Anyone have any suggestions for front end guis for users to run python code? I’ve created a few tools for webscraping data, automation, and modeling but non technical users freak out when they try to run the code themselves. Any suggestions? (Have tried Juypter Notebooks and a few other IDEs)

like

What biotech stocks are yall trading right now?

GILD, INO, MRNA

like

I was hired a few months ago by D to do data science. I am not doing data science currently, is there a way I can switch roles internally? Should I talk to my coach? Or should I just start looking for a new job. I’m starting to feel that the field is so hot rn my small stint at D won’t be a big deal to other firms.

like

Anyone have a ballpark range on dataiku price per seat or however they price? It looks like an interesting product.

What’s the best/simplest way to understand Sets vs Groups in tableau?

like

1) Hypothetically if you were looking to sabotage a colleague, can you code an excel document to throw off their numbers or calculations, (if you send them that document and they download it for use)?

like

Has anyone interviewed for a Data Scientist position at Facebook and have any advice on the coding and product sense interview?

like

Meeting with a recruiter at one of the FAANG companies. I won’t be able to move jobs because of H1b /GC at least for the next year. Any idea if I should start off with that or should I just try to understand the work that they have been doing?

like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal