Does Pyspark data engineering need extensive sql query like 1000 lines of query writing like we do in Power BI role or basic to medium sql is enough for data engineer role?

like
Posting as :
works at
You are currently posting as works at

Doesn't the query depend on what you want to achieve? Haven't worked on power bi so not sure why 1000 lines of SQL. But if you are using spark, it's not just the SQL, performance tuning is your priority.

like

Depends but mostly it will be multiple Queries in the form of procedure or program.

So we use stored procedure also in Pyspark based project?

Related Posts

Hi Fishes,

Pls reply or DM if interested asap.

Post Photo
like
like

What’s a fair operations manager salary

I need to get out of a toxic relationship. I don’t have a car. I feel stuck like that scene from Maid. I’m pre-approved and have a small down payment saved. I just need to do it. What should I be thinking about to stay positive?

like

what’s the quickest way to find all offsetting entries in a huge data sheet?

like

Anyone already stressing over the July bar?

like

I am a newly graduated fresher mechanical engineer working in an electro mechanical company at Dubai, My current salary is 2500 AED, I would like to know about your comments and suggestions regarding my career which could help me through my future ventures.

like

Not a huge sneakerhead but are Nike Airforce 1s still cool? 😭😭😭

like

Any IPO readiness checklists/blogs/podcasts floating around?

like

Gents--I'm getting married later this year and I'm struggling over what to get as groomsmen gifts primarily because every groomsmen gift I've received is currently collecting dust. Any ideas on thoughtful/practical groomsmen gifts?

like

Just completed my first myetherwallet.com transaction and participated in my first ICO (ATL/ ATLANT)...i feel invincible!

like

I would think Miami, LA, Chicago, NYC. Anyone attend the ALPFA convention?

like

Has anyone recently got their perm approvals? If yes, when did you file ...

like

Hello people,

Is there any site , channel or pdf for a more guided way to crack interview for data science related positions?
I've been trying on my own but getting overloaded with too much content available and not sure which one to look into first.

Thanks in advance

like

Did anyone else get. Warning email from HR for RTO? I am really scared what action they’ll take . I was planning to travel my base branch in Jan but now it’s difficult to look for an apartment in Bangalore on such a short notice . Please share your advise .

like

What are the funniest ways clients, friends, or family have referred to RB incorrectly? Such as “Ronald McBurger” or “R and B"

like
like

How do you get out of the burnout cycle? I’ve felt overwhelmed for a while and now I’m in the “can’t function at all” stage. It’s so hard explaining this to my manager because I already have a workload that’s lighter than others (not because of performance or my request). Any tips on how to get re-engaged, since the overwhelming feeling will come back?

like

Back when the pandemic first hit, everybody at the firm took a 20% pay cut (with 20% reduction in billables...which was meaningless because I still had work to get done in a timely manner). Pay cuts still in effect through end of year. Anybody else out there still working under pandemic-related salary reductions?

like

Hi ,any interview tips for senior analyst role??

Additional Posts in Data engineers India

like

I am currently working as an ETL developer. It involves GUI and some SQL. I want to know how to switch to data engineering. I am planning to apply for the job next year where. So, I will be having 1 year of ETL experience at that time.
What should I learn now as per Indian market. Specific tools which will have good future growth? I am good in dsa, mysql, python, programming concepts, and theoretical CS.

like

Please suggest

YOE 2.4
Azure data engineer

Fractal - 14.5
@Rich Products Corporation - 18.5

like
like

Hi I'm working as a data engineer and having 1.4 YOE with current CTC as 4LPA, what will be good time for me to switch and how much CTC should I expect?

Please DM if interested asap

Post Photo
like

Hi currently iam looking for big data developer role, i have got offer for data engineer ,dataops - company name (comcast), but i have only limited idea about that role, could anyone please help me to understand about this role. Comcast

While describing, it was more like support role

Hey Fishes,

I'm in a biggest confusion here. And its pushing me to depression. Please help me if possible. I got 2 offers and both are good reputed product based companies. And offered the same role.

Fractal Infosys Tata Consultancy Wipro Accenture Microsoft IBM Cognizant

Post Photo
like

Which company pays highest for Data Engineers (Fixed CTC) for anyone with 5+ years of experience?

like

Hi fishes, I have 3 years of experience in DevOps, I have a good knowledge of python and DSA, I want to work in Data Science.

Can someone please suggest how can I switch to Data Science ?

like

HI FRIENDS

I AM WILLING TO TAKE UP ##BIG DATA ENGINEER COURSE - FROM ##SUMMIT MITTAL ##TRENDYTECH - CHARGING RS.55,000.

MANY REVIEWS AND LINKEDIN POSTS SHOWS PEOPLE CRACKED 65 LAKHS TO 1 CRORE CTC WITH 1YOE IN ##INDIA OR ##UK OR ##DUBAI.

I HAVE EXISTING CTC OF 30 LPA - TECH STACK
SQL +SSIS + SSRS + ETL + POWER BI + DATA WAREHOUSING - 5YOE .

IS IT GOOD COURSE THAT MUCH AMOUNT FOR ##BIGDATA ENGINEERING.PLEASE GUIDE ME.

like

Hi fishes,

I get this question a lot in interviews, so can anyone please let me know the answer?

When to use groupBy key and reduceBy key?

like

Does anyone know about applied materials???
What is salary band for 4yoe Hadoop developer

like

Please suggest what is the expected CTC for AWS data engineer role.
With 11yoe.
Current CTC 22LPA

And which are good companies that pay good with good wlb.

like

Hello fellow data engineers,

Can anyone suggest some free courses to study spark and Pyspark?

Thanks.

like
like

Hello Everyone, I am currently working as an analyst on a Project which is based only on excel. I want to switch to a Data Engineer role, I have 1 YOE & CTC 7.4 Could you please suggest and guide me on how I can train myself to switch to a Data Engineering Job Profile. I've no idea about the tech used in Data Engineering. I just want to Switch to a tech profile. I'll be more than happy if you guys can guide me through this journey.
Thanks in Advance.

like

Pls DM if interested.
Location- Bangalore

Post Photo
like

Pls DM if interested

Post Photo
like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Send download link to your phone

OR

Scan your QR code to download
Fishbowl app on your mobile

By continuing you agree to Terms of Use and Privacy Policy.

Messaging rates may apply

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal