Does anyone know how to work with nested loops on Pyspark with 3M Rows as Input file to loops?

Thanks!

like
Posting as :
works at
You are currently posting as works at

Avoid Python Loops: Use PySpark's operations where possible. You can leverage PySpark’s RDDs to work with row-wise operations. map() or flatMap() allows you to perform transformations that act like loops but in a distributed manner.

like

Thanks man!

Related Posts

I got Medical Reimbursement of around in current month salary, but I don't think I had applied any bill or submitted any such amount. Can someone tell what exactly is this amount for?

like

Finger Lickin' Good

Post Photo
likefunny

Do you like humor incorporated into resumes? Why or why not?

like

What is the experience required to become senior manager in US taxation

Ey work life balance

funnylike

What is everyone’s thoughts about Hilton? Just got asked to interview

like

What’s the shortest amount of time it’s respectable to be in a job before leaving?

like

Hey Fishes, Can I rejoin Infosys again. I left Infosys in dec 2020. If yes, please refer me for React Js. YOE - 5.5Infosys

I have 2 year and 10 months experience in TCS. I am a Java developer, and know java, spring boot, rest, spring MVC, oracle database. And I have to serve 90 Daya notice period.
Please help me to get a job.

like

What is street parking like in Little Italy? Looking at a place but will have to park on the street, specifically near grape street/Columbia street

Toured One Paseo in Del Mar this week and it’s just so damn perfect💔 But that price tag 😭💸 super new to SD. Any recommendations for similar vibes elsewhere that might be a bit more reasonably priced? Looking to stay away from downtown/super urban areas but not interested in extremely isolated areas either.

like

How much will be in hand salary if Barclays gives a CTC of 25 LPA?
Is the employer contribution towards PF deducted from the CTC?

like

Corporate Development M&A Comp, WLB, etc.. at a big Shop (Blackrock, Blackstone)

like

Problem gambler here having to deal with multiple payday loans. Does anyone have any advice on how to handle? Been in a situation with pay day loans in the past? Any guidance would be great.

like

Are any other paralegals struggling with billable hour requirements? I have 5 years of experience in corporate transactions, M&A, and real estate work, and have never had billable hour requirements. I'm in a new job now where I do and I'm nowhere near meeting them. I'm getting a lot more admin work than billable work, and it's stressing me out.

like

Anybody going to join TCS hyd on 10th or 12th. ?

like

When a recruiter tells you the salary range in the first interview, is that firm? Is there typically room to negotiate above this if you're chosen?

likefunny

recommendations for firm with good real WLB and open to remote work in Dallas or Atlanta? Doesn’t have to be true big law. Terrified to get it wrong…again… for the 4th time. 6th year Real Estate.

like

Anyone here from EY ? How’s interviewing there? Help EY

likefunny

Additional Posts in DataScientists Network

Looking for thoughts.

I have 13 years of experience as a Business Analyst at various capacities (Regulatory, Data Verticals) in Banking with Bachelors degree in Accounting & Semi Qualified management accountant

Currently pursuing PGP in Data Science and in verge of completion.

Alongside I have got certifications in Cloud & Visualization tools.

Now I want to move completely to the Analytics/ML side. Will the companies consider my BA experience in Data & Regulatory experiences?

like

Microsoft Azure
Aws
Developer, administrator, architects

Remote support from india, for USA customer

like

Hi,
Can anyone tell me what kind of questions to expect in the 2nd round of interview with BCG Gamma for Predictive Analytics?

In the 1st round of interview, I was asked ML questions and basic Modelling approach on 2 cases.

like

How to complete Azure associate data scientist

like

Need help with preparation for AWS Cloud Practitioner certification.

like

Why are we as a group not so active in helping each other ? I understand there are many members who joined with aspirations to learn DS. But existing members from DS are not active. Sad sorry

likehelpful

Hi Folks,
EY GDS is hiring for multiple roles spanned across Data Visualization, Data Engineering, Digital, Data Science, QA, RPA, UI. DM me for complete Job Description and Referral for a suitable opening.

Post Photo
likehelpful

I am a data scientist with 5yoe, highest qual - masters(data science) , current ctc -18 ,
Planning for a switch in next 3yrs what is the salary i can expect

like

Other than MAANG, which companies are good for the data scientists in India (both product and service)? Good in terms of project complexity, rigor of analytical approach, scale of problems, implementation opportunities, usage of latest tech stack, innovation, mlops best practices etc.

like

Microsoft Azure
Aws
Developer, administrator, architects

Remote support from india, for USA customer contact on 8851891563

like

QBurst is hiring data scientists with 3- 6 YOE. Deep Learning, TensorFlow, PyTorch, ScikitLearn, NLP preferred. WLB is awesome, appraisals happen twice a year and permanent wfh option is there.
DM me for referral. Happy to help :)

like

Hi Fishes, I have 2 YOE in data science and my current ctc is 11 lpa. How much hike can I expect while switching jobs? Thanks!

like

Hi Fishers, currerntly at tcs 8 exp with 25l salary as a data scientist.. what hike should i expect next.

like

I am currently working as analytics professional. Some of the work included but not limited to are Insights generation, root cause analysis, creation and maintenance of data pipeline for owned charters and projects.

Regularly used tools are SQL (Hive, Trino, MSSQL), python, dashboarding tool.

Currently looking out for opportunities. If you have something into data science and analytics then please let me know. Will share my resume with you.

Regards,

like

What is the salary range for data visualization at Baker Hughes for 1-3 years experienced professional

like

Does Legato Health Technologies matches counter offer usually ? How many days before its better to ask them before joining?

Tech - Data Science Role

like

Hi Fishes,

I've got offer from Dunnhumby, Fractal and Gartner for the role of Data engineer. Which one should I go for?

Tech Stack - Python, PySpark, Azure Data Factory and Synapse analytics.
YOE - 2.2
Gartner - 16LPA
Dunnhumby - 15 fixed + 1.5 Joining bonus+ 10% variable
Fractal - 13.5 LPA

likehelpful

Hi @all

There are following open position. Affine

Principal Data scientist - Experience -7-10 year's
Principal Business analytics - 7-10 YOE.
Data Engineer 4-7 YOE

DM if you are interested

like

I'm a Data scientist with 5 years of experience (all experience is in data science). How much CTC can I expect ?

like

I am having an experience of 9+ years into analytics domain with hands on exposure to data science and machine learning.
Please can you refer me in your organization for any suitable position.
Thanks in advance.

like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal