Hi All,

I am an ETL developer who got a chance to work on spark recently within my organization. Initially struggled and now got comfortable with scala spark in my project. I am looking to change job in data engineering but most JDs I see are asking for pyspark and not scala spark, only few are asking scala or Java or python.

Should I really learn python and pyspark as well to be able to find more opportunities? Or should I only apply for scala based roles?

Also I heard from one of the friend

likehelpful
Posting as :
works at
You are currently posting as works at

So if you know Scala and/or Spark, PySpark will be comically easy to learn. And you should learn it since you’ll have to work with data scientists, who almost exclusively code in Python/PySpark.

like

The learning curve wouldn’t be huge. Especially if you know what you are trying to code. There are enough documentations out there that provide you with the equivalent code in sparkSQL, pyspark and Scala

like

Scala is a huge asset to know. PySpark is great for working with other people. PySpark engineering will pay X. Scala engineering will pay X+20-50% because there are fewer people that know it. I got raked over the coals finding a Scala engineer to support a legacy process until it could be converted over. So, in an ideal world, learn PySpark and learn to convert your code from Scala to PySpark - and you'll make bank.

As a heads up most folks are coming to PySpark from Python on more the software engineering or desktop ML learning end... There are major difference between cloud ETL and those other topics.

like

Everything Director 1 said is correct.

Yeah as S1 said, you should learn it and it should not be that difficult. Python is also used stuff like ML and lambda functions so it’s quite useful and will help you a ton going forward

like

Try Spotify

like

How did you got into that position? I am looking to change or go into Data Engineering

like

As for me, by accident. One day I was told to fix a stored procedure doing ETL work, then had to move that logic to SSIS (picked it up on the fly), then was asked to do a pilot on spark to see performance improvements.

At the end, somehow people called me a data engineering expert. I was a web developer.

likefunny

Wouldn’t worry too much about learning pyspark before applying since knowledge of any spark api should be enough to get in the door. there’s something like 10 languages that can interact with spark so the core of it is that you understand spark

like

+1 on PySpark and adding to that,a lot of the Scala advantages don’t matter in the Databricks notebook environment as an example . Notebooks don’t support features of IDEs or production grade code packages, so if the intent is to strictly work with notebooks, don’t expect to many benefits from Scala’s advantages

like

Related Posts

Looking to break into investment banking M&A as a lateral analyst. Been doing Valuations for almost a year (graduated spring 2018 from ug). I've practiced modeling and finance questions. Cont...

like

Fish that transitioned from Citi to Goldman Sachs
What do you miss from Citi Or what did you hate?

What would make u want to go back?

Did you feel like the workload was less when you got to Goldman or did it get harder?

like

Hi guys,

I am into java full stack development with 4 yoe.
I have two options
Accenture with 11.8 fixed + 20% variable. (Has joining bonus)
Amadeus with 16 fixed.

I am more concerned about the technology (angular,java,cloud), learning and job security.

I am confused between them. Please suggest. TIA.

like

Hey Folks, need help in choosing between American Express and JPMorgan Chase. Seeking insights on basis of WLB, Work and brand. YOE- 7 years . Skill- Data Engineer. Also is it worth to switch from Deloitte India given the current market scenario. TIA

like

People on H1b. How have you managed your client travels pre pandemic? Does Deloitte do an amendment every time you start an engagement and travel to a different client location?

USI Producer Job, how is it? Currently an Account Manager and looking at this opportunity. Also, really any advice from service to a sales transition is much appreciated!

like

Been asked to present my current client’s plan and PCA in second interview for a role. Am I being over cautious by not feeling comfortable with this?

like

I want to leave my captive company. I don’t think I am ready to be totally on my own even with a cluster group etc. There is an option to be subcontractor in another independent company. I can write she said like 300 companies in many states, it would be helpful because I have many friends out of state. Is taking 25% of sales for giving CRM and companies too much? Seems like a lot. It is like “owning your book” but I assume I would pay a high price if I want to totally manage myself in a futu

likehelpful
like

Would an MBA be required to switch from Product Management within LS/biotech to Tech (faang or other b2c)?

like

Question: I have a client who started a restaurant. He leased the land and bought shipping containers where he spent money contracting out to turn them into bars. I’m having a tough time determining how to depreciate this! First thought was leasehold but he’s not making improvements to a structure...he owns the structure! But nonresidential real estate at 27.5 years doesn’t make sense either as it’s not a brick and mortar building. Curious on others thoughts! TIA

like
like

Hi,
I'm thinking of changing my career to data analytics. What course should I be doing?

like

Currently at Big 4. Have offers for a PL/Principal at MBB and for a VP role at a Fortune 20 company. The VP role is more interesting and comp a bit better but I’m told would accelerate faster at MBB. The initial plan was to end up at MBB. What’s the best for future flexibility?

like

I am not in account but would love to have a more comprehensive take away from meetings particularly client feedback verbalized. How do you account folks do it? Tips and tricks? Thank you!

like

How is the project for 'Tyson' client?

like

Interesting all the noise about these PPT, my take:

The pandemic took all the upside of an IB analyst life (client meetings in person, fancy trips, suit up, worshiped at networking events) and left all the juniors with the downside (long hours, monkey-like type of job, aggressive work environment).

Not that the complaints are not sort of legitimate but you may need to revise the decision-making process that led you to accept a return offer. Specially if you were looking for less hours.

like

Anyone have experience working CWX side of Facebook (Meta) ? These are the full time contract roles that potentially turn into permanent roles directly with Facebook (Meta). Had a recruiter reach out and offering me comparable TC and such, just curious if anyone has experience in these roles and success/failure of transitioning into permanent role. Thanks in advance!

I've cleared 2 rounds of interview , Technical and Managerial for Luxoft Poland. They are also in process of going for Client round. I opted for Advance hiring process. Will there be offer only after Client round qualification from Luxoft?

Can anyone please help?

like

Could someone explain what finance closing process is? I passed CPA and have audit experience but I don’t really understand how industry accounting cycle works :( What does it mean to close the books? Are they referring to 12/31 deadline? Do all JE have to be recorded within this date? How does one close it?

like

More Posts

When you fake it, till you make it.

Post Photo
likefunny

which is better in terms of work life balance
ZS vs Whitehatjr?

Is it easy to change projects in Tiger Analytics.

Also how is the performance bonus paid in tiger? Monthly or at the end of the year only

like

Request you all to provide 11 likes to unlock the features .
Please

like

What is a good alternative to coffee to stay awake throughout the day? (other than sleeping 8 hours...)

How to make a jump to cloud security when I just have SOC experience? Currently studying for Solutions Architect cert

like

Anyone take the WI Real Estate state exam? Any tips on what to study?

like

What is the salary range for Manager and Associate Director in Deloitte Consulting?

like

Anyone who works at Stripe able to provide some insight on culture and work/life balance? Is it really as toxic as the reviews seem?

like

Hello everyone!!

I have been into graphics designing for 4 yrs and now heading towards UX Designing but I'm feeling kinda lost because everybody wants experienced person so how freshers gonna make it.


Any advice how to move ahead or any job referral for internships or entry level ?

like

Just wanted to confirm if we have an option for buyouts during the NP,is it normally negotiable to 30 days,how does it work?

like

Cognizant Will cognizant give a Diwali bonus?

funnylike

Hello Fishes

How is the WLB, annual hike, work culture in MasterCard?
Role- Middleware

Any insight to PwC tuition reimbursement?

like

For any Alvarez and Marsal fish, how has your work life balance been? Very interested in hearing from those in their healthcare industry group, but anyone is welcome to respond.

like

We are looking for Search Language Specialists - Manual Software Testing for our Team.

Requirements :
Primary language proficiency (Read/write/Speak) as Native speaker : Marathi & Bengali (Mandate)

Work Location: Bangalore

Work Type : Hybrid

Degree : IT/Technical Degree

SoftSkills : 
Good Communication skills and willingness to work on internationalisation Localization and Translation query.

Good Team Player.

Experience : 3 years

like

5 years of experience and I have brought very good results in Lead Generation and E-commerce with Digital Marketing.

I am skilled in SEO, Paid Ads, WordPress Website Development, Analytics mainly Performance marketing.

In e-commerce I have brought good conversion rate just by improving the ui/ux of the site.

How much do you think I can be paid in Navi Mumbai?

Context : I started from a startup agency and currently working in a Big well known agency.

like

Hi Fishes,
What should be the salary for Manager / Sr Manager having 14+ yoe in project/product service operations? I am at 32 lpa which I think is quite low having worked with top consulting firms for close to 12 years.

like

Anybody with Big4 exp made the switch to salesforce as business architect recently? What does this job entail and qualification needs? The jd says “extensive” big4 exp

like

Additional Posts in Data & Analytics Consultants

Got Amazon career essentials (ACE) assessment to complete for Business Analyst role
To anyone who has experience with it- What kind of work simulation questions should I be expecting?
Is there a way I can prepare for it or any resources one must know of?

like

I have been working in healthcare clinics and health techs for a few years now. I’m an independent consultant. Bust, a friend of mine referred me to a big bank job. The role is of data governance manager. What should I know for the interview, or how can I prepare for it? Any thoughts? They told me I don’t need financial background. Data is data, but they wanted to know metadata management and how do I build frameworks/implement enterprise systems.

like

What are the top sales & marketing metrics that tech companies track? I have an interview for a Data architect role and the data aspect is my strength but need to brush up on the business insight part

like

Does your company split data science roles into different practices like data & analytics and software engineering? Does this inhibit collaboration & Where does a data engineer fall?

like

My manager continues to ask me to help other teams and do work on other billable projects. I’m sick of it .. and I’m going to explicitly say that no I can’t do it because my hours are taken by the existing project we are doing. Is this appropriate?

like

Is 128k a fair base salary for Consultant in Deloitte Analytics & Cognitive. MS CS and about 3 yrs experience. In one of the fastest growing but midtier tech cities in the US.

like

Anyone have a ballpark range on dataiku price per seat or however they price? It looks like an interesting product.

Anyone in the KPMG Lighthouse practice that would be willing to take a look at my husband’s resume and put in a referral? He’s interested in the Associate, Data Scientist position.

Hiring for data scientists and BI analysts across a couple levels, DM me if interested!!

like

Docket + R - I have an container running a rocker image and I’m trying to launch a bash terminal and the RStudio server (/init) at the same time. Is this possible? I tried adding CMD [“/init”] too.

like

Mεntal mαth in iηtervieωs. Thoughts, tips, rants, grievances?

like

Meeting with a recruiter at one of the FAANG companies. I won’t be able to move jobs because of H1b /GC at least for the next year. Any idea if I should start off with that or should I just try to understand the work that they have been doing?

like

Can anyone at Ford Motor Company tell me more about the “Learning Analytics and Engagement Specialist” remote position? Very curious and would love a referral!

like

1) Hypothetically if you were looking to sabotage a colleague, can you code an excel document to throw off their numbers or calculations, (if you send them that document and they download it for use)?

like

If you were to hang your own shingle as a consultant what BI tool would you use to deliver dashboarding to clients?

Assume that you aren’t plugging in to source systems as I primarily work in CDD.

like

Any conda users in here have to install a binary and use conda forge? Need some help installing pdftotext. Throwing a weird message saying “Found conflicts!” Any and all help would be appreciated.

like

I was hired a few months ago by D to do data science. I am not doing data science currently, is there a way I can switch roles internally? Should I talk to my coach? Or should I just start looking for a new job. I’m starting to feel that the field is so hot rn my small stint at D won’t be a big deal to other firms.

like

Anyone have recommendations on the best app/site to learn Python? 🐍
I’m a complete beginner.

like

Noob question - I want to consolidate the dates into a single dates column. Would numpy or pandas help me reorganize this dataset effectively or should I do it within excel?

Post Photo
like

KPMG folks, thoughts about Lighthouse relative to other big 4s? About to start the interview process. Coming from EY FSO DDnA

like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal