Best resources to learn pyspark for data engineering?

like
Posting as :
works at
You are currently posting as works at

There are a number of excellent resources available to learn Apache Spark and Databricks, including books, online courses, tutorials, and documentation. Here are a few suggestions:

1. "Learning Spark: Lightning-Fast Big Data Analysis" by Holden Karau, Andy Konwinski, Patrick Wendell, and Matei Zaharia. This is a comprehensive and well-written book that covers the fundamentals of Spark, including RDDs, dataframes, and SQL.

2. "Spark: The Definitive Guide" by Bill Chambers and Matei Zaharia. This book is published by O'Reilly, and it covers a wide range of topics related to Spark, including architecture, programming, performance, and deployment.

3. The Databricks documentation. Databricks provides extensive documentation on Spark, including tutorials, examples, and API reference. This is a great resource for learning the basics of Spark and how to use it with Databricks.

4. Coursera's "Scalable Machine Learning" course. This course, taught by Professor Ameet Talwalkar of UCLA, covers the fundamentals of machine learning and big data analytics using Spark.

likehelpful

Are you using Databricks or AWS? There are a number of udemy courses with a good amount of reviews that cover the basics. DataCamp also has a course, but their format removes you from those two environments which are necessary skills to have in my opinion and don’t give you a full understanding of how to design compute-responsible code.

like

I’m using databricks. Thank you for the detailed info!

Look up Ramesh Retnasany on udemy, thank me later

like

Databricks academy is a thing…

like

This.

Be sure to understand spark first before diving into how to use pyspark. How you write code has a big impact on performance.

Related Posts

If I accept the promotion to Partner/Principal what's the least number of years I could work without it being a financial mistake. I have an opportunity to do something completely unrelated that's my passion, but I feel like that's the wrong decision when I'm this close.

The other opportunity should still be there for 2-3 years

like

Currently a Wealth Advisor for Citi but I honestly hate it here with a passion it’s the worst company I’ve ever worked for in my life and I’ve been in Wealth Management in the banking channel for over a decade. Series 7, Series 63, Life, CFP, ChFC, and soon to have my MBA with a focus in Mergers & Acquisitions. I’m look to enter another area of Finance at this point instead of starting over somewhere else from scratch, but not sure what direction to go. Interested in Investment Banking or M&A.

A CRO offerred me an internal consulting/biz dev role. They offered 145k TC (20% bump) and slightly better benefits. They also offered accelerated path (6mo to 1yr) to MC. While promotion to MC at my current company is at least 1 year away. The job is interesting to me and fits my careery goal. But my big concern is that I've only been with my current company for a year (this is my first job, post PhD). Would this affect my resume in the future in terms of being viewed as "job hopping"?

like

Considering joining a small 50-person company as the only attorney (prior attorney is retiring). (I am currently in-house at a F50). For others who have done the same path, what surprises did you have when moving from a large company to a small one?

For example, it sounds like I should not expect a document management system, based on a prior post in this bowl. I’m guessing expenses like state bar fees and cell phone are not reimbursed.

Any advice on things to expect (or not expect)?

like

How recession-proof are in house roles in finance (e.g., hedge funds, PE firms, etc.), especially when newly lateraling from Big Law?

like

How long did it take you to find the right position the first time you went in-house? I am 5 months into it and regretting turning down the (lowball) offer I got a couple months ago.

like

Audit A2 in NYFS right now. Is there any hope for me to get into FDD now, and does it even make sense for me to jump at this point rather than make senior first? I need some advice. No deal experience

likefunny

Equity research associate considering to pivot into IBD. Is this a wise choice? Heard ER is an exit for some of the IB folks out there.

like

What is the experience required to become senior manager in US taxation

I don’t want to do this auditing stuff anymore...but I don’t want to quit without a job and I don’t want to leave my team while we’re in busy season.

likeupliftingfunny

Looking for some material for a crash course in IT strategy consulting - books article etc. any recommendations would be appreciated

like

If you’re ever feeling like BigLaw is pigeonholing your career, just remember that Rep. Liz Cheney started her career as a project finance associate at White & Case from 1996-1999.

likefunnyuplifting

Can anyone confirm hierarchy in KPMG ?

Is Assistant Manager equivalent to Manger or Senior Associate in the other big four ?

Book recommendations on leadership, career development, etc?

like

I’m planning to buy the study materials for the SHRM-CP certificate through Amazon. What are the best ones to obtain?

likehelpful

Been thinking about a career switch. Am a lawyer with top experience in energy and construction transactions and have industry specific insights and knowledge. Wondering if the management consulting firms would be an interesting place to go. Also have MBA for what it’s worth. Thoughts?

like

has anyone made the jump from hospital administration to healthcare consulting?

likesmart

Best books about product management for a good exit to industry? I already have an MBA and PMP but looking for a refreshed perspective since it’s been years since my formal education.

Currently reading “Product Managements Sacred Seven” but it’s geared toward the startup world. I am in healthcare and not looking for a startup role. TIA!!!

like

Any good books for someone trying to understand the business side of marketing? Just got my review an they're asking I bring more of that MBA mindset to projects I'm on.

like

More Posts

How are experienced hires vs recent grads treated at top consulting firms? I understand that each branch of each firm will be different from the other and it depends on the leadership. But have there been any uniform truths that people have noticed?

like

Will there be any hike if I moved to associate 2 in pwc ac, if yes how much can I expect?

I wake up every night holding my breath lmao. Idk if that’s classic sleep apnea or if my body is subliminally trying to off itself while I’m not paying attention.

funnylike

Thinking of getting a peloton. Anyone actually experience weight loss and overall health improvement from having one?

I make excuses for not going to the gym and the number 1 being convenience so I figure if it’s in my actual home then I have no excuses!

Be nice, thanks! 😊

like

Starting a new job in Pyrmont in January. Where should I live? I'm a single female in my early 30s, looking for 1BR ~$500/w. Ideally somewhere near the water/beaches.

Been doing some research and Pyrmont is a pain to get to. I'm new to Sydney so any advice is much appreciated!

like

Wondering what people think about my situation. I’ve been offered a role with a company. The job description didn’t mention college degree as a requirement. On my CV I have a section listed as “Education” and under it I have “Business Studies” with the university name and the 3 years that I attended the course (2.5 to be exact). It’s a 4 year degree and under this I have my High School equivalent with the years I attended there. The education wasn’t mentioned in the interviews. Continued below.

like

Jummah Mubarak Fam ☪️💚

like

How is service line consulting in Deloitte India (not USI), is it only applicable for Indian clients? Has anyone joined recently? Faced any issue with Work life balance ?
Deloitte Deloitte India Deloitte USI PwC EY

Do you shoe check others at the office?

likefunny

Spending 48 hours in Belgium this summer. Do you recommend Brussels or Antwerp?

What is expected salary for Senior Manager @EY Dublin Ireland? Looking to relocate, please help with your information, thanks EY

like

Hii Fishes, I am java fullstack with 9 yoe, currently based out of thane Mumbai have offers from HERE Technologies mumbai goregaon , HSBC India India pune, Citi pune. All are giving almost same CTC with almost same role. Which organisation should i join . Also I have question if pune will suit me or not. Citi is having poor wlb as per Glassdoor and some linkdin feedback. How is hsbc bonus , job security, wlb. Thanks

like

Hey Fishes 🐟,

Where can I get study material for SAP IS-U ? I visited lot of sites but couldn't find detailed explanation for each concept.

like

Hey guys, question regarding new years resolutions. Anyone know some good books about how to lead better and step up to become a great CD?

I want to work for The Walt Disney Company corporate offices. Anywhere. I don't care where there's an office. I want to work for that company. I don't know how to drive in rain or snow, but if they have an office in places that snow, I WILL MAKE THE SACRIFICE 🤣

But I still have questions though 🤔 -

What is the culture like?
What are the work hours?
Does Disney pay for relocation for new hires like some of the FAANG companies do?

@

like

Hello guys,
I am looking for opportunities in Risk Management, Risk Reporting, Model Development, Model Validation. Do ket me know if there are any vacancies in your companies.

Any Slalom 🐠 lurking who could share insights into a Salesforce Practice Lead role posted on LI?

Has anyone here worked at or worked with CMS Payments Intelligence (CMSPI)? I was contacted by a recruiter for a position there and have a first interview on Monday. I'm currently pretty happy with my current job so this would have to be something special to entice me to leave. Would love any info on the culture etc.

Thanks!

like

Current and past employees of Comcast, tell me everything I need to know about the company for the interview.

like

Anyone use edgenuity? I'm starting Monday with students, never been trained, no guidance. Don't know how to set expectations if I don't know. ice breaker ideas? I'm not in my classroom.. media center.

Additional Posts in Data & Analytics Consultants

Have you ever failed to deliver? What was it and why?

like

In the most unnecessarily complicated phrasing worthy a dissertation, what are you working on this week?

like

could i dm someone to criticize/suggest feedback on my resume? i’d like to believe i have good experience but unsure why im not receiving any bites

like

Any recommended courses or books on customer care/cust success analytics or marketing analytics in general?

like

Anyone good in MySQL?? I’m struggling

like

Can anyone at Ford Motor Company tell me more about the “Learning Analytics and Engagement Specialist” remote position? Very curious and would love a referral!

like

Is it realistic to join KPMG Lighthouse or Gamma without a Masters degree but with previous experience as a data scientist at a big data startup?

This might be a long shot ... but I have a Statistics assignment due for tonight in a few hours

Does anyone know how to do Linear Discriminant Analysis with more than one predictor - by hand? I know how to compute in R but my professor wants it done by hand .... fml

likefunny

Do any of the top consulting firms have analytics roles (preferably not super technical) that don’t require weekly travel?

Asking for a friend:
What’s the interview process for a FSO Manager in EY joining the DnA BI team like? Has anyone been through the process recently? Thanks

like

Any advice on which platforms are best for learning data science/ML? I'm looking for a platform through which both I can develop my intermediate ML skills and my less experienced teammates can learn data analytics from the ground up. Particularly interested in any thoughts on Codeacademy and Dataquest.

My company issues top of the line 16” MBP’s for engineers and lower end 13” MBP’s for all others. I work in analytics/ data science and struggle with the fact that my computer can only run 1 external display! No one else on my team cares. How can I navigate requesting a more expensive computer without annoying my leadership? I already own an upgraded MBP as my personal - can I offer to swap my work and personal machines?

like

Coming from ACN, senior analyst, 3yrs total experience so far & wanting to get into the data analytics/BI work. Currently have had experience working with SQL developer, OBIEE/OBIA, Teradata in ERP and Healthcare projects. I also have my AWS Cloud Practitioner Cert, hoping to eventually get the Data Analytics Cert later. What are my chances of finding a role with another company that'll let me work in BI/data analytics and build up my skill at the same time? Been thinking of leaving ACN.

like

I’m a college student studying data science. I’m interning at Deloitte currently is to become a data science consultant for non-profits. I don’t know if this is a role that even exists, let alone how I would start down that path. Any tips or advice would be greatly appreciated!!

like

Anyone works/heard of DMV Group? Mind if i shoot some questions about them?

Anyone in a non-technical analytical consulting roll?

like

Best/most used SQL Dialect? Been practicing on Leetcode and am curious as to which SQL Dialect is most used in up to date companies.

like

It’s been ages since I’ve taken college statistics. Can anyone recommend an online course where I can review and brush up?

like

What’s the different between business analyst and data analyst? Sometimes I see them merged or people say “business/data analyst”

like

How is the Deep Learning specialization on Coursera?

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal