Hi, I want to learn how to build ETL pipelines with Pyspark and Azure. As right now i have experience in feature engineering only with SQL and Pyspark.

Can anyone suggest anything online from where i can learn and do hands on?@

Posting as :
works at
You are currently posting as works at

Follow these preparation strategies below.
1.) Sql
2) Data modelling and data warehousing
3) DBMS concepts
4) problem solving skills , DSA is suggested (Python or Java or R )
5) distributed framework (Hadoop ,mapreduce, spark )
6) data warehousing tools or analysing Apache tools like hive, sqoop, hbase, impala, Cassandra
7) cloud technology (AWS or azure or gcp )
8) system design knowledge and their different theorem like cap, paclec and types of scalling of distributed system

Recommendation:
1 . Learn Hadoop architecture from https://youtube.com/playlist?list=PLdW3asq7z3KlWy24tinxPDM_ewIXbt0I2

Understand the spark architecture. Ask 2 question in every topic why they have designed framework like this what if it doesn't designed such a way . Relate with map reduce because spark replaced MR .

Start coding on leetcode or codechef or any coding platform .

Thanks.

like

From where to learn system design for data engineering

Related Posts

Hey ladies, how to make 2019 less toxic and more of self love in this consulting world? Suggest your own ideas.

likehelpful

Quick question - are online coding schools worth it? I’m thinking of a career change and I’ve thought about a coding Boot Camp but saw an online full stack developer course by mammoth interactive. Would I be wasting my time and money with something like this if I wanted to go into an industry as a coder? Or should I just go back to school?

like

Anyone at UHG? Someone within HR? Need to pick a brain on a few things..levels, merits, etc

like

Has anyone ever done interview coaching or anything similar? Looking for resources or tips because I’ve had the chance to interview at some great companies but I keep blowing it...

like

Switching careers into UX, recently finished a UX bootcamp. I can’t get a straight answer from the instructors on UX job title definitions and responsibilities (cont’d)

like

Ex-consultant now in marketing. Used to be an avid follower here and got great insights on companies when trying to exit consulting so thought I’d try here for advice. Outside of FAANG, what are good large CA-based tech companies that provide good pay, WLB, and culture? DD has great people and WLB, but pay is low and you’re not given the tools and resources to succeed + truly make an impact. Google and Apple are ideal but need backups. Like is Salesforce still good? A company along those lines.

like

I’m an associate copywriter at a digital marketing agency! My only client is Instagram which is SO cool. A goal of mine is to work in-house at IG. Is it common for clients to bring creatives in-house from agencies they work closely with?

like

Time to negotiate my raise. What should a managing director for a national, non B4 firm make?

like

Anyone know of salary expectations of a RBC Enterprise Strategy Director salary? I saw non-Strategy Directors are at approx ~$150k base with $40k. I have 10+ YOE in consulting and product management.RBC

like

How do you help younger teams think more critically about the work they bring to you? Especially when they often get paired with more senior creatives who end up picking up the slack for them. ...

like

How’s Decoded as a place to work?

like

Any leads for referral in Myntra, Oracle , Walmart for data scientist position? Please help.

like

Hi, I am currently working as corporate communication officer with 6.5 years of experience. How much salary should I expect my current CTC is 7.5Lac PA.

like

For those who have moved latterly in the last 3-6 months, have you noticed a change in salary range given the rising cost of living?

like

Can some one refer me to a position at FIS Global, I have the job ID.

Any experience with Hodinkee insurance for watches? My research says it’s a better deal than scheduled on HO policy. Better coverage and $0 deductible. I have 3 Rolexes and a Breitling.

like
like

Additional Posts in Data Engineering

Hi,

I am looking for job change and any help or refferals would be appreciated. I have 2+ years of experience working on Azure, MSBI Tech stack, Apache Airflow, etc.

Preferred location:- Bengaluru, Hyderabad, Pune.

Thanks in advance.

I have an offer from DXC technology (7.5 fixed + WFH for 8 months) and 3i infotech (9.25 fixed + 0.75 variable). Can anyone help me to choose the company?
YOE - 2 years
Role - Data Engineer

like

Hi everyone, I am 4.9 years experienced , I joined Fractal 3 months ago as a data engineer, but still haven't got any project. Process is so slow that I have received only 4 to 5 project calls. I switched in hopes that will learn the new tech and tools. But now i am starting to forget even the one i knew. Should i go for change or wait a little more time as fractal is reputed in AI/ML? I have interview lined up with Impetus as for now. Would it be better choice?

Anyone working at Autodesk ? How is the work life balance , work culture , hike percentage on appraisal there . Thanks in advance .

like

Can someone please refer me?

GCP Data Engineer

Experience: 4 years

like

Hello
Can I pl get help in SQL
Thank you

like

For a lead data engineering role, which is a better company to join majorly in terms of work life balance ? Paytm or airtel?

like

Hi All,

How much DSA level is required for DE interviews as I am DE and preparing for good product based companies or you can please let me know how you have prepared for DE interviews ?

Currently focusing on sql -> python -> pyspark -> data modelling

Any suggestions

like

What do you think are the new things to learn in data engineering in Azure platform

Hello,

Can someone suggest a few good resources to learn about designing data pipelines and how to gain hands on experience on the same ?

like

Hi Fishes, anyone here who have attended TrendyTech's Big Data course by sumit mittal sir
Please DM me

like
like

Can anyone suggest a good course/Set of courses to become Azure/GCP data engineer

Currently I am a Microsoft SQL DBA, 8 Years exp looking for suggestions

like

Hello everyone, I am Data Engineer with skills on Azure and am holding couple of offers KPMG Digital Lighthouse as Consultant : 12.5 LPA fixed +20% variable +1.25L bonus Tiger Analytics as SE : 14.5 fixed YOE : 4.1 years Can you please help me with which is better in terms of growth and WLB with your insights ? Tata Consultancy KPMG KPMG KPMG India Tiger Analytics

likehelpful

Hi All, I've offer from TheMathCompany Inc
Impetus technologies inc
Tredence Inc.
dunnhumby
for the role of Data engineer.

Salary is kind of same in all of them . TheMathCompany has permanent WFH. Which one should I choose ?
PS - I've 2.2 YOE

like

What are some good product based companies to work as Data Engineer?
Good in terms of learning, interesting work and good work culture.

like

Please suggest

YOE 2.4
Azure Data engineer

Fractal 14.5
Roch product corporation 18.5

like

What to choose - McKinsey or MasterCard for data engineering role?
YOE - 2.2 Yrs

like

Hi All,

I'm looking for referral in data engineering.
I'm having 2 year of relevant experience in data engineering.

like

I plan to enroll into the big data trendytech course ? Any honest reviews ? Is it really that good as claimed ?

like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Send download link to your phone

OR

Scan your QR code to download
Fishbowl app on your mobile

By continuing you agree to Terms of Use and Privacy Policy.

Messaging rates may apply

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal