Hi, I want to learn how to build ETL pipelines with Pyspark and Azure. As right now i have experience in feature engineering only with SQL and Pyspark.

Can anyone suggest anything online from where i can learn and do hands on?@

Posting as :
works at
You are currently posting as works at

Follow these preparation strategies below.
1.) Sql
2) Data modelling and data warehousing
3) DBMS concepts
4) problem solving skills , DSA is suggested (Python or Java or R )
5) distributed framework (Hadoop ,mapreduce, spark )
6) data warehousing tools or analysing Apache tools like hive, sqoop, hbase, impala, Cassandra
7) cloud technology (AWS or azure or gcp )
8) system design knowledge and their different theorem like cap, paclec and types of scalling of distributed system

Recommendation:
1 . Learn Hadoop architecture from https://youtube.com/playlist?list=PLdW3asq7z3KlWy24tinxPDM_ewIXbt0I2

Understand the spark architecture. Ask 2 question in every topic why they have designed framework like this what if it doesn't designed such a way . Relate with map reduce because spark replaced MR .

Start coding on leetcode or codechef or any coding platform .

Thanks.

like

From where to learn system design for data engineering

Related Posts

What are some of your recos for the best comms/PR books and sites to learn more deeply and further your knowledge in your role?

Anyone have any thoughts on Primary vs Secondaries PE as a career after coming from Tech M&A? Wondering if Secondaries is a good move?

like

Anyone leave PA to work at a firm? What’s your experience been like? Any regrets? Tips?

like

Thoughts on getting CFP when done with CPA (1 test left) and pivoting from audit to personal financial planning? Estates, taxes, savings, etc. Or should I just stick the OG plan and go into industry FP&A when I make senior/CPA?

like

How early in someone’s career do you generally identify if a homegrown person is partner material or not? M and up or earlier?

like

I studied finance in undergrad and had a couple of internal audit internships. I only started a few month ago but I never saw myself doing IT Audit and I don’t like it thus far! Would I need to get a CPA in order to move into audit or something more finance/accounting oriented like FDD, Valuation, etc.? (Still need to get to 150, thinking of doing an online MAcc)

like

How long did it take you to find the right position the first time you went in-house? I am 5 months into it and regretting turning down the (lowball) offer I got a couple months ago.

like

Hey all, started my role as a recruiter at an agency recently and had my first placement! It was great until the candidate wants to renegotiate the base and benefits after the current company counter-offered when the candidate no longer sees the future. Any tips on what to say to bring back to focus? The current role is a Sr. Acct Receivable (less than 2yr) and going into collections specialist at a global law firm.

like

Is the PGDM course from UpGrad good for moving forward in the data science domain. ? Please help me as I’m thinking of pursuing something alongside work as I cannot leave my work due to financial condition. I wanna excel forward in this domain. Currently having 2.5 YoE in data science and analytics.

like

Hi Fishies,

I hold two offers, one from Capgemini (14F+1 Joining Bonus) and UST (12 F + 1 Variable). Which organization should I choose? Which is better from career perspective. If I ask for Counter offer from UST will they match Capgemini's offer?

like

Looking to exit from 195k base to a sales role base 140 + 40 in guaranteed bonus year 1 to make up for the lengthy sales cycle. I know in the long run total comp through equity and sales commission bonus can be much higher but having a hard time making the jump… mostly my ego to drop drastically in TCO the first year after spending all these years trying to increase base. Still interviewing - unicorn saas account exec. What would you do?

like

Is It better to join a PE Tech company in an impending recession or stay with a public company ?

like

How is Persistent in terms of Appian projects? Does it have good development projects in Appian and what are the chances of growth?

like

After nearly 20 years in the military serving in T&E and HR roles, I’m considering moving on and have been conducting lots of research into consulting. BS, MA, and SPHR cert - no MBA. I’m looking for suggestions or advise about transitioning. DC Metro area is likely landing spot.

like

Any designer transitioned to a product manager role? How did you do it? (Outside of internal work transition, more interested in acquiring a new role in a diff company from designer to pm) thanks!

like

Any advice on how to switch from big 4 to tier 1 consulting firms like BCG and MCK?

like

Top 10 books to read about consulting life ?

likefunny

Hi Folks,

Hope you all are doing well.

I have multiple offers in hand from the below-mentioned companies.

Need your inputs and suggestion regarding which company would be better to join in terms of career growth and opportunities in Data Engineer role.


ITC Infotech
Telstra
Teklink International
Factspan analytics
Bosch

All companies are giving offers around 18-20(Fixed +Variable).

YOE - 3.4 years

Tech Stack - SQL Server,SSIS,Azure ,ADF,ADB,Pyspark,Azure Synapse, SparlSQL

like

Is it worth it to go back to school for a better track to a FAANG company? The division I’m currently in seems like there are no good opportunities, and I have the funds available to finish my undergraduate full time (Canada). I applied to Amazon before for a highly senior position and made it to final interview so I think I would have a better chance for a new grad type role.

like

Has anyone taken the below course for learning Microservices with Java, Spring Boot, Docker and Kubernetes?

Post Photo
like

More Posts

Boston people - I have a set of GMAT veritas prep books for sale for $50. DM me if interested.

like

What's the best way to find a freelancer? My company needs help with work that would likely involve custom development/Apex. The only person with any SF knowledge is myself and it's limited...

When work from office is going to resume???

like

Hey, guys! Need one suggestion. I'm 2022 btech passout from tier 3 college. My current ctc is 5.5 lpa in Harman , working in an automotive devision with a role of devpr(mostly the work is in c++ and linux). I have a average coding skill. Tho, I'm not very good at competitive programming. How to take a switch in big product based company. What are skill sets i need to learn. I'm really confused! Some says learn full stack. I have to increase my pay from 5.5 to atleast 30 in next 3 yr

I’m seeing most of the commission only jobs are mainly cold call sales. Is that the industry norm? I’ve only done inside sales salary + commission.

like

Hi,
Joining SAP labs, Bengaluru on 8th of August.
Looking for a share of accommodation. Female preferred.
Thank you..

like

I shifted from time domain to frequency domain😅

Post Photo
like

Hi Fishes,
I have offers from below org. with almost same CTC-
Birlasoft
Atos
Wipro

Tech stack- ETL developer
Exp- 7 years
Immediate joiner

Kindly suggest which one is better?

like

Would you rather be a big fish in a small pond, or a small fish in a big pond?

like

Is it a good time to switch jobs

like

So, recently got put on a PIP after six months at my new job., two days after I told my boss and HR I recently learned I have a cognitive disability caused by an illness. The PIP was mostly because I forgot some deadlines, but also they said I need to be more professional in emails (even though others use zero punctuation and a lot of emojis. To clients.) and simultaneously ask more questions and ask fewer questions. Should I just assume this is unattainable and look for a new job again?

like

I just got a new job, but it's not what I expected. I was expecting a career opportunity and a chance to learn and grow in my field, but instead I'm stuck doing the same thing as before, just from home.

like

Hi All, I’m thinking about getting certified in change management but not sure if I should go the Prosci route or CCMP route. CCMP is a lot cheaper compared to Prosci but would it measure up just the same?

I guess it would be similar to the discussion of the SHRM-SCP vs SPHR.

like

How did you find out what you want in life? Feeling lost mid-career..

like

Anyone willing to split a Wall Street Prep premium package course? Total cost is $500

like

Does anyone talk about economic diversity? You know middle class and economically challenged people who cannot afford art schools and expensive private colleges?

likeuplifting

Hi, does anyone know what the salary ranges for L4/L5 tech roles in UK are?

like

Hi, what will be the yearly hike in citi. I have offer for C10 level. 7 YOE.

like

What are peoples thoughts on working for a Banking Company in a Product or Tech role? I know there are mixed feelings since it’s not working directly with interest, etc.

If you currently work in banking, what was your reasoning (besides pay)?

If you don’t, would you ever accept an offer if it in a Tech or Product role?

like

Additional Posts in Data Engineering

Anyone here works in Dunnhumby or Gartner?

like

Hi folks. Just collecting some info.
Please drop ur ctc if u r 6or7yoe as data engineer. How much u r getting and if u switching how much u r expecting. This will help the upcoming data engineer to negotiate better.

like

Hey fishes,

My current salary is 13lpa in SQL, Plsql domain. I just interviewed with mindtree and they are offering 16lpa in big data . Should I switch or negotiate for more. Yoe-4 and new to big data.

Thanks

like

Hi dear ones,
Kindly let me know if there are any job openings with good salary package, in data science field for someone with 1 YoE as an intern. If anyone could refer, it would be much appreciated. Thanks in advance 😊

like

Hi everyone ,need your guidance.

I am working as SAP BODS developer which is a GUI based ETL tool , at EY GDS having 1.3 years experience.
I wanted to shift to data engineering based on cloud technology, so wished to get guidance on what things to learn and is it possible to get into this role in other companies.

TIA .

like

Hi guys I work in One of GCP partnered org. My role sr SQL developer I resigned the current job GOT ROLE in FICO child org as a solution integration consultant role -(AWS,PENTAHO ,SQL we will be using ) -offered CTC 25 fixed + 1L JB + 2.5 BONUS + 0.5 L WFH SETUP = total 29LPA if everything will go smoothly ( bonus vary with comp growth it can be between 1.8L to 2.5l .so final CTC will get down to 28LPA as well - Bangalore but WFH ).Current org retain with 25L +2 LP VARIABLE HYDERABAD COMPANY

like

Hello Fishes, I needed a suggestion on below:
I have been working on AWS cloud services from ~2 years. Recently joined new org. where the work is on Azure Cloud and more into Azure Databricks, Synapse Analytics, CosmosDb etc.
I'm not exactly sure at this point If this is a good move to switch technologies.

Can you fishes pls guide or help me if this Azure Cloud related services stated above are good in terms of current tech stack in market demand or did i made a wrong move? Pls help.. 🙏

like

Why Data Modeler role is on high demand these days?

Guys pls reply…
How is Impetus Leaplogic project for a data engineer ? In terms of technology used and WLB?

like

Tiger Analytics - Senior Software Engineer(Data Engineering), [WFH]

Fractal Analytics - Azure Data Engineer, [WFO]

Airtel Africa - Big Data Developer, [Currently WFH]

Which one to choose, the compensation is more or less the same.

Looking for a learning environment.

YOE - 2

like

What are some good product based companies to work as Data Engineer?
Good in terms of learning, interesting work and good work culture.

like

Epam system(not epam anywhere) or Impetus technologies?

Tech stack- data engineer - spark, hive, hadoop, java
YOE- 8.5 years
Both providing long term wfh option
Ctc almost same

Which would be better to join for long term settlement and growth perspective irrespective of ctc offered?

Please suggest.

like

Hello People,

Just wanted some suggestions i have been working as a python developer for the last 3 years
Now I am thinking of switching to a data engineer position I need your suggestions is it good to switch or a python developer It is good to continue or what's your suggestions thanks in advance

like

I got an offer for 36 in Citi , have another offer from a SBC for 39.5 , do you think Citi can match that offer? I’m willing to join Citi only. Please help.
C12 position offered , 9 YOE.

like

Please suggest

YOE 2.4
Azure Data engineer

Fractal 14.5
Roch product corporation 18.5

like

Hi people,Does anyone have given visa hackerrank interview recently? May I know the kind of questions being asked and their level?

like

Hi fishes,I have around 5.2 yrs of hands on exp in data engg. and able to clear service based companies interviews easily.Now I want to switch to product based companies as I am looking for some good work,good salary but I am not really strong in dsa as it's not part of my daily work.I don't really use trees and graphs as part of my daily work.
How shall I switch to product based companies considering DSA is their 1st round itself.what level of DSA can I expect from companies for data engineer position?

like

Nagarro OR Deloitte which firm is good for interesting big data products? And which firm has better work life balance?

like
like

What do you think are the new things to learn in data engineering in Azure platform

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal