Hi, I want to learn how to build ETL pipelines with Pyspark and Azure. As right now i have experience in feature engineering only with SQL and Pyspark.

Can anyone suggest anything online from where i can learn and do hands on?@

Posting as :
works at
You are currently posting as works at

Follow these preparation strategies below.
1.) Sql
2) Data modelling and data warehousing
3) DBMS concepts
4) problem solving skills , DSA is suggested (Python or Java or R )
5) distributed framework (Hadoop ,mapreduce, spark )
6) data warehousing tools or analysing Apache tools like hive, sqoop, hbase, impala, Cassandra
7) cloud technology (AWS or azure or gcp )
8) system design knowledge and their different theorem like cap, paclec and types of scalling of distributed system

Recommendation:
1 . Learn Hadoop architecture from https://youtube.com/playlist?list=PLdW3asq7z3KlWy24tinxPDM_ewIXbt0I2

Understand the spark architecture. Ask 2 question in every topic why they have designed framework like this what if it doesn't designed such a way . Relate with map reduce because spark replaced MR .

Start coding on leetcode or codechef or any coding platform .

Thanks.

like

From where to learn system design for data engineering

Related Posts

Hi fishes,
I am currently working as Branch Manager (scale-2 officer) in one of the leading public sector banks and have 5 yr experience in core banking. I am having 2.5 years experience in the IT sector on vision plus as a software engineer (before joining the bank). I have knowledge of core banking as well as good knowledge of banking tools like finacle, caps, imac etc. I want to switch to private financial institutions.
Any suggestions about such institutions and the domain for I can apply

I have started my career in backend web development. (NodeJS)
It's been one year.
So far I like what I'm doing.
However I hear so many stories of how people in cloud , ML or data science earn huge packages.
So, to survive or earn well, should I consider switching to any of these ?
If possible could you help me with what to learn and any resources for that ?

like

Do you have a mentor? I’m in the autumn of my career and I’ve had a  modicum of success in production and it’s time to give back.

like

I am currently a care coordinator with one of the big insurance companies. I want off the phone, I have been applying for Rn jobs in utilization but have not gotten the last 5-6-7 I applied for, I lost count… I am thinking more analytics and business side of things now. I finished my MSN LM to maybe stand out from the large amount of applicants they get. BUT I have no experience in analytics.. so I have a chance? How do I get training in tableau, sql, interqual, project management, alteryx..

like

Hi, I'm getting lot of interview calls for SAP SD Consultant but they want someone experienced 2 to 3 years of experience, 1 or 2 implementation done, support or rollout etc, Relevant experience of sap is zero I have, for BFSI industry is 3+ years I've, so what would u suggest??I'm not getting any work, 5 months gap already now in doing SAP course, hana certifications etc,any fresher in SAP, how u made it in SAP? I'll start working anywhere now, in SAP I'm not getting anything, tired or have i

like

Anyone think of quitting consulting and going pro? There is a huge pilot shortage right now and it’s probably the best time ever to get into it. But the money....ugh.

like

How does everyone feel about the online SHRM learning for SHRM-CP and SHRM-SCP for Certification?
Do you skip around? Do you follow the path? Where does one start?

like

Looking for your career advice! I currently work as a Content Marketing Manager but would love to move into a Head of Marketing/Marketing Director role. What would be your best tips to make that move? I assume it will take time of course, but I'm wondering what/where I should invest in terms of my training and next career steps. Thanks!

like

What additional skills can I get certified in to make me more marketable as an accountant? Other than CPA

like

Feeling lowballed received an offer to PNC product management development program. Excited to be making a break into product but it’s paying 20% less than what I’m currently earning.

like

Thinking about transitioning from advertising (10 years as a copywriter) into public relations. Has anyone in this bowl done that? What types of roles should I be looking for?

Hey! I’m from Spain, and I’m thinking about find a job overseas, maybe in central america or southeast asia… I have experience in finance consulting… do you have any recommendation, contact or tip that could help me?

I don’t know if try to find a job first and move then, or move out first and find a job from there.

Thanks!

like

Asking on behalf of a friend:
How difficult is it to get an analyst job in current job market (no relevant prior experience) bcom grad

like

Hello Are they any other HR licenses that are as valuable as the SHRM and HRCI ?

like

New fish looking for career guidance.

I'm a corporate recruiter with 3 years of experience. Finishing my degree in HRM 5/2022. I'm thinking of getting my SHRM -CP shortly after graduation.

I'm not sure of where to go after that, but I doubt I will stick with recruiting or my current firm.

Has anyone here started off in recruiting and branched out to other HR functions? Do you like your new role? Do you wish you stuck with recruiting?

like

Hi Fishes..I recently joined HCL.. and I am into a Support profile. So i would be having rotational shifts here and also there won't be much of learning. Is there any career upskill program in HCL through which I can move myself to a different domain or technology within HCL? If anyone has more information than we can Talk on teams as well..

like

I joined on Nov 2020 in EY GDS as Senior 1,so when I will be getting progression for senior 2?

like

First time poster here!

After spending 4 years as a mid-level ops manager, I tried to break into HR through talent acquisition. Now after 6 years in TA, I've had a hard time getting out of it. My goal was always to get into a more broad HRBP position, but I've since worked for smaller orgs without a solid HR career path. When I ask about my career goals, they keep telling me, "Soon!" Or "Maybe in 6 months!"

I'm testing SPHR in June. Any advice for securing the type of role I'm aiming for?

like

I am looking to make a career change to HR. I currently make 75K as a sales and development coach. I had a recent offer for a state government position in a talent acquisition role but I’d be taking a $25K pay cut. I am extremely over qualified for the position but it could possibly be a stepping stone and gives me the time to acquire SHRM certifications and possibly start taking classes to get my MBA. My question is those currently in an HR role, would this move make sense to you?

like

Is PE Ops a good starting point if you want to learn more about entrepreneurship and prepare for your own start up?

like

More Posts

likehelpful

If anyone in the group works for Fannie Mae…do you have any tips for the interview process? I have an interview coming up.

I’m also unsure of what to say for a salary range, I don’t want to cut myself short. It is a risk management associate position. I have 5 YOE in mortgage ops with transferable skills

like

Hi, I have an offer with BA continuum as Team Lead for Mumbai location, can I ask HR for change of location to Gurugram?

Thoughts on Razorfish Health?

like

Looing to Buy used in NYC. Any suggestions of where to go?

This is second time I got offer from CGI but now I have again a better opportunity. So if this time I reject will there be any problem for me in future?

Is the salary structure different for people working in "Infosys consulting" and regular Infosys.

What are the differences between these two entities.

like

Does interviewer nowadays still write early feedback prior to sharing the full feedback with recruiters?

like
like

What has been difficult for you in the endeavor to complete half your deen?

like

What is average salary of DevOps MTS( memeber of technical staff) for 2+ yrs experience in SalesForce?

like

Looking for perspective: Did money play a role in evaluating whether someone could be your life partner?

like

After the Cerner and oracle acquisition, do we have any chance of getting any raise and promotion while getting a new offer letter from oracle?

like

Hi Everyone,
Can anyone please help me with referal
Eoy:4.7years
Tech stack : react js and javascript
Will available from July 13th2022

Any NYC fish?
Please DM , I am new here (less than a week) and would like to hang out and make some friends in the community.

like

Any female fish in the nyc area like dating guys 35+?

like

Pune office lo evaraina flatmate kosam chustunnara ?
M here , dm me if we can take together, i am searching this weeken

like

I think people who “kiss” their dogs/pet on their mouths are disgusting

likefunnysmart

Me: Today will be a great day
My anxiety:

Post Photo
likefunny

Has anyone got retained by Mphasis due to external offer? If yes, please tell me when did you get the revised offer letter(after or before revoking resignation) and how much time it took? And did they fullfill your promise?

Additional Posts in Data Engineering

Require referral for Data Engineer position in reputed companies in Pune.

Yoe - 1.2yr

TIA.

like

Anyone here works in Dunnhumby or Gartner?

like

Can anyone suggest a good course/Set of courses to become Azure/GCP data engineer

Currently I am a Microsoft SQL DBA, 8 Years exp looking for suggestions

like

Tiger Analytics - Senior Software Engineer(Data Engineering), [WFH]

Fractal Analytics - Azure Data Engineer, [WFO]

Airtel Africa - Big Data Developer, [Currently WFH]

Which one to choose, the compensation is more or less the same.

Looking for a learning environment.

YOE - 2

like

Hi,
I am working as Data Analyst. Now, I seen my interest towards Data Engineering.

Please help me to decide, Should I go with
Azure Data Engineer
Or
with Big Data Technology like Spark, Kafka, Hadoop, Sqoop, Airflow.

Suggest me some good course to start.

Thanks 😊

like

Why Data Modeler role is on high demand these days?

Hi people,Does anyone have given visa hackerrank interview recently? May I know the kind of questions being asked and their level?

like

What are some good product based companies to work as Data Engineer?
Good in terms of learning, interesting work and good work culture.

like

Hi Fishes,

I've 2.5 years of experience in Data Engineering and technology stack that I worked in is PySpark, Spark SQL , SQL Server, Azure Data Factory and Azure Databricks.
I'm currently serving notice period and looking for opportunities. Please let me know if anyone's hiring.

Thanks.

like

I have 2 offers one from my Current company and one from Cisco my current company is giving 10 percent more than cisco Where should I go should I join cisco or stay in my current company( a small stablished startup )Considering my final goal as FAANG (similar) and I come from a Small College Cisco Amazon Uber Walmart

like

Guys pls reply…
How is Impetus Leaplogic project for a data engineer ? In terms of technology used and WLB?

like

Hi folks, i am looking for a job change i have 2.6 yoe in data engineering with tech stack - hive, hadoop, pyspark, sql, python , aws. These days i am not getting much calls. I worked for x company for 2 yrs and switched to y company in feb 2022. But recently i faced a situation. I cleared all rounds and send my docs to sunlife financial but now when i am trying to connect with them they are saying your candidature is on hold. I alao resigned from my current company. What should i do?

like

Please suggest

YOE 2.4
Azure Data engineer

Fractal 14.5
Roch product corporation 18.5

like

How is Tavant technologies for Data Engineer?

like

Hi all,
Can someone guide me on how should a person start learning for transition into a data engineering role at a beginner level. Currently I have known Python and SQL in my job. Not able to figure how to go forward?
Any advice would be great!!

like

Hello Fishes, I needed a suggestion on below:
I have been working on AWS cloud services from ~2 years. Recently joined new org. where the work is on Azure Cloud and more into Azure Databricks, Synapse Analytics, CosmosDb etc.
I'm not exactly sure at this point If this is a good move to switch technologies.

Can you fishes pls guide or help me if this Azure Cloud related services stated above are good in terms of current tech stack in market demand or did i made a wrong move? Pls help.. 🙏

like

Hi guys, i have been working as a Data Engineer from past 7 years. I have mostly worked on MSBI stack and also have fair experience in PBI. Now that most of the companies are moving towards cloud based technologies, could you help me with the right pathway going forward for better opportunities.

like
like

Hey fishes,

My current salary is 13lpa in SQL, Plsql domain. I just interviewed with mindtree and they are offering 16lpa in big data . Should I switch or negotiate for more. Yoe-4 and new to big data.

Thanks

like

Hi everyone, I am 4.9 years experienced , I joined Fractal 3 months ago as a data engineer, but still haven't got any project. Process is so slow that I have received only 4 to 5 project calls. I switched in hopes that will learn the new tech and tools. But now i am starting to forget even the one i knew. Should i go for change or wait a little more time as fractal is reputed in AI/ML? I have interview lined up with Impetus as for now. Would it be better choice?

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal