New to data engineering. What are the most important and in demand skills to learn?

like
Posting as :
works at
You are currently posting as works at

Sql and python
Pick a cloud platform to learn the pipeline and orchestration tool

like

Try apache beam

DE is pretty broad. I’d say a good place to start is learning python, pandas and pyspark. Airflow is increasingly popular for building data pipelines. Familiarize yourself with SQL and NoSQL (Mongo). If you’re interested in ML, scikit-learn has the most easy to understand and comprehensive learning material.

like

SQL. Everything else is role/firm specific.

Get comfortable with ETL. Start with the concept itself and the million variations that the concept entails.

A lot of ETL is done thru SQL. So the next concept to master after SQL is ETL.

Related Posts

USI Producer Job, how is it? Currently an Account Manager and looking at this opportunity. Also, really any advice from service to a sales transition is much appreciated!

like

Good evening Everyone, I have been a PSR for 10 years with the same company, but worked for 5 different practices. Was looking to move up in our company but from what I’m understanding, we have to have a degree in management. How would I go about trying to move up in and out of my position without a degree?

like

Which one should I go for?

1. Oracle - 18.3 LPA fixed + probable bonus (I may or may not get)
2. Amadeus - 18 fixed + 1.5 variable.

Amadeus is also giving joining bonus of 75k. Amadeus is paying more, may give better hikes too, but I am finding it hard to let go off Oracle's brand name on my resume even if it's means around 2L less over an year. I am planning to work for atleast 3 years in my next company.

What would you have done in my place? Should I go for Oracle or Amadeus?

like

Associate at McK; made ~$220K cash TC (maybe 260k or so in 22).

Curious what roles in FAANG will allow me to maintain comp and work chill hours?

I have a STEM undergrad and 5 years of non consulting work experience. Very good at numbers, strong communication skills, decent at coding (but I don’t want to be a SWE or DS).

like

Hi, I am currently working at Bisleri in a quality control role with 6 YOE. My current CTC is very low (4 LPA) and I have not found any motivation or growth in my role since COVID. I'd like to be in the design industry (Graphic Design or UI UI Design). Currently I am freelancing in my spare time. My question to Design professionals is: how feasible will the switch be? Will I be accepted as an intern/junior designer in my early 30s?

like

What are some high paying career paths that an auditor can pursue? Slaved for 3 years and not even 4k, while every single one of my peers in IT are at minimum 5k with good wlb...... accountants dont seem to get paid remotely well here

like

Hi.. I'm currently working as Data Analyst in TVS. Most of my work in Advanced Excel only. How can I move forward myself into Data Science by learning new technologies or any courses? Please guide me

like

What is the moment that made you realize you wanted a career in entertainment?

like

Has anybody had success pivoting off the bench into a non-managerial role?

like

Currently a paralegal at a small start up financial services company been there for just a little over a year. I wear “multiple hats” but more recently I’ve done a lot of PM work and some BA work. My position title is changing soon but I’m wondering how helpful that would be for wanting to go into PM work at a tech company. Anyone have any career pivot advice? (Also I’m only 1.5yrs old in my career so I’m definitely still new, and I have previous experience as a project assistant).

like

Why did we choose PA instead of a rotational program again?

like

Graduated with a degree in managerial science and minor in marketing and a sales certificate from a very prestigious business program in GA. Have had 4 different management positions in the past. Looking for opportunities in sales that allow me to travel constantly. Preferably base + commission but open for anything with traveling (within/out the country) anyone can help me network? Thank you!

like

Going from 165 as an ACD to 210 as a CD? Is that about right?

like

What’s the best way to break into life sciences VC/PE from a LS consulting background?

like

Hello fishes,

My wife have 2 years of experience working in back officea and she is persuing career in SAP Basis, but we both are not sure about market, can someone give a pointer on what should be the best thing to get job and with good salary along with wlb?

like

What is it about D&A that most clients (or even partners) don't appreciate? Thinking about the painpoints or opportunities that are unseen at the top or require lateral thinking

like

Are all the Oracle certifications free irrespective of our team after we join Oracle? Or WT can do certification only of our manners assigns any to us?

like

After you are stealth laid off, other than job search, what do you do? I assume they won’t give you work and even if they do, you can politely decline? How do you feel about you are not progressing anymore and seeing other people progressing in their career? I have no intention to make partner, but still feel a bit stuck not doing that much.

like

Hi Guys,

I have offers from Amadeus Labs, and Powerschool similar roles, same package. Any recommendations based on work culture, growth, salary hike?
Thanks in advance.

like

Has anyone made a move from audit to consulting as a senior and could you speak a bit about the process??

like

More Posts

How many billable hours do people in prosecution tend to average each month? I’ve gone through waves of feeling like my billable hours requirement is reasonable to feeling like I’m drowning and can’t possibly do it for that long.

like

There may come the point where constant meetings, being around the same people, or the same environment will cloud your creativity. When this happens, you may need time to yourself in a different environment to think clearly without any disturbance. 

Great ideas can wake you up in the middle of the night, during a shower, while walking in the park, or any other setting outside of a meeting.

IG: Burnoutcoachcarlo

Post Photo
like

Anyone familiar with Ephesoft Transact that would let me pick their brain?

like

Anyone here given Wissen first round java interview? Suggest some tips and questions. LOOKS LIKE TOUGH ASSESSMENT ON VIDEO CALL. YEO 3+

Anyone using Edward Jones? Feel like we did a mistake and rolled over all my husbands retirement accounts from different employers and our kids 529 under them Jan 2020. Since then market is down and accounts lost20%. But the fees every month is pretty high around $200. I am worried to move it to fidelity or some other place and manage myself(not very confident), then I have to sell all of the Edward Jones managed funds and buy new. sell low buy low makes sense but don’t want to loose anymore.

likefunny

Bro Its been 3 years I j still don't have a 5star Island I need help😭

like

I’m referencing a cell (numeric) to a text cell. Is there a way to format that number to include commas? See example in comments.

Relocating to Chi. My wife has deep experience as creative recruiter, creative dept manager, digital producer and PM. Any leads for this fiercely talented pro and lovely human being?

like

Can Anyone in Cerner Corporation refer me for this role Asset manager. I'm an ex employee.

I can’t find Binance on the App Store! There is one app but it appears to be only for price change notifications not buying or selling?

How long did it take for you to provide your conflicts to the new firm for a lateral move?

like

Dear all,
May I know what is the variable pay cycle in barclays ? Iam joining this November..pls advise

like

Does JPMC Chicago (10 S Dearborn) have any monthly parking employee discounts? Tyty

like

Just need 14 points from Olave and Evans …. Not looking good. 😔

like

Sorry, not sorry for helping keep clients and agencies from being sued. (Also a bit lazy to use a meme rather than come up with an original idea IMO)

LinkedIn post:

Post Photo
like

I am starting to question my existence in this world. I have a nice job, decent salary, i work out and eat healthy, maintain my health, read 3 to 4 books every month, dont have any personal commitments (single), in my early 30s and things cant be any better. Of course with the exception that i had a real bad break-up with my fiancé 6 months ago. And i dont have any symptoms of depression, but there are days when i just feel like what is all this for?

likehelpful

I thought the power reserve on DJ41 Ref126333 was about 70hours, but I noticed it’s about 40-48 hours on mine. Wear mine about every other day, but not on weekends and then Monday morning when I go to wear it, it’s dead and have to wind it. Anyone else seeing this?

like

How did you decide where to live? Have a young family, working remote, and currently living in a city. Looking to move to the suburbs but a bit overwhelmed with the number of options. Also struggling since we don’t live close to family and don’t like the areas family members currently live in.

like

Accenture strategy folks.. can someone confirm if work from home is about to end?

Phantom of the opera on Broadway - is side orchestra worth about 30-40 more than rear mezzanine? What is better experience for this show?

like

Additional Posts in Data & Analytics Consultants

Is there a good online course on storytelling with data? Something that can help structure your thought process and design choices when creating visualizations and Dashboards in Tableau, Power BI etc.

like

What do you think when associates/senior associates misspeak on a concept during a meeting or interview?

I’ve learned something new on each project, and I’m afraid of coming off as an idiot so I rarely contribute to conversations; only after I have done research on the topic and have prepared do I speak/give ideas

like

today I choose violence

Post Photo
funnylikesmartuplifting

What is it about D&A that most clients (or even partners) don't appreciate? Thinking about the painpoints or opportunities that are unseen at the top or require lateral thinking

like

Anyone who has done masters in business analytics Singapore? Could you help on courses, Scholarships and careers and jobs. Looking for help.

Hello, Anyone here from Deloitte Omnia AI who could refer me for an open opportunity.

like

Currently in consulting where I get to do data visualizations sometimes but not always, have an opportunity to shift to a data analytics focused role. Struggling to decide as I am not fully sure if I would be competent in/happy with daily data work. Advice, thoughts? 😣 I just never find what I like…

like

Folks applying for Data and Analytics roles in the industry at M, SM, D Levels, what technical/functional topics do you prepare for on the Analytics side?

like

Hi! Does anyone know what kind of knowledge is needed to crack the Data Scientist interview at Google?

I'm currently working in Marketing Analytics (primarily using SQL and a bit of pyspark). Apart from mastering SQL, I wanted to know from people working at Google what does it take to get there as a Data Scientist? (I'm very interested in Strategy Ops)

I plan to another 6-8 months in preparation and upskilling myself before I even apply.
Any help and guidance is appreciated! Thanks :)

like

Got Amazon career essentials (ACE) assessment to complete for Business Analyst role
To anyone who has experience with it- What kind of work simulation questions should I be expecting?
Is there a way I can prepare for it or any resources one must know of?

like
like

Mεntal mαth in iηtervieωs. Thoughts, tips, rants, grievances?

like

1) Hypothetically if you were looking to sabotage a colleague, can you code an excel document to throw off their numbers or calculations, (if you send them that document and they download it for use)?

like

Any cyber security folks here? or those interested in systems engineering? Potential role with TC 120-180, awesome team and wlb, at least compared to consulting.

like
like

What is the best way to handle group financial consolidation in a multinational company?

like

MBB data science expert role (i.e., mostly identical progression as generalists but less up or out pressure) or third (?) tier tech companies (DataRobot, C3, etc.)?

Assume starting TC is same at both or 10% more at the tech companies. That's the figure they're roughly giving me during interviews. Also coming from Deloitte.

like

I was hired a few months ago by D to do data science. I am not doing data science currently, is there a way I can switch roles internally? Should I talk to my coach? Or should I just start looking for a new job. I’m starting to feel that the field is so hot rn my small stint at D won’t be a big deal to other firms.

like

Is 128k a fair base salary for Consultant in Deloitte Analytics & Cognitive. MS CS and about 3 yrs experience. In one of the fastest growing but midtier tech cities in the US.

like

I am currently working mostly in Data Analytics, but I want to transition into Data Engineering. I've applied to a couple Data engineer positions but it seems I don't have the qualifications for them. Should I maybe look for Jr. Data Engineer positions to get experience and try to move up from there? If so anybody know of any Jr. Data Engineer positions available?

like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal