Hi hi. New to data analyst role at my company. We have extremely large datasets with many many columns, each with many many values, and there is very little documentation. When you start at a new place, what is your approach to learning the datasets so that you can use them effectively??

like
Posting as :
works at
You are currently posting as works at

Read whatever documentation is available, and prior reporting / files for context clues. Then set up meetings w senior analysts / form good relationships w them so they can assist in any future endeavors as you’ll need their insights.

likehelpful

This happens more than it should. If you don’t have a dictionary and have some free time/no deliverables or tight deadlines is a good idea to start making your own. With the lack of documentation you’re going to have to ask coworkers and other people on the project if they know what the Columns mean. I assume these databases are a combination of many different ones so it will help to know the source of the information when figuring out columns. 

You can also run a basic function depending on what program you’re using to at the very least see the column types. 

When figuring out column it also helps to run descriptive statistics. This can help you figure out what the columns mean if you’re stumped. 

like

EDA (exploratory data analysis), start with the totals and then slice and dice them and always make sure they add up to the previous total. Make some charts, look at trends and distributions. Then ask questions about the insights and if they meet expectations. Don’t wait or look for documentation... it never comes and doesn’t exist

like

I’d ask other teams what reports they run , what fields they use or what the most relevant information is. Also ask someone to help you map company lingo to what it’s actually called in the data. When I first joined BCG I wasn’t totally aware of the difference in all the titles but was extremely important when viewing data.

like

Very little documentation is a bit hard to work with!
1. You should ask for a data dictionary and if possible an entity relationship diagram (I.e how are the different tables linked).
2. Schedule a data discovery session with your tech team or data owner to get an overview of main data sources and datasets if there are limited technical artefacts.
3. Play around with data by creating summary stats, profiles and distributions to link it with business understanding (how many customers, what’s their demographics and regional split, how many low/medium/high value, what product or plan they are on etc). Reconcile it with existing reports or dashboards so that you are sure you’re looking at the fields correctly and synthesising the right variables.
4. Develop a deeper understanding by linking various dimensions to create customer personas and profiles - that way you’ll learn how to effectively use data to also understand business and subsequently identify opportunities and issues.
Hope it helps.

likehelpful

First of all, thanks for asking this question. I’m still relatively new to analytics and this has happened to me on each new job.
In addition to all mentioned above - one of the most helpful things for me was to get useful bits of queries/scripts from colleagues just to understand what tables/columns they were using, and especially the joins. Once I had a solid list of “most used tables” I got to exploring those.

like

One common thing missing in the comments here is to understand the end goal and problem you are trying to solve using data.

It’s impractical and very very time consuming, almost every time, to familiarize yourself with most, let alone all, tables in a firm. So, a better way would be to move backwards from the end goal and ideate on what kind of features might help. Subsequently, look for relevant datasets and focus on those.

As you talk to your team members, you would probably expand on the feature space and datasets you are looking for.

Do some ground work as others have recommended. Several times names of features are obvious. Perform some validity checks on the nature of entries, summarize them and validate the intuition with experienced folks on the team.

like

1. Check any existing reports. You can use it as reference.
2. Discuss with senior team members how they have used it before.
3. Do some preliminary analysis and share the analysis with people and gather feedbacks

like

If you are the Analyst, there is an Engineer. BCG 2 has the most concise method to follow. Go find the Data Engineer/Architect in charge of the database and ask them.

Also, often on these database integrations, there is an integration manager they will have a very good business understanding of what the data is and why it is being brought together - if they are still around it's a good idea to reach out to them.

like

Consider building out a data dictionary as you are fact finding. It may evoke stronger inputs from stakeholders while also adding long-term value.

like

A lot of fantastic suggestions listed above. A few more off the top of my head, to do as you make your dictionary:

- check for legacy processes; some columns may be blank all together, or become uniform after a specific date (or other criteria)
- check the count of distinct values and distributions in each column; when there are few unique values, examine the top few, and when there are many, use count tables or histograms for a quick view of the distribution
- use cross-tables on column names that seem related to either other column names or values you've seen before
- make a correlation matrix for continuous variables, look for relationships that stick out
- graphing variables against each other can be helpful, but not when there are too many. Save this kind of plotting only for when it seems appropriate to the pair or as a last resort for variables you want to "squeeze" for info--and have some boiler plate plotting code on hand when you do

like

Wow a lot of amazing feedback here!! Thank you guys for taking the time to provide me with very detailed and thorough advice here. I very much appreciate it and know how to start now :) you all just increased my business value!!

Related Posts

Hey guys
I'm being offered a new grad product development engineer 2 position in AMD Canada
Can someone help me out with the salary range for this role?
Thanks!

like

Please mail your CV to ammar.azizi1@gmail.com along with the Job ID from the EY careers website ( https://careers.ey.com/ ) for a referral. Please only send your CV if you have not applied directly or already asked someone else to refer already.

Please also DM me here because sometimes the CV goes into my spam folder. Cheers!

like

What's the market salary range for an Sr. Analyst role whose working as a project management responsibilities for India and Global team?

like

Hi,
I am ITGC SOX auditor working with EY as consultant under Risk advisory. I have done my MBA from tier 2 institute.
Total yoe: 3.8 years
Can anyone please refer for ITGC auditor role in deloitte usi.
Any leads would be appreciated.
Thanks in advance

like

Hey folks, I just want to know how to identify by seeing an offer letter of TCS, is it a C2H or Parament position ?

like

Hey guys,

I have an interview at Microsoft next week for the role of SharePoint consultant. What should my preparation be like?

Received an offer for 65k for an entry level business valuations position in La. Is this a fair offer or should I negotiate?

like

Got an offer of 8LPA from JPMC for the role of Test Analyst. YOE - 1.4 years, Location - Malad, Mumbai.
I would like to know how much I can negotiate with the HR on this offer.
I don't have any other offer in hand.

like

AMERICAN EXPRESS India is hiring for below role:
1. Business Analyst/ Assistant Manager – Enterprise Data Strategy, Credit & Fraud Risk (Req ID -21030910)
2. Manager / Senior Manager : Product Development, Risk Products & Data Strategy (CFR) (Req ID -21030284 / 21030287)
3. Business Analyst / Assistant Manager – Digital Analytics
(Req ID - 21029077)

JD screenshot is attached.
Kindly DM me job title, req id and your email id for referral

Post Photo
like

Struggling with imposter syndrome here. Over the past five years at my employer I went from independent contributor as a developer to a more of a Leadership position. what I’m finding though is that I don’t get nearly the kudos or mentions that I used to as an IC. I often wonder if I’m doing a good job or if I’m failing I tend to overthink and read into peoples life interactions. is this common as you enter leadership? Any tips to maintain confidence? I really hope I’m doing a good job

like

Anyone heard of ECS consulting? Are they any good as an exit op?

Hi All,

I have recently received an offer letter from Finicity - MasterCard company. My job role is the Senior technician support role with Level 8.

I am holding an offer of 12.2LPA having 4.4 yrs of experience in Citrix Administration.
MasterCard is offering me 13.5LPA as fixed pay and 7% as variable pay.

Not ready to negotiate more, should I accept the offer?

like

I’ve reached a point in my career where I want to take a step back and do an easy job just for the medical insurance benefits. Any ideas what I could do (or who is hiring)? Would contracts manager/admin/specialist be an easy job?

like

Is $180k OTE competitive compensation for a Head of Product Design in a startup? Remote role. East Coast US hours. Managing a team (hands on too) so likely not a checkbox 40h work week.

like

Hi all
I started out as a graphic designer. Worked for about 4yrs in different roles. Then i started a digital design role for 2 yrs and then made my transition into ux. i have been in a ux role for 5 years. But i want to steer my career towards something more creative. Like in 5-10 years I wouldn’t want to be a specialist or a ux director. I d like to be an art director or creative director. I love branding & marketing but also product. Would love something that allows me to do both?

like

Hey Guys,

Is there any openings for fresher with ServiceNow CSA certified?

like

Referral Available for following
Comment or DM for details

Post Photo
like

What's the maximum fixed salary that Verizon (India) can offer for "Sr Engr Cslt-App Dev" role, kindly suggest as HR is saying it will be difficult to get approval for 26LPA fixed ?

like

Anyone know of jobs in writing insights and recapping trends based on research? Especially the implications for different sectors? I think I’d love that role, and want to learn more about day to day

like

More Posts

Rajma is bae 😍

Post Photo
likefunny

Best place for holiday office parties?

like

Recommendations on slim fit denim shirts. Not for the office of course but I feel like it’s a staple I need in my wardrobe.

Any senior writers looking in NY? Send me your books.

like

@
I'm trying to find good examples of design systems with gamification. Does anyone have some recommendations? Thank you! 🌞

like

What’s happening to these new apartment rates 😩😩😩

like

Hello US fish!
What advice would you give a London-based writer/CD/ECD looking to work (remotely) for US-based clients? I’m relocating to Costa Rica for a few months next year so trying to make the time difference work. I don’t mind what sectors but, ideally, short-term writing gigs. Any tips on approaching clients/agencies? Which recruiters are worth reaching out to or are they all in league with Satan? Should I just give up now and get a job counting turtles? Thx

like

Please mail your CV to ammar.azizi1@gmail.com along with the Job ID from the EY careers website ( https://careers.ey.com/ ) for a referral. Please only send your CV if you have not applied directly or already asked someone else to refer already.

Please also DM me here because sometimes the CV goes into my spam folder. Cheers!

like

Hi Everyone.

Is post graduation (CSE Pg Couse/MBA) required to grow a person's career as SAP Consultant?

I am confused.
Pls explain with real time examples if any.

Thanks in advance.

like

What's the market salary range for an Sr. Analyst role whose working as a project management responsibilities for India and Global team?

like

Ugh. So much work. 😩

uplifting

When will I receive F%F Settlement . My LWD is Sept 3rd. I already raised a request .

like

Hi,
I am ITGC SOX auditor working with EY as consultant under Risk advisory. I have done my MBA from tier 2 institute.
Total yoe: 3.8 years
Can anyone please refer for ITGC auditor role in deloitte usi.
Any leads would be appreciated.
Thanks in advance

like

How do I become an MBB employee? Is it possible to gain experience at big 4+ACN and then get referred in??

like

Hey folks, I just want to know how to identify by seeing an offer letter of TCS, is it a C2H or Parament position ?

like

WPF merging with Alight Solutions!

like

Has anyone been admitted to Massachusetts by transfer of UBE score? I submitted my application in July and took and passed the state law exam today. How long after this were you admitted? Thanks!

like

I feel lucky my family didn't catch covid19 from me- there is no way to really protect them

like

Which firms are strong in tech consulting and have plenty of West-coast clients?

I’m based in LA and for the last 5 years almost all of my projects have been east coast.

like
like

Additional Posts in Data & Analytics Consultants

Hi! Does anyone know what kind of knowledge is needed to crack the Data Scientist interview at Google?

I'm currently working in Marketing Analytics (primarily using SQL and a bit of pyspark). Apart from mastering SQL, I wanted to know from people working at Google what does it take to get there as a Data Scientist? (I'm very interested in Strategy Ops)

I plan to another 6-8 months in preparation and upskilling myself before I even apply.
Any help and guidance is appreciated! Thanks :)

like

What is Accenture’s data scientist interview process like?

like

Anyone here work or has interviewed for a data science role at DataRobot? Would love to get some insight. Third interview coming up.

like

Mεntal mαth in iηtervieωs. Thoughts, tips, rants, grievances?

like

Hello, Anyone here from Deloitte Omnia AI who could refer me for an open opportunity.

like

I was hired a few months ago by D to do data science. I am not doing data science currently, is there a way I can switch roles internally? Should I talk to my coach? Or should I just start looking for a new job. I’m starting to feel that the field is so hot rn my small stint at D won’t be a big deal to other firms.

like

How would you recommend to find a data analytics job in the US? Currently a Sen Con at a big 4 in Aus

like

McKinsey & Company Does somebody have experience regarding
compensation differences between
QuantumBlack & McKinsey? I am in the process at
QB and disappointed with the salary that's
proposed. (New grad Data science from top
university applying for London) McKinsey & Company QuantumBlack

like

Can anyone from Blackstone share some insight about the Data Science Management Program? Looks like a fantastic opportunity!

like

Anyone have a ballpark range on dataiku price per seat or however they price? It looks like an interesting product.

like

Does your company split data science roles into different practices like data & analytics and software engineering? Does this inhibit collaboration & Where does a data engineer fall?

like

In the most unnecessarily complicated phrasing worthy a dissertation, what are you working on this week?

like

Does anyone work at EY in their Decision Modeling team? Not asking for answers, but just wondering if someone is able to share what concepts I should be familiar with when prepping for the case study (i.e. do I need to know how to run an LBO or is a DCF sufficient)?



like

New data offering: name.co, or namesoftware.com for the main domain? I’m leaning toward name.co, but I worry about not having the .com.

like

What’s so good about the Georgia Tech OMSC and OMSA? Is it just the cost? Seems to be recommended by most of reddit

likehelpful

Folks applying for Data and Analytics roles in the industry at M, SM, D Levels, what technical/functional topics do you prepare for on the Analytics side?

like

Has anyone created live dashboards via Data Studio instead of Tableau? Thoughts?

Is 128k a fair base salary for Consultant in Deloitte Analytics & Cognitive. MS CS and about 3 yrs experience. In one of the fastest growing but midtier tech cities in the US.

like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal