Help me save the day: have a massive excel file (600k rows) with duplicate records (column a, b and c are duplicate, not a duplicate if a and b are the same though). How can I filter for just these records? If it didn’t crash excel I could concat into one column, Conditional format for duplicates, and sort by color. What’s a work around?

likehelpful
Posting as :
works at
You are currently posting as works at

Do the work in Power Query

likehelpful

Used power query. Worked perfectly once i got the hang of it! Will definitely be using this in the future. You saved the day, thank you!

like

You can add a column with an if then formula that says “if the cell in this column of this row is equal to the cell above (before) it, give me an asterisk, otherwise blank. Copy that all the way down to the bottom row, Then use [cut/paste special/paste values] for the whole column. You can now use that column to sort and/or filter out the duplicate rows by copying it into another column and changing the cell addresses being compared.

You can use this method to compare multiple columns in the same row to to ensure it’s truly a duplicate. For example if you want to find and mark only rows where 3 different cells match those immediately above them, you can do 3 different columns with this formula and apply filters, or even better just use the concatenation function to create a new column consisting of all 3 cells combined into one text string and then copy/paste special/values to make it sortable/filterable.

HMU if you want to dive deeper lol

like

Use R

like

Alteryx. I had 0 knowledge of how to use it and built a workflow that did exactly what you described in 2 days. I googled “alteryx equivalent of X in excel” and it wasn’t too complicated after understanding basic Alteryx UI

like

Why won’t “remove duplicates” with all three fields checked worked? Under the Data tab. May churn for a min or two but should work.

like

You can probably use conditional formatting to highlight those rows in a color than filter by color to see only those rows.

like

Do it on a more powerful computer

like

If it’s not super sensitive data, google sheets might be your friend here.

Can you use remove duplicates? Alt A C

Need a bit of logic to determine which is the correct record. Hence why I want to isolate them first.

Can you just add an if / and formula (if a=b and b=c) to a new column and filter on that?

Concatenate them and do a countif to see which rows appear more than once.

If the sheet crashes paste values of the concatenation before the countif.

Use Alteryx or Access honestly

Locally run access databases make me cry

like

Another vote for Alteryx. Or Knime.

I have run 100k data in Power Query and it has been fine for me. I don’t know if it’s possible but maybe you include massive amount of null rows? Maybe filter out null rows first then try again. Tableau Prep allow you to use for free for 14 days. It works great and really easy to get familiar with.

Echo alteryx. It is honestly ridiculous how capable alteryx is with massive datasets. They even have a handful of tools designed to work with datasets 100m records and up. This would be incredibly easy in alteryx (couple tools)

Related Posts

Someone has joined the DARK side 😀

Post Photo
like

Taking FAR soon! Currently reviewing the book and wanted to see if y’all had any advice on what modules/sections to cover more then others? Been hearing Gov/NFP is a big one. Anyone care to share what the exam was like for them and what it mainly covered?

like

Is it possible for a vet tech to make 6 figures?

like

Where is GME going today?

like

Looking to break into investment banking M&A as a lateral analyst. Been doing Valuations for almost a year (graduated spring 2018 from ug). I've practiced modeling and finance questions. Cont...

like

22- Asked to upload document in ibegin
23- Verified all documents
What next ?
Current status as show below on ibegin

Post Photo
like

Tcs Hr has said to me that you will receive welcome mail by today EOD. What time frame I can expect the mail.

likefunny

I'm a senior and started out over a year ago. What would be some good low risk investment options? Have about $15k to invest. Open to investing in India or in the US too. (No crypto pls)

like

Hi ,

Can someone refer me in Citibank for mainframe developer

Only sentence that matters... not a single Republican objecting to the election results is objecting to the victory they won on exactly the same day on exactly the same ballots.

likesmart

Did the NYSBA forget about re-takers? Is this a grave travesty and denial of Art 15 rights to those who are not being allowed to sit for the exam? What about their financial distress and their ability to retain and seek employment?

Post Photo
like

I handleda very difficult and convoluted succession for a black family. My client was astute and intelligent. We got along great. His family was difficult. Willfully ignorant.

Part of the succession dealt with clawing back $450,000 in insurance proceeds that a niece fraudulent received from the decedent by making herself the beneficiary on policies. A separate attorney was hired to handle that part before I was hired to handle all other matters, including settling a $500,000 debt with the...

like

Hey people

I have recently received an offer letter from PwC and have signed and accepted the offer (via Workday portal).
Now when I check the tasks against the job on workday, I can see no tasks pending and under completed tasks: a green tick against 'review and sign offer letter (step complete)' (and I can see the offer)
I am told that I would receive onboarding emails from HR, but I haven't yet.
My DoJ is in the fourth week of July. When do we generally expect these communications?

like

Hi fishes, anyone here who can refer me for mechanical engineering roles? I'm aero engine design engineer with 10 years experience working at Bangalore right now. Any referral is really nice appreciated. Thanks

likeuplifting

Junior IT developer looking to break into Customer Success to continue my coding practice on the side.
Anyone looking to hire? Worked at a unicorn SAAS company as a SDR. Great performer with references to back it up! Before that it’s been all sales! Mostly car sales and was able to travel to Aussies how to sell cars as well! I love being able to take my time to establish and building meaningful rapport with just about anyone! Lookin go for open environment, structure and great training!

like

Hi Fishes,
I belong to construction industry with PGDM from NICMAR in Advance Construction Management having expirence of 2yrs. I want to know how can I boom my career even after so much of higher education and expirence industry has slow career growth and toxic work culture. Iam not getting how to move further with such underpaid industry even sometime I get the thoughts for switching the career with data science which again needs time and money investment please suggest DS scope for Civil eng

like

“Those glasses don’t make you look younger, they make you look older—but slightly more cool.”

🥺

likefunnyhelpful

Compliance question here, I have always thought of the SALT deduction on the fed return as a perm item & have treated as such in such sch M-3. I see that some people put it as a temp on sch M-3. What is the justification for treating it as a temp on the M-3. You don’t set up a deferred on the def RF for the state tax deduct & it is also a rate impacting item (I know that some temps can be rate impacting). But this all leaves me to side with perm. Does anyone know the logic behind temp treatment?

like

I had requested a super brutal resume review a couple of days back here and got some excellent feedback. Added indents and cut down content (there was a lot more text earlier) according to suggestions.

Will be extremely grateful for an additional review with the changes. I am looking for business intelligence and business analyst roles.

I am also looking for a referral in Deloitte +kpmg for a systems/bi analyst role, will be super grateful for some help there too.

Post Photo
like

Additional Posts in Excel Genius

Will need to migrate to office 365 soon. What’s the difference for excel from the 2013 version?

likehelpful

What’s the keyboard shortcut to get into the “insert options” dialogue when you’ve inserted a new row with Ctrl+Shift+Plus?

like

Best free data sites to tap into to build a city analysis model? Basically I want to "discover" cities with certain dimensions, like: population, elevation, climate, and proximity to other things (ski resorts, particularly)?

like

Has anyone had an issue with index match and lookups returning NA when trying to manipulate data from a password protected workbook?

Can dates be grouped just by Month-Year in pivot tables on a Mac without having the year as its own dimension?

Post Photo
like

Timesaver: If you’re working on a file saved on the cloud (teams, share point, etc.) and need the URL to communicate out, tap FILE -> INFO -> COPY PATH and the url will be on your clipboard. Pre-established sharing privileges of orig folder will apply. Hopefully this saves you time reshuffling open windows! Bonus: Same on standard MS suite files.

likesmart

Recently made a job switch and have been using a lot more excel. Any free resources to help build a solid foundation / fee based courses to help improve my skills?

like

How do I look up values from a column in a row? E.g. Values in A1:A6 in B2:F2?

like

Is it possible to round a number (say rounding $98 to $100) without losing precision in following calcs? So I only want to round visually and I want the calcs from that cell to remain off the $98 not $100

likehelpful

Vlook up isn't recognizing a couple of cells even though it's feeding off a pivot and when I check with an " if =" it comes up as being the same.
Any advice on what I need to check?
Thanks

like

How can I copy a formula from one sheet to another?i tried the simple Ctrl+C, Ctrl-V, but the formula refers to the old sheet.

like

Need help building an electric schedule for a production schedule with a Gantt chart style view (sort of)

Currently have an excel list with resource (production line), start date and time, and then also the name of the item being produced.

Any ideas how to tackle this??

I’m thinking a stacked bar chart with the vertical categories to be my resources and horizontal the date/time and then each individual bar to be the name of the item being produced.

Thank you for the help Excel Gods 🙏

like

For a linear regression analysis… the equation of a line is Y=MX+B, where M is the slope. So if I’m measuring a line across 365 days and M=.2, can I say that I have increased production by 20% over a year. Or is production increasing 20% each day?

like

This is dumb, but. How can I pull and average the numbers in column b associated with the 4 values in column a? So, (sum of all As/total number of As) without filtering?

Post Photo
like

How long did it take you to become good at excel? Like fast?

like

How do I set the formula to calculate “If x falls between 1-10, return 1-10, if it falls between 11-20, return 11-20, and so on”

like

What is the best FP&A tool, Anaplan, Jirav, Planful, Vena? Any other recommendation?

like

Is there a way for me to automate creating columns to do this so I can drag across and create my columns? For example--
1Q2021 2Q2021 3Q2021 4Q2021 2021 1Q2022?

likesmart

Anybody use a personal computer for work? How's your experience been, any horror stories/watch-outs?

like

Would anyone be willing to teach me vlookup?

like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal