What's the best way to compare the similarity of text between a smaller set of documents with a larger set of documents? Therefore, two vectors of vastly different sizes.

like
Posting as :
works at
You are currently posting as works at

It depends: How many documents in the corpus? Is the difference in size uniform (I.e. only two sizes among all documents?)

Does order of text matter? Or will bag of words analysis suffice? If the latter, I’d assume TF-IDF should work just fine, paring down the words to highlight most unique terms, allowing comparison on the most unique words… but I’ve experienced BERT classifier having superior results, however it is not interpretable (aka it is good, but you won’t know why it is good).

How about TF-IDF and compare that side by side with BERT?

helpful

I've had 20 documents compared with 1000 documents and they do vary in size. I've had 5 documents compared with 125k documents and they also vary in size. The bag of words suffice so the order of text does not matter. I must be doing something wrong or not understanding something though... I thought I did do TDIDF and tried cosine similarity but could not get the matrix to be M x M? I tried doing a dot product of the vectors? Eli5 please lol.

Related Posts

Already spoke with a partner who wanted something done this morning for him to review and then we discuss again. Sent it to him about 4 hours ago. Should take him half an hour to review max. At what point can I check in with him to see when we can catch up so I’m not waiting in the office all day?

like

I’m struggling to find industry senior accounting roles that are offering much above 100K in the NYC market. Currently a S2 in audit in the B4, has this just been the norm recently or am I not looking for the right opportunities?

like

What’s the most unethical thing you’ve seen OC get away with?

like

Anyone have recommendations for comfortable office chair for home office? Our firm is back in the office, but still use my home office chair most evenings so I’m looking for something comfortable

like

What’s your take on Lydecker? What is the culture? Market salary? Expectation on billable hours? Mentorship? Etc.?

like

Any advise for wrist pinch pain? (Not looking for the surgery route). I also have an ergonomic keyboard.

like

Where did you get your leather briefcase/satchel? What are features you will be looking for in your next one?

likefunny

My firm recently changed its name. How do I update that on fishbowl? It keeps asking me to type in my work email and then doesn’t give me a way to change the name after I verified it 🤷🏻‍♂️

like

Working remotely for a US company but looking at moving to Canada (still keeping my US job) does anyone have experience with this? I’m assuming this would make PR trickier?

like

Does anyone use wealthsimple trade? Are there monthly or quarterly reports I can download for my investments?

In your experience have engineering salaries posted on Glassdoor been accurate? I’m seeing many that seem low

like

Anyone have insight on Adams and Reese (culture, pay, hours, etc)?

like

Hi Fishes,
I am currently working with TCS with 9.88lpa CTC.
I have got offer recently with Dell Techonolgies Chennai as a Senior Engineer with 17.88lpa CTC. Is it good to move to this company?. Anyone aware about the WLB in this company?

Tech Stack - SCCM/MECM, INTUNE

like
like

Hi People,
I'm looking for opportunities in Investment Banking Analyst roles in Mumbai.
I have 5 year's work experience and a finance major.
Please ping me if there's an opening in your organization and I would like to know more.

like

I am told that CSM is not a sales job (so not commission based) yet you are expected to respond to targets. Can you please explain the difference and whether there still is an element of pressure with hitting your targets?

like

Anyone else at CyberArk Impact this week? Anything exciting going on?

When it comes to therapy do I look up a place online and make an appointment or do I need some sort of referral? Not sure if I need to see a therapist or physiatrist really.

like

Anyone need assistance hiring for Q4? Comment below and I’m happy to assist!

Is it a red flag if, after almost two years in a full quota carrying and selling role I want to take a lower role at another company in order to break into a field (tech sales) I want to work in? It’s better pay - closer to home and at a company with a better reputation?

like

Additional Posts in Data & Analytics Consultants

from God import pandemic
From China import coronavirus, mortality_rate

covid_19 = pandemic.coronavirus.set_mortality(mortality = 1.)

like

Any cyber security folks here? or those interested in systems engineering? Potential role with TC 120-180, awesome team and wlb, at least compared to consulting.

like

Learning cloud architecture right now so apologies if these are obvious questions: Best ways to store video metadata to perform analysis on it? I'm looking at AWS and GCP. Saw that Cassandra would be good for it, but not sure what the GCP equivalent is? But also I'm a bit confused on if Cassandra is just a DB and storage would need to be elsewhere? All help is welcome! I've been spinning my wheels on Google trying to understand

like

MBB data science expert role (i.e., mostly identical progression as generalists but less up or out pressure) or third (?) tier tech companies (DataRobot, C3, etc.)?

Assume starting TC is same at both or 10% more at the tech companies. That's the figure they're roughly giving me during interviews. Also coming from Deloitte.

like

So pretty much you work In consulting, get paid half of what fang people make and In many cases work IB hours to meet clients continually changing demands . All the while the consulting project is built on a half thought out rushed tech stack/architecture? Meanwhile tech companies expose their employees to best in bread tech stacks? I really don’t understand why anyone in the field would work as a consultant doing data work? The only + is in consulting there are lots of low hangin fruit 🍎.

like

Can anyone at Ford Motor Company tell me more about the “Learning Analytics and Engagement Specialist” remote position? Very curious and would love a referral!

like

I need to do a timeline with lots of information over a long period of time. I wanted to that in Tableau, but partner will not pay license and data is sensitive so I cannot use Tableau public.
Do you have suggestions?
The requirements are the following:
- No online input data upload (so local editing)
- The produced output file must be local and easily exportable and readable
-timeline must be interactive (filter/click/zoom in/out/view detals etc.)
- free

Any kind of help is really appreciated

like

What are the top sales & marketing metrics that tech companies track? I have an interview for a Data architect role and the data aspect is my strength but need to brush up on the business insight part

like

Is 128k a fair base salary for Consultant in Deloitte Analytics & Cognitive. MS CS and about 3 yrs experience. In one of the fastest growing but midtier tech cities in the US.

like

When industries refer to business intelligence engineers, are they looking at a senior level or entry level? Some of these job descriptions seem rather outlandish in terms of years of experience required like 5+

like

What is the best way to handle group financial consolidation in a multinational company?

like

Anyone who has done masters in business analytics Singapore? Could you help on courses, Scholarships and careers and jobs. Looking for help.

Is there a good online course on storytelling with data? Something that can help structure your thought process and design choices when creating visualizations and Dashboards in Tableau, Power BI etc.

like

Is anyone working with Elastic at DoD?

1) Hypothetically if you were looking to sabotage a colleague, can you code an excel document to throw off their numbers or calculations, (if you send them that document and they download it for use)?

like

Hi! Does anyone know what kind of knowledge is needed to crack the Data Scientist interview at Google?

I'm currently working in Marketing Analytics (primarily using SQL and a bit of pyspark). Apart from mastering SQL, I wanted to know from people working at Google what does it take to get there as a Data Scientist? (I'm very interested in Strategy Ops)

I plan to another 6-8 months in preparation and upskilling myself before I even apply.
Any help and guidance is appreciated! Thanks :)

like

Big 4 Data Scientists, how did you exit to big tech firms? I’ve been unlucky with online applications and even referrals. I think it might be my resume or how I present myself. Can any free birds please give guidance?

like

Has anyone else begun to resent data science?

like

Have a lay-off meeting lined up, and now look for next opportunity. My background is around building machine learning models, data wrangling, visualization, feel confident with python. Open to any industry. Appreciate any lead on job opportunity. 3yrs of work experience

like

Single book recommendation for user research?

Goal is interview prep. Ideally focused on applications in tech.

like

New to Fishbowl?

Download the Fishbowl app to
unlock all discussions on Fishbowl.
That was just a preview…
Sign Up to see all discussions
  • Discover what it’s like to work at companies from real professionals
  • Get candid advice from people in your field in a safe space
  • Chat and network with other professionals in your field
Sign up in seconds to unlock all discussions on Fishbowl.

Already a user?
Login here

Share

Embed this post

Copy and paste embed code on your site

Preview

Download the
Fishbowl app

See what’s happening in your industry
from the palm of your hand.

A phone with Fishbowl app

Scan your QR code to download
Fishbowl app on your mobile

Download app

Sign up for free to view this conversation on Fishbowl

By continuing you agree to Terms of Use and Privacy Policy

Already have an account? Log in

Sign up for free to continue using Fishbowl

By continuing you agree to Terms of Use(New) and Privacy Policy(New)
Messaging rates may apply

Already have an account? Log in

For account settings, visit Fishbowl on Desktop Browser or

General

Legal