How To Train Your Embedding Models

from Nils Reimers “Training SOTA Text Embedding & Neural Search Models”

Can we just average BERT? No.

base bert out of the box performs badly, but we can train bert with either unsupervised methods or with NLI data. dataset size scaling law applies here.

Global And Local Structure of Vector Space

• Global structure: Relation of two random sentences,
• Local structure: Relation of two similar sentences.
• Loss function must optimize local and global structure.
• Contrastive or triplet loss might only optimize the local structure.

Multiple Negative Ranking Loss

A nice loss function is Multiple Negative Ranking Loss

• Given positive pairs: $(a_i,p_i)$, it should be close in vector space.
• Any other negative pairs, denoted $(a_i,p_j)$, should be distant in vector space $(i\ne j)$. Intuitively, it’s rather unlikely that two randomly selected questions are that similar.
• We then compute the ranking loss with Cross-Entropy:
  • Given $a_1$, what is the right answer out of $[p_1,p_2,p_3]$?
  • Compute the similarity scores to obtain $[s(p_1),s(p_2),s(p_3)]$,
  • Compute cross entropy loss with gold label $[1,0,0]$
• Mathematically, MNRL can be denoted as: $L=-\frac{1}{n}{\sum }_{i=1}^n\frac{exp(sim(a_i,p_i))}{{\sum }_{j}exp(sim(a_i,p_j))}$ where $sim$ is the similarity function for two vectors.
• MNRL considers either cosine similarity or dot product similarity.

Hard Negatives