Salience highlights important sentences by treating your document as a graph where sentences that talk about similar things are connected. We then figure out which sentences are most "central" to the document's themes.
The first problem we need to solve is finding the sentences in a document. This is not as easy as splitting on newlines or periods. Consider this example:
Fortunately, this problem has been adequately solved for decades. We are going to use the **Punkt sentence splitter** (2003) available in the Natural Language Toolkit (NLTK) Python package.
## Step 2: Apply an Embedding Model
Now we have <Math tex="N" /> sentences. We convert each one into a high-dimensional vector that captures its meaning. For example:
Now we create a new <Math tex="N \times N" /> adjacency matrix <Math tex="\mathbf{A}" /> that measures how similar each pair of sentences is. For every pair of sentences <Math tex="i" /> and <Math tex="j" />, we need the **cosine similarity**:
Each <Math tex="A_{ij}" /> represents how strongly sentence <Math tex="i" /> is connected to sentence <Math tex="j" />.
- <Math tex="A_{ij} = 1" /> means sentences are identical in meaning
- <Math tex="A_{ij} = 0" /> means sentences are unrelated
- <Math tex="A_{ij} = -1" /> means sentences are opposite in meaning
You could work with these embedding vectors one at a time, using two for loops to build the adjacency matrix leetcode style. However, there's a way to delegate the computation to optimized libraries. Instead, organize all embeddings into a single matrix:
Since <Math tex="\mathbf{E}" /> is <Math tex="N \times D" /> and <Math tex="\mathbf{E}^T" /> is <Math tex="D \times N" />, their product gives us an <Math tex="N \times N" /> matrix where entry <Math tex="S_{ij} = \mathbf{e}_i \cdot \mathbf{e}_j" />.
This is an <Math tex="(N, 1)" /> vector where each element is the magnitude of one sentence's embedding. Now we need to visit every single element of <Math tex="\mathbf{S}" /> to make the adjacency matrix <Math tex="A_{ij} = \frac{S_{ij}}{n_i \cdot n_j}" />:
**Quick benchmark:** For a <Math tex="194 \times 768" /> embeddings matrix (194 sentences):
- Computing everything in Python for loops: **33.1 ms**
- Using <Math tex="\mathbf{E} \mathbf{E}^T" /> for dot products, but element-by-element normalization in Python: **10.9 ms** (saves 22.2 ms)
- Using numpy **broadcasting** for normalization too: **0.13 ms**
Broadcasting is a numpy feature where dividing arrays of different shapes automatically "stretches" the smaller array to match:
```python
def cos_sim(a):
sims = a @ a.T
norms = np.linalg.norm(a, axis=-1, keepdims=True)
sims /= norms # Divides each row i by norm[i]
sims /= norms.T # Divides each column j by norm[j]
return sims
```
The `keepdims=True` makes `norms` shape <Math tex="(N, 1)" /> instead of <Math tex="(N,)" />, which is crucial—when transposed, <Math tex="(N, 1)" /> becomes <Math tex="(1, N)" />, allowing the broadcasting to work for column-wise division.
**Important assumption:** This assumes your document has a coherent main idea and that sentences are generally on-topic. We're betting that the topic with the most "semantic mass" is the *correct* topic. This is obviously not true for many documents:
The algorithm might highlight the criticism because multiple sentences cluster around "danger", even though the document's actual position is pro-nuclear. There's nothing inherent in the math that identifies authorial intent vs. quoted opposition.
**Bottom line:** This technique works well for coherent, single-perspective documents. It can fail when multiple competing viewpoints have similar semantic weight.
## Step 5: Normalize the Adjacency Matrix
The idea from **TextRank** is to treat similarity as a graph problem: simulate random walks and see where you're likely to end up. Sentences you frequently visit are important.
But first, we need to compute the **degree matrix** <Math tex="\mathbf{D}" />. This tells us how "connected" each sentence is:
**Intuition:** A sentence with high degree (<Math tex="d_i" /> is large) is connected to many other sentences or has strong connections. A sentence with low degree is more isolated.
Now we use <Math tex="\mathbf{D}" /> to normalize <Math tex="\mathbf{A}" />. There are two approaches:
Traditional normalization <Math tex="\mathbf{D}^{-1} \mathbf{A}" />:
- This creates a row-stochastic matrix (rows sum to 1)
- Interpretation: "If I'm at sentence <Math tex="i" />, what's the probability of jumping to sentence <Math tex="j" />?"
- This is like a proper Markov chain transition matrix
- Symmetry preservation: if A is symmetric (which cosine similarity matrix is), then the normalized version
stays symmetric
- The eigenvalues are bounded in [-1, 1]
- More uniform influence from all neighbors
- Better numerical properties for exponentiation
The traditional <Math tex="\mathbf{D}^{-1} \mathbf{A}" /> approach introduces potential node bias and lacks symmetry. Spectral normalization
provides a more balanced representation by symmetrizing the adjacency matrix and ensuring more uniform
neighbor influence. This method prevents high-degree nodes from dominating the graph's structure, creating a
more equitable information propagation mechanism.
With traditional normalization, sentences with many connections get their influence diluted. A sentence connected to 10 others splits its "voting power" into 10 pieces. A sentence connected to 2 others splits its power into just 2 pieces. This creates a bias against well-connected sentences.
Spectral normalization treats the graph as **undirected**, which matches how
semantic similarity works. Well-connected sentences keep their influence
proportional to connectivity. Two sentences that are similar to each other
should have equal influence on each other, not asymmetric transition
probabilities.
## Step 6: Random Walk Simulation
We simulate importance propagation by raising the normalized matrix to a power:
- <Math tex="\mathbf{1}" /> = vector of ones (start with equal weight on all sentences)
- <Math tex="k" /> = random walk length (default: 5)
- <Math tex="\mathbf{s}" /> = raw salience scores for each sentence
**Intuition:** After <Math tex="k" /> steps of random walking through the similarity graph, which sentences have we visited most? Those are the central, important sentences.
## Step 7: Map Scores to Highlight Colors
Now we have a vector of raw salience scores from the random walk. Problem: these scores have no physical meaning. Different embedding models produce wildly different ranges:
- Model A on Doc 1: `[0.461, 1.231]`
- Model B on Doc 2: `[0.892, 1.059]`
We need to turn this vector of arbitrary numbers into CSS highlight opacities in `[0, 1]`. Here's the reasoning behind creating the remapping function:
I could do trivial linear scaling - multiply by a constant to get scores into some range like <Math tex="X" /> to <Math tex="X + 2" />. But let's try to make the top sentences stand out more. One trick: exponentiation. Since human perception of brightness is not linear, exponentiation will preserve order but push the top values apart more. It makes the top few sentences really pop out.
**Building the remapping function**
Given a salience vector <Math tex="\mathbf{s}" /> with values ranging from <Math tex="\min(\mathbf{s})" /> to <Math tex="\max(\mathbf{s})" />:
1. **Find an exponent** <Math tex="p" /> such that <Math tex="\max(\mathbf{s}^p) \approx \min(\mathbf{s}^p) + 2" />
Sure, it takes more work to find the right exponent for our target spread of 2, but that's still easy with a simple solver.
2. **Find a threshold** <Math tex="\tau" /> such that 50% of the sentences get clamped to zero.
Since I'm using this for editing documents, I only want to see highlights on roughly half the sentences—the important half.
For each document, I use a simple 1D solver to find <Math tex="p" /> and <Math tex="\tau" /> that satisfy these constraints.
**Final thought:** This last step—converting the output from TextRank into highlight colors—is the weakest part of the system. I have no idea if it's actually correct or whether it even allows meaningful comparison between different embedding models. It works well enough for the intended purpose (quickly seeing which sentences to keep when editing), but the numerical values themselves are essentially arbitrary.
