---
title: How Salience Works
---

import { Math } from "~/components/math/math"

# How Salience Works

Salience highlights important sentences by treating your document as a graph where sentences that talk about similar things are connected. We then figure out which sentences are most "central" to the document's themes.

## Step 1: Break Text into Sentences

We use NLTK's Punkt tokenizer to split text into sentences. This handles tricky cases where simple punctuation splitting fails:

*"Dr. Smith earned his Ph.D. in 1995."* ← This is **one** sentence, not three!

## Step 2: Convert Sentences to Embeddings

Now we have <Math tex="N" /> sentences. We convert each one into a high-dimensional vector that captures its meaning:

<Math display tex="\mathbf{E} = \text{model.encode}(\text{sentences}) \in \mathbb{R}^{N \times D}" />

This gives us an **embeddings matrix** <Math tex="\mathbf{E}" /> where each row is one sentence:

<Math display tex="\mathbf{E} = \begin{bmatrix} a_1 & a_2 & a_3 & \cdots & a_D \\ b_1 & b_2 & b_3 & \cdots & b_D \\ c_1 & c_2 & c_3 & \cdots & c_D \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ z_1 & z_2 & z_3 & \cdots & z_D \end{bmatrix}" />

Where:
- <Math tex="N" /> = number of sentences (rows)
- <Math tex="D" /> = embedding dimension (768 for all-mpnet-base-v2, 1024 for gte-large-en-v1.5)
- Each row represents one sentence in semantic space

## Step 3: Build the Adjacency Matrix

Now we create a new matrix <Math tex="\mathbf{A}" /> that measures how similar each pair of sentences is. For every pair of sentences <Math tex="i" /> and <Math tex="j" />, we compute:

<Math display tex="A_{ij} = \frac{\mathbf{e}_i \cdot \mathbf{e}_j}{\|\mathbf{e}_i\| \|\mathbf{e}_j\|}" />

This is the **cosine similarity** between their embedding vectors. It tells us:
- <Math tex="A_{ij} = 1" /> means sentences are identical in meaning
- <Math tex="A_{ij} = 0" /> means sentences are unrelated
- <Math tex="A_{ij} = -1" /> means sentences are opposite in meaning

The result is an <Math tex="N \times N" /> **adjacency matrix** where <Math tex="A_{ij}" /> represents how strongly sentence <Math tex="i" /> is connected to sentence <Math tex="j" />.

## Step 4: Clean Up the Graph

We make two adjustments to the adjacency matrix to get a cleaner graph:

1. **Remove self-loops:** Set diagonal to zero (<Math tex="A_{ii} = 0" />)
   - A sentence shouldn't vote for its own importance

2. **Remove negative edges:** Set <Math tex="A_{ij} = \max(0, A_{ij})" />
   - Sentences with opposite meanings get disconnected

**Important assumption:** This assumes your document has a coherent main idea and that sentences are generally on-topic. We're betting that the topic with the most "semantic mass" is the *correct* topic.

**Where this breaks down:**
- **Dialectical essays** that deliberately contrast opposing viewpoints
- **Documents heavy with quotes** that argue against something
- **Debate transcripts** where both sides are equally important
- **Critical analysis** that spends significant time explaining a position before refuting it

For example: "Nuclear power is dangerous. Critics say it causes meltdowns. However, modern reactors are actually very safe."

The algorithm might highlight the criticism because multiple sentences cluster around "danger", even though the document's actual position is pro-nuclear. There's nothing inherent in the math that identifies authorial intent vs. quoted opposition.

**Bottom line:** This technique works well for coherent, single-perspective documents. It can fail when multiple competing viewpoints have similar semantic weight.

## Step 5: Normalize the Adjacency Matrix

The idea from **TextRank** is to treat similarity as a graph problem: simulate random walks and see where you're likely to end up. Sentences you frequently visit are important.

But first, we need to compute the **degree matrix** <Math tex="\mathbf{D}" />. This tells us how "connected" each sentence is:

<Math display tex="\mathbf{D} = \text{diag}(\mathbf{A} \mathbf{1})" />

Here's what this means:
- <Math tex="\mathbf{A} \mathbf{1}" /> means "sum up each row of <Math tex="\mathbf{A}" />"
- For sentence <Math tex="i" />, this gives us <Math tex="d_i = \sum_j A_{ij}" /> (the total similarity to all other sentences)
- <Math tex="\text{diag}(...)" /> puts these sums on the diagonal of a matrix

The result is a diagonal matrix that looks like:

<Math display tex="\mathbf{D} = \begin{bmatrix} d_1 & 0 & 0 & \cdots & 0 \\ 0 & d_2 & 0 & \cdots & 0 \\ 0 & 0 & d_3 & \cdots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \cdots & d_N \end{bmatrix}" />

**Intuition:** A sentence with high degree (<Math tex="d_i" /> is large) is connected to many other sentences or has strong connections. A sentence with low degree is more isolated.

Now we use <Math tex="\mathbf{D}" /> to normalize <Math tex="\mathbf{A}" />. There are two approaches:

Traditional normalization <Math tex="\mathbf{D}^{-1} \mathbf{A}" />:
  - This creates a row-stochastic matrix (rows sum to 1)
  - Interpretation: "If I'm at sentence <Math tex="i" />, what's the probability of jumping to sentence <Math tex="j" />?"
  - This is like a proper Markov chain transition matrix
  - Used in standard PageRank and TextRank

Spectral normalization <Math tex="\mathbf{D}^{-1/2} \mathbf{A} \mathbf{D}^{-1/2}" />:
  - Used in spectral clustering and graph analysis
  - Symmetry preservation: if A is symmetric (which cosine similarity matrix is), then the normalized version
  stays symmetric
  - The eigenvalues are bounded in [-1, 1]
  - More uniform influence from all neighbors
  - Better numerical properties for exponentiation


The traditional <Math tex="\mathbf{D}^{-1} \mathbf{A}" /> approach introduces potential node bias and lacks symmetry. Spectral normalization
provides a more balanced representation by symmetrizing the adjacency matrix and ensuring more uniform
neighbor influence. This method prevents high-degree nodes from dominating the graph's structure, creating a
more equitable information propagation mechanism.

With traditional normalization, sentences with many connections get their influence diluted. A sentence connected to 10 others splits its "voting power" into 10 pieces. A sentence connected to 2 others splits its power into just 2 pieces. This creates a bias against well-connected sentences.

Spectral normalization treats the graph as **undirected**, which matches how
semantic similarity works. Well-connected sentences keep their influence
proportional to connectivity. Two sentences that are similar to each other
should have equal influence on each other, not asymmetric transition
probabilities.


## Step 6: Random Walk Simulation

We simulate importance propagation by raising the normalized matrix to a power:

<Math display tex="\mathbf{s} = \mathbf{1}^T \tilde{\mathbf{A}}^k" />

Where:
- <Math tex="\mathbf{1}" /> = vector of ones (start with equal weight on all sentences)
- <Math tex="k" /> = random walk length (default: 5)
- <Math tex="\mathbf{s}" /> = raw salience scores for each sentence

**Intuition:** After <Math tex="k" /> steps of random walking through the similarity graph, which sentences have we visited most? Those are the central, important sentences.

## Step 7: Map Scores to Highlight Colors

Now we have a vector of raw salience scores from the random walk. Problem: these scores have no physical meaning. Different embedding models produce wildly different ranges:
- Model A on Doc 1: `[0.461, 1.231]`
- Model B on Doc 2: `[0.892, 1.059]`

We need to turn this vector of arbitrary numbers into CSS highlight opacities in `[0, 1]`. Here's the reasoning behind creating the remapping function:

I could do trivial linear scaling - multiply by a constant to get scores into some range like <Math tex="X" /> to <Math tex="X + 2" />. But let's try to make the top sentences stand out more. One trick: exponentiation. Since human perception of brightness is not linear, exponentiation will preserve order but push the top values apart more. It makes the top few sentences really pop out.

**Building the remapping function**

Given a salience vector <Math tex="\mathbf{s}" /> with values ranging from <Math tex="\min(\mathbf{s})" /> to <Math tex="\max(\mathbf{s})" />:

1. **Find an exponent** <Math tex="p" /> such that <Math tex="\max(\mathbf{s}^p) \approx \min(\mathbf{s}^p) + 2" />

   Sure, it takes more work to find the right exponent for our target spread of 2, but that's still easy with a simple solver.

2. **Find a threshold** <Math tex="\tau" /> such that 50% of the sentences get clamped to zero.

   Since I'm using this for editing documents, I only want to see highlights on roughly half the sentences—the important half.

The final opacity mapping is:

<Math display tex="\text{opacity}_i = \text{clamp}\left(s_i^p - \tau, 0, 1\right)" />

For each document, I use a simple 1D solver to find <Math tex="p" /> and <Math tex="\tau" /> that satisfy these constraints.

**Final thought:** This last step—converting the output from TextRank into highlight colors—is the weakest part of the system. I have no idea if it's actually correct or whether it even allows meaningful comparison between different embedding models. It works well enough for the intended purpose (quickly seeing which sentences to keep when editing), but the numerical values themselves are essentially arbitrary.

---

[← Back to App](/)