scipy: pdist indexing

scipy.spatial.distance.pdist(X) gives the pair-wise distances of X, $\mathrm{dist}(X[i],X[i])$ not included. The distances are stored in a dense matrix D.

Question: how to get $\mathrm{dist}(X[i],X[j])$ without expanding this dense matrix by scipy.spatial.distance.squareform(D)?

A good answer from How does condensed distance matrix work? (pdist):

def square_to_condensed(i, j, n):
    assert i != j, "no diagonal elements in condensed matrix"
    if i < j:
        i, j = j, i
    return n*j - j*(j+1)/2 + i - 1 - j

Take $n$ == X.shape[0] == $4$ as an example. In the code, $i$ and $j$ are swapped if $i<j$, so only consider the bottom triangular region, i.e. $i>j$ always holds in the discussion below.

Suppose the indexing function is $\mathrm{f}$: $\mathrm{f}(i,j)=k$ when $\mathrm{dist}(X[i],X[j])=D[k]$

len(D) = $(\genfrac{}{}{0ex}{}{n}{2})=(n-1)+(n-2)+\cdots +1$.

• When $j=0$, $\mathrm{f}(i,j)=i-j-1$
• When $j=1$, $\mathrm{f}(i,j)=(n-1)+(i-j-1)$
• When $j=2$, $\mathrm{f}(i,j)=(n-1)+(n-2)+(i-j-1)$

So the pattern here is:

Further more: