Hierarchy of Transformation

1. Affine and projective transformation

For two arbitrary ideal points which are of the form `(x,y,0)^t`, these points are always on the line `l_{infty}=(0,0,1)^t` since `(x_1,y_1,0)^t times(x_2,y_2,0)^t=l_{infty}`.
A point on `l_{infty}` is transformed to another point on `l_{infty}` by affine transformation. But it is transformed to a point on the vanishing line by projective transformation.

2. How to remove the projective distortion

Projective transformation causes the distortion such that parallel lines in affine space are not parallel any more because a vanishing line comes about.
Let the vanishing line be `l=(l_1,l_2,l_3)^t`, then `l` is mapped to `l_infty` by a matrix `H` as follows:

\begin{equation} H=
\begin{pmatrix}
1 & 0 & 0 \\
0 & 1 & 1 \\
l_1 & l_2 & l_3
\end{pmatrix}
\end{equation}

While a point is transformed by `H`, a normal vector is transformed by `H^{-t}`. So `H^{-t}l=l_infty`.

\begin{equation} H^{-t}l=
\begin{pmatrix}
1 & 0 & -\frac{l_1}{l_3} \\
0 & 1 & -\frac{l_2}{l_3} \\
0 & 0 & \frac{1}{l_3}
\end{pmatrix}
\begin{pmatrix} l_1 \\ l_2 \\ l_3 \end{pmatrix} =
\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}
= l_\infty
\end{equation}

Therefore, `H^{-t}` transforms any line `l` into `l_infty`, which means that it restores the projective distortion. But it still remains the affine distortion.

3. The angle between two lines

For two lines `l` and `m`, the angle `theta` between them is the following:

\begin{align*}
\cos\theta&=\frac{l_1m_1+l_2m_2}{\sqrt{(l_1^2+l_2^2)(m_1^2+m_2^2 )}}=
\frac{l^tC_{\infty}^*m}{\sqrt{( l^tC_{\infty}^*l)(m^tC_{\infty}^*m)}},\\ \\
C_{\infty}^*&=
\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}
\text{: dual degenerate conic}
\end{align*}

4. How to remove the affine distortion

Even though the projective distortion is removed, there is still the affine distortion, which a square shape turns into a parallelogram shape. In other words, the angle between lines is not preserved.
Assume that two lines `l`, `m` and dual degenerate conic `C_infty^ast` in the original space are transformed to `l^{prime}`, `m^{prime}` and `C_infty^{prime ast}` by affine transformation. Then `lprime=H^{-t}l`, `mprime=H^{-t}m`, and ` Cprime_infty^ast =H C_infty^ast H^t`.
For the angle `theta` between `l` and `m`, the numerator of `costheta` is `l^tC_infty^astm=``(l ^{primet}H)(H^{-1} C_infty^{prime ast} H^{-t})(H^t m^{prime})=``l ^{primet} C_infty^{prime ast} m^{prime}`.
When `l` and `m` are perpendicular, `l^{primet}C_infty^{prime ast}m^{prime}=0`. It yields as follows:

\begin{align}
l^{\prime t}C_{\infty}^{\prime \ast}m^{\prime} =
l^{\prime t}HC_{\infty}^*H^tm^{\prime} =
\begin{pmatrix} l_1^{\prime} & l_2^{\prime} & l_3 ^{\prime} \end{pmatrix} &
\begin{pmatrix} A & \vec{t} \\ \vec{0} & 1 \end{pmatrix}
\begin{pmatrix} I & \vec{0} \\ \vec{0} & 0 \end{pmatrix}
\begin{pmatrix} A & \vec{0} \\ \vec{t} & 1 \end{pmatrix}
\begin{pmatrix} m_1^{\prime} \\ m_2^{\prime} \\ m_3^{\prime} \end{pmatrix} =0, \\
\begin{pmatrix} l_1^{\prime} & l_2^{\prime} & l_3^{\prime} \end{pmatrix}
\begin{pmatrix} AA^t & \vec{0} \\ \vec{0} & 0 \end{pmatrix}
\begin{pmatrix} m_1^{\prime} \\ m_2^{\prime} \\ m_3^{\prime} \end{pmatrix} &=
\begin{pmatrix} l_1^{\prime} & l_2^{\prime} \end{pmatrix}
\begin{pmatrix} s_{11} & s_{12} \\ s_{21} & s_{22} \end{pmatrix}
\begin{pmatrix} m_1^{\prime} \\ m_2^{\prime} \end{pmatrix} =0 \\
\text{where}
\begin{pmatrix} s_{11} & s_{12} \\ s_{21} & s_{22} \end{pmatrix} =S=AA^t
\end{align}

`S` is symmetric, `s_{12}=s_{21}`, and `s_{22}` can be set to `1` since `S` is defined up to scale.
`s_{11}l_1^{prime}m_1^{prime}+s_{12}(l_1^{prime}m_2 ^{prime} + l_2 ^{prime} m_1 ^{prime} )+l_2 ^{prime} m_2^{prime}=0`
If we get two pairs of `l` and `m` which are perpendicular, then `S` can be calculated.
Having calculated `S`, `H` can be estimated assuming `A` is positive-definite. From `S=A A^t`, the eigenvalues of `A` are the positive sqaure roots of the eigenvalues of `S`.
After removing the affine distortion, there still remains the similarity distortion.

5. Hierarchy of transformation in 3D is similar to that in 2D

As a point on `l_{infty}` is transformed to another point on `l_{infty}` by affine transformation, a point on the plane `Pi_infty=(0,0,0,1)^t` is also transformed to another point on `Pi_infty`.
As a point on `l_{infty}` is transformed to a point on the vanishing line by projective transformation, a point on `Pi_infty=(0,0,0,1)^t` is transformed to a point on the specific plane.

When a cube is transformed in projective space, each a pair of parallel lines is not parallel any more and intersects at a point which is not at infinity such as `v_1`, `v_2`, and `v_3`.
These points `v_1`, `v_2` and `v_3` consist of a plane `Pi` which does correspond to `Pi_infty` in affine space.
If we know this plane `Pi`, it can be transformed to `Pi_infty` by some matrix `H`, so the projective distortion can be removed.
Let `Pi=(pi_1, pi_2, pi_3, pi_4)^t=(bar{pi}^t, pi_4)^t`, then the following `H` restores `Pi` to `Pi_infty`:

\begin{align} H=
\begin{pmatrix} A & \vec{0} \\ \bar{\pi^t} & \pi_4 \end{pmatrix}, &\quad H^{-1}=
\begin{pmatrix} A^{-1} & \vec{0} \\ -\frac{1}{\pi_4}\bar{\pi^t}A^{-1} & \frac{1}{\pi_4} \end{pmatrix},
\quad H^{-t}=
\begin{pmatrix} A^{-t} & -\frac{1}{\pi_4}A^{-t}\bar{\pi} \\ \vec{0} & \frac{1}{\pi_4} \end{pmatrix} \\ \\
H^{-t}\Pi &=
\begin{pmatrix} A^{-t} & -\frac{1}{\pi_4}A^{-t}\bar{\pi} \\ \vec{0} & \frac{1}{\pi_4} \end{pmatrix}
\begin{pmatrix} \bar{\pi} \\ \pi_4 \end{pmatrix} =
\begin{pmatrix} \vec{0} \\ 1 \end{pmatrix} = \Pi_{\infty}
\end{align}

`bar{pi}` is the orientation of `Pi` and `Pi` is a homogeneous representation, so `Pi~frac{1}{pi_4}Pi`.

Reference