Question: If a vertex V is transformed by a matrix M, why do I need to transform the associated normal vector N by the transpose of the inverse of M?

The answer is going to be in two parts.

First we need to understand what we really mean when we say "a point lies on a plane." Recall that by convention we represent a plane P by the row vector [a b c d] and a point V by the column vector [x y z 1]T.

 Representing a plane: Representing a point: [a b c d] /||\ xyz1 \ | |/
The statement "vertex V lies on plane P" is true if and only if ax + by + cz + d = 0. In other words, whenever P • V = 0.

Now consider what happens when we transform all the vertices of a shape by some matrix M. In that case, every vertex V is replaced by V' = (M • V).

So how then would we transform planes? Clearly we want to transform planes so that they still contain the same vertices. In other words, we want to find P' such that:

P' • V' = 0   if and only if   P • V = 0.

But this means that we want P' = (P • M-1), since then:

P' • V' =
(P • M-1) • (M • V) =
P • (M-1 • M) • V =
P • V = 0