As we have seen in the previous section, the dimensionality course in our data was amplified due to the n-gram technique. We would like to be able to use n-grams to bring back word ordering into our DFM, but we would like to reduce the feature space at the same time. To accomplish this, we can use a number of different dimensionality reduction techniques. In this case, we will show how to use the SVD.

The SVD helps us compress the data by using it's singular vectors instead of the original features. The math behind the technique is out of the scope of the book, but we encourage you to look at Meyer's, Matrix Analysis & Applied Linear Algebra, 2000. Basically, you can think of the singular vectors as the important directions in the data, so instead of using our normal axis, we can use these singular vectors in a transformed space where...