To assist you in learning fundamental NLP concepts and building your NLP applications, we will start with NLP concepts and techniques that enable commercial NLP applications. This guide covers both theories and code practices. It presents NLP topics, so beginners as well as experienced data scientists can benefit from it.

Many of the techniques mentioned earlier, such as Word2Vec, Doc2Vec, LSA, LDA, and Ensemble LDA, are included in the Python Gensim module. Gensim is an open source Python library widely used by NLP researchers and developers, together with other NLP open source modules, including NLTK, Scikit-learn, and spaCy. We will learn how to build models using these modules. In addition, you will also learn about the Transformer-based topic modeling BERTopic in a separate chapter, and a BERTopic use case in the last chapter for NLP use cases.

You will also get to practice implementing your model for scoring and predictions. This implementation perspective enables you to work with data engineers closely in model deployment. We’ll conclude the book with a study of selected large-scale NLP use cases. We believe these use cases can inspire you to build your NLP applications.

What is Gensim

New NLP learners may find the Gensim library cited in many tutorials. Gensim is an open source Python library to process unstructured texts using unsupervised machine learning algorithms. It was first created by Radim Řehůřek in 2011 and is now developed and maintained continually by 400+ contributors. It has been used in over 2000 research papers and student theses.

One of Gensim’s merits is its fast execution speed. Gensim attributes this advantage to its use of low-level BLAS libraries through NumPy, highly optimized Fortran/C, and multithreading under the hood. Memory independence is also one of their design objectives. Gensim enables data streaming to process large corpora without the need to load a whole training corpus in RAM.