How to Train a Logistic Regression Model

Training a logistic regression classifier is based on several steps: process your data, train your model, and test the accuracy of your model. NLP engineers from Belitsoft prepare text data and build, train, and test machine learning models, including logistic regression, depending on our clients' project needs.

What is a Logistic Regression Model?

Building classification models is one of the common tasks in NLP development.

One common classification task is sentiment analysis.

It involves classifying text as positive, or negative.

For example, you can build a system based on a classification model that automatically goes through thousands of product reviews written by users to figure out how many are positive reviews and how many are negative.

There are many models for sentiment analysis. One of them is based on the logistic regression algorithm.

The logistic regression algorithm is easy to train. It also provides a good baseline result to measure performance before trying more complex models on the same data for comparison.

#1 Feature Extraction

Raw text cannot be directly used by logistic regression or other machine learning models. You need to represent text data numerically, in other words, vectorize it using a vocabulary, or in other words, extract features from text data. 

Extracting features is typically the first step in the process of training a logistic regression model dealing with unstructured text data.

For example, if we have the vocabulary ["I", "am", "happy", "sad"], the features will be individual words "I", "am", "happy", and "sad" from that vocabulary. And feature extraction will be converting words from analyzed sentences into numbers. That's why, the vector representation of the sentence "I am happy" will be [1, 1, 1, 0], where "1" means that the word from vocabulary exists in the sentence, and "0" means it doesn't.

The binary vector method (1/0) is mostly used to explain what a vocabulary is, what features are, and how to convert text into a vector (so machine learning models can process it). It's useless for real-life scenarios.

Create a Vocabulary for this NLP Task

Vector is commonly represented as a list of digits enclosed in brackets.

Before representing a text as a vector, you have to build a vocabulary, or the list of unique words from all your raw reviews, by going through all the words from all these texts and saving every new word to the vocabulary, removing all repeated words.

Extract Features

The process of extracting features is based on the previously created vocabulary. 

We check if every word from the vocabulary appears in the text (in our case, the review).

If it does, that word or feature gets the value “1”, if it doesn't, it gets “0”.

Sparse Data Problem

The issue of sparse representation of data refers to situations where vectors contain a high proportion of features equal to zeros. 

It means that our vocabulary is very large because we have a lot of text with different words, but each separate text we want to classify contains only a few words from the vocabulary that can be represented as “1”. 

All other words from the vocabulary aren't present, but we still must represent them as '0' in the vector for this text.

The larger the vocabulary, the more time is necessary to train your model, and the longer it will take to make predictions. 

This is because the model must compare more and more words from each analyzed text to all the words in the vocabulary (to understand how each word's weight contributes to the classification of the whole text as positive or negative). 

Even if words from the vocabulary are not present in the analyzed text, they still have weights and must be represented in the feature vectors as zeros.

Frequency-based Feature Selection Method for Classification

In machine learning within NLP, feature selection means choosing which features (numerical inputs) you will use in your model. The word "selection" refers to deciding which features to include.

With full vocabulary vectors, each word is a feature (potentially thousands of features). This brings the sparse data problem. 

To avoid that, you can select a better option and use only two features — the total positive frequency and the total negative frequency.

This is called frequency-based feature selection because you select not words but word frequency sums as your features instead.

Note: This article is not the finalized version and will be completed soon.