When diving into the world of deep learning and neural networks, choosing the right loss function is crucial for achieving optimal performance. Two commonly used loss functions for classification problems are binary_crossentropy and categorical_crossentropy. While they might seem interchangeable at first glance, especially when dealing with binary classification scenarios, understanding the nuances of why binary_crossentropy and categorical_crossentropy give different performances for the same problem is essential. The performance discrepancies arise from differences in how these loss functions interpret the input data and calculate the loss, leading to variations in training dynamics and ultimately, model accuracy. This difference in performance highlights the importance of carefully selecting the appropriate loss function based on the specific characteristics of your dataset and the nature of the classification task. Choosing the wrong loss function can lead to suboptimal model training, slower convergence, or even complete failure to learn the underlying patterns in the data. This article delves into the intricacies of these two loss functions, exploring their mathematical foundations, practical applications, and the factors that contribute to their varying performances.

Understanding Binary Crossentropy

Binary_crossentropy, also known as log loss, is specifically designed for binary classification problems, where the goal is to predict one of two possible outcomes (e.g., yes/no, true/false, cat/dog). It measures the dissimilarity between the predicted probability and the true label, which is either 0 or 1. The function penalizes incorrect predictions more heavily as the predicted probability deviates further from the true label. This incentivizes the model to make confident and accurate predictions. Binary crossentropy assumes that each data point belongs to only one class; it directly models the probability of belonging to the positive class.

The mathematical formula for binary crossentropy is relatively straightforward: -[y log(p) + (1 - y) log(1 - p)], where ‘y’ is the true label (0 or 1) and ‘p’ is the predicted probability. This formula calculates the average cross-entropy across all data points in the training set. The lower the cross-entropy value, the better the model’s performance. When ‘y’ is 1, the loss is -log(p), and when ‘y’ is 0, the loss is -log(1-p). This ensures that the loss is always non-negative and that it increases as the prediction becomes less accurate. According to Andrew Ng’s deep learning course, “Binary cross-entropy is a standard loss function for binary classification problems” [1].

In practice, binary_crossentropy is commonly used in applications like spam detection (identifying whether an email is spam or not), medical diagnosis (determining whether a patient has a certain disease), and fraud detection (assessing whether a transaction is fraudulent). These scenarios all involve predicting a single binary outcome. For instance, in spam detection, the model learns to associate certain features of an email (e.g., sender address, subject line, content) with the probability of it being spam. The binary crossentropy loss then guides the model to adjust its parameters so that these probabilities align as closely as possible with the true labels (spam or not spam).

Delving into Categorical Crossentropy

Categorical_crossentropy, on the other hand, is designed for multi-class classification problems, where the goal is to predict one of several possible outcomes (e.g., classifying images of different animals, identifying the type of flower in an image). It measures the dissimilarity between the predicted probability distribution and the true distribution of classes, which is represented as a one-hot encoded vector. One-hot encoding means that for each data point, only one element in the vector is 1 (representing the true class), while all other elements are 0. Categorical crossentropy assumes that each data point belongs to exactly one class out of the available classes.

The formula for categorical_crossentropy is slightly more complex than binary crossentropy: - Σ [y_i log(p_i)], where ‘y_i’ is the true probability (0 or 1) for class ‘i’, and ‘p_i’ is the predicted probability for class ‘i’. The summation is performed over all classes. Similar to binary crossentropy, the goal is to minimize the average cross-entropy across all data points and classes. A key requirement for using categorical crossentropy is that the output layer of the neural network must use a softmax activation function. Softmax ensures that the predicted probabilities for all classes sum up to 1, creating a valid probability distribution. Using softmax is crucial for the proper functioning of categorical crossentropy. TensorFlow documentation states that, “Categorical crossentropy is best used when the output layer uses the softmax activation.” [2]

A common application of categorical_crossentropy is image classification, such as classifying images of different types of vehicles (e.g., car, truck, motorcycle, bicycle). The model learns to extract features from the images and associate them with the probabilities of belonging to each vehicle type. The categorical crossentropy loss then guides the model to adjust its parameters so that these probabilities align as closely as possible with the true labels (one-hot encoded vectors indicating the correct vehicle type for each image). In this case, the use of one-hot encoding allows the loss function to effectively compare the predicted probability distribution with the known ground truth.

Why the Difference in Performance?

The primary reason for the differing performance of binary_crossentropy and categorical_crossentropy lies in the way they handle the output layer and the interpretation of the target variables. When you have a binary classification problem (two classes), you can technically represent the labels using either a single output neuron with a sigmoid activation (and use binary crossentropy) or two output neurons with a softmax activation (and use categorical crossentropy). However, the way these setups learn and the information they utilize is different.

Here’s a breakdown:

• Output Layer and Activation: Binary_crossentropy typically uses a sigmoid activation function in the output layer, which outputs a single probability value between 0 and 1, representing the probability of belonging to the positive class. Categorical_crossentropy, on the other hand, typically uses a softmax activation function, which outputs a probability distribution over all classes. Even in a binary case, softmax treats it as two separate, mutually exclusive classes.
• Target Variable Encoding: Binary_crossentropy expects the target variable to be a single value (0 or 1). Categorical_crossentropy expects the target variable to be a one-hot encoded vector. This means that even for a binary problem, you’d need to convert your labels into a two-element vector (e.g., [1, 0] for class 0 and [0, 1] for class 1).
• Loss Calculation: Binary_crossentropy directly models the probability of one class versus the other. It directly penalizes the model for being wrong on either the positive or negative prediction. Categorical_crossentropy, even in the binary case, considers the probability distribution across both classes. It implicitly enforces that the probabilities sum to one, which can introduce subtle differences in the learning process.

For example, consider a scenario where you’re classifying images of cats and dogs. If you use binary_crossentropy, the model learns to predict the probability of an image being a cat (or, equivalently, not being a dog). If you use categorical_crossentropy, the model learns to predict the probability of an image being a cat and the probability of it being a dog, ensuring that these probabilities sum to 1. This seemingly small difference can lead to variations in the learned representations and, consequently, in performance. The featured snippet below highlights a key difference.

Featured Snippet: The core difference lies in how the output layer is handled. Binary crossentropy uses a single output neuron with a sigmoid activation, while categorical crossentropy uses a softmax activation across multiple neurons, even in the binary case, which forces the model to consider the probabilities of all classes simultaneously, impacting the learning process.

Choosing the Right Loss Function

Selecting the appropriate loss function is critical for achieving optimal results. If you have a true binary classification problem where each data point belongs to one of two mutually exclusive classes, binary_crossentropy is generally the preferred choice. It’s computationally efficient and directly models the probability of the positive class. However, if you have a multi-class classification problem, or if you want to treat a binary problem as a special case of a multi-class problem, categorical_crossentropy is the better option. This is especially true when you want to leverage the properties of the softmax activation function. Always ensure that your output layer and target variable encoding are consistent with the chosen loss function.

Here are some rules of thumb to guide your choice:

1. Binary Classification with Exclusive Classes: Use binary_crossentropy with a sigmoid activation.
2. Multi-Class Classification: Use categorical_crossentropy with a softmax activation.
3. Binary Classification Treated as Multi-Class: Use categorical_crossentropy with a softmax activation and one-hot encoded labels. This approach can be useful if you anticipate extending the problem to more classes in the future.

Additionally, consider the specific characteristics of your dataset. If your data is imbalanced (i.e., one class has significantly more samples than the other), you might need to use techniques like class weighting or oversampling to mitigate the impact of the imbalance on the loss function. You should also experiment with different learning rates and optimization algorithms to find the combination that yields the best performance for your chosen loss function. Remember that the best choice often depends on the specific details of your problem and requires empirical evaluation. You can find more in-depth explanations and examples on machine learning forums such as Stack Overflow [3].

When should I use binary crossentropy?

Use binary crossentropy for binary classification problems where each sample belongs to one of two mutually exclusive classes. The output layer should use a sigmoid activation function.

When should I use categorical crossentropy?

Use categorical crossentropy for multi-class classification problems where each sample belongs to one of several mutually exclusive classes. The output layer should use a softmax activation function.

Can I use categorical crossentropy for binary classification?

Yes, you can, but you need to treat the binary problem as a special case of multi-class classification. One-hot encode your target variables and use a softmax activation function in the output layer.

What is one-hot encoding?

One-hot encoding is a representation of categorical variables as binary vectors. For example, if you have three classes, class 0 would be represented as [1, 0, 0], class 1 as [0, 1, 0], and class 2 as [0, 0, 1].

• Binary crossentropy is efficient for binary problems.
• Categorical crossentropy handles multi-class scenarios effectively.

Understanding the nuances of loss functions is vital for successful deep learning projects. The subtle differences between binary_crossentropy and categorical_crossentropy can have a significant impact on your model’s performance. By carefully considering the nature of your classification problem, the encoding of your target variables, and the activation function of your output layer, you can make an informed decision that leads to better results. Remember to experiment and evaluate different options to find the optimal configuration for your specific task. This comprehensive understanding empowers you to build robust and accurate classification models. Now that you grasp the core differences, perhaps you’d like to explore related topics, such as different activation functions or optimization algorithms? Consider exploring advanced loss function techniques to further refine your model’s performance.

[1] Andrew Ng. “Deep Learning Specialization.” Coursera. [2] TensorFlow Documentation. “tf.keras.losses.CategoricalCrossentropy.” TensorFlow. [3] Stack Overflow. “Binary vs. Categorical Crossentropy: When to Use Which?” Stack Overflow.

Question & Answer :
I’m trying to train a CNN to categorize text by topic. When I use binary cross-entropy I get ~80% accuracy, with categorical cross-entropy I get ~50% accuracy.

I don’t understand why this is. It’s a multiclass problem, doesn’t that mean that I have to use categorical cross-entropy and that the results with binary cross-entropy are meaningless?

model.add(embedding_layer) model.add(Dropout(0.25)) # convolution layers model.add(Conv1D(nb_filter=32, filter_length=4, border_mode='valid', activation='relu')) model.add(MaxPooling1D(pool_length=2)) # dense layers model.add(Flatten()) model.add(Dense(256)) model.add(Dropout(0.25)) model.add(Activation('relu')) # output layer model.add(Dense(len(class_id_index))) model.add(Activation('softmax')) 

Then I compile it either it like this using categorical_crossentropy as the loss function:

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) 

or

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) 

Intuitively it makes sense why I’d want to use categorical cross-entropy, I don’t understand why I get good results with binary, and poor results with categorical.

The reason for this apparent performance discrepancy between categorical & binary cross entropy is what user xtof54 has already reported in his answer below, i.e.:

the accuracy computed with the Keras method evaluate is just plain wrong when using binary_crossentropy with more than 2 labels

I would like to elaborate more on this, demonstrate the actual underlying issue, explain it, and offer a remedy.

This behavior is not a bug; the underlying reason is a rather subtle & undocumented issue at how Keras actually guesses which accuracy to use, depending on the loss function you have selected, when you include simply metrics=['accuracy'] in your model compilation. In other words, while your first compilation option

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) 

is valid, your second one:

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) 

will not produce what you expect, but the reason is not the use of binary cross entropy (which, at least in principle, is an absolutely valid loss function).

Why is that? If you check the metrics source code, Keras does not define a single accuracy metric, but several different ones, among them binary_accuracy and categorical_accuracy. What happens under the hood is that, since you have selected binary cross entropy as your loss function and have not specified a particular accuracy metric, Keras (wrongly…) infers that you are interested in the binary_accuracy, and this is what it returns - while in fact you are interested in the categorical_accuracy.

Let’s verify that this is the case, using the MNIST CNN example in Keras, with the following modification:

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) # WRONG way model.fit(x_train, y_train, batch_size=batch_size, epochs=2, # only 2 epochs, for demonstration purposes verbose=1, validation_data=(x_test, y_test)) # Keras reported accuracy: score = model.evaluate(x_test, y_test, verbose=0) score[1] # 0.9975801164627075 # Actual accuracy calculated manually: import numpy as np y_pred = model.predict(x_test) acc = sum([np.argmax(y_test[i])==np.argmax(y_pred[i]) for i in range(10000)])/10000 acc # 0.98780000000000001 score[1]==acc # False 

To remedy this, i.e. to use indeed binary cross entropy as your loss function (as I said, nothing wrong with this, at least in principle) while still getting the categorical accuracy required by the problem at hand, you should ask explicitly for categorical_accuracy in the model compilation as follows:

from keras.metrics import categorical_accuracy model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[categorical_accuracy]) 

In the MNIST example, after training, scoring, and predicting the test set as I show above, the two metrics now are the same, as they should be:

# Keras reported accuracy: score = model.evaluate(x_test, y_test, verbose=0) score[1] # 0.98580000000000001 # Actual accuracy calculated manually: y_pred = model.predict(x_test) acc = sum([np.argmax(y_test[i])==np.argmax(y_pred[i]) for i in range(10000)])/10000 acc # 0.98580000000000001 score[1]==acc # True 

System setup:

Python version 3.5.3 Tensorflow version 1.2.1 Keras version 2.0.4 

UPDATE: After my post, I discovered that this issue had already been identified in this answer.