ML Interview Q Series: How do the Softmax function and the Sigmoid function differ from each other?

📚 Browse the full ML Interview series here.

Comprehensive Explanation

Sigmoid and Softmax are two commonly used activation functions in machine learning, especially in classification tasks. They can appear superficially similar, but they serve different purposes and are applied in distinct scenarios. Understanding their forms, uses, and limitations is key to designing neural networks that fit the underlying data and problem requirements.

Sigmoid Function

The Sigmoid activation function typically takes a scalar input z and outputs a value in the interval (0, 1). It is often applied to binary classification tasks, where the output can be interpreted as the probability of belonging to a particular class.

Above, z is a real-valued number (could be the logit from a neuron). The exponential term e^{-z} represents how negative or positive z is. Larger positive values of z push the output closer to 1, and large negative values push it closer to 0.

Sigmoid is commonly used:

In binary classification for the final output neuron.

In certain hidden layers for squashing values, though less popular now compared to ReLU due to vanishing gradient issues.

Softmax Function

The Softmax activation function is typically used when dealing with multi-class problems in which you want a probability distribution over multiple mutually exclusive classes. It takes a vector of real numbers z with components z_i, and transforms them into probabilities that sum to 1.

Here, z_i denotes the logit (the raw score) for the i-th class out of K possible classes. The numerator e^{z_i} grows or shrinks depending on z_i’s magnitude. The denominator is the sum of exponentials of all logits e^{z_j} for j=1 to K. As a result, each softmax output is constrained to the interval (0, 1), and the sum of all K outputs is 1, making it a well-defined probability distribution for classes that are mutually exclusive.

Softmax is typically used:

In multi-class single-label classification tasks.

At the final layer of a neural network to get class probabilities.

Key Differences

In binary classification scenarios, Sigmoid is used to output a single probability for the positive class. Meanwhile, Softmax generalizes this idea to multiple classes by ensuring the outputs sum up to 1 and individually represent the probability of each class.

Softmax is used when exactly one class can be correct, because it forces the probability distribution to be exclusive, while Sigmoid can be used for multi-label classification scenarios where each label is treated independently.

Sigmoid ranges from (0, 1) for each input dimension independently, but does not enforce that different dimensions sum to 1. Softmax explicitly divides each exponentiated logit by the sum of all exponentiated logits, forming a proper probability distribution across classes.

Softmax can be more appropriate for multi-class tasks because it captures relative probabilities across classes. Sigmoid treats each output dimension independently and works best for tasks like multi-label classification where each label is a separate yes/no decision.

Why can't we just use Sigmoid for multi-class single-label classification?

Sigmoid is generally not used for standard multi-class single-label classification because each class probability would be predicted independently, without normalization across the classes. This makes it difficult to interpret or compare probabilities across classes. You could end up with all outputs > 0.5 or < 0.5, and you wouldn’t have a distribution that sums to 1. In contrast, Softmax ensures a proper normalized distribution, which is more meaningful for tasks where the model is supposed to pick exactly one class.

What if we need multi-label classification with multiple correct labels?

In a multi-label scenario, each label is treated as an independent yes/no classification. For that purpose, Sigmoid can be applied to each output unit independently, allowing each label to have a separate probability. Softmax would not be appropriate for multi-label classification, because softmax assumes the classes are exclusive and the probabilities sum to 1, which would contradict the possibility that multiple labels could all be valid at once.

How do we handle numerical instability with Softmax?

When the values of z_i in the softmax function become large in magnitude, the exponential computations can cause overflow. A common way to mitigate this is to subtract the maximum logit (say max_z) from all logits before exponentiation. Concretely, instead of e^{z_i}, we compute e^{z_i - max_z} in both numerator and denominator. This ensures stability because subtracting max_z keeps the exponent close to zero or negative, reducing the chance of overflow.

Can Sigmoid and Softmax appear in the same model?

It is possible to have a deep neural network where internal or auxiliary heads use Sigmoid (for something like multi-label aspects) and the main classification head uses Softmax (for a multi-class classification). The architecture design depends on how the problem is structured. For instance, some complex tasks might have a multi-class single-label classification component alongside separate binary properties being predicted.

How can we implement Sigmoid and Softmax in frameworks like PyTorch or TensorFlow?

In PyTorch, you can use torch.nn.Sigmoid or torch.sigmoid(x) for the Sigmoid function, and torch.nn.Softmax or torch.softmax(x, dim) for the Softmax function. In TensorFlow, tf.nn.sigmoid(x) applies the Sigmoid function elementwise, and tf.nn.softmax(x) applies the Softmax across a given dimension.

import torch
import torch.nn as nn
import torch.nn.functional as F

x = torch.tensor([2.0, 1.0, 0.1])

sigmoid_output = torch.sigmoid(x)     # Per-element Sigmoid
softmax_output = F.softmax(x, dim=0)  # Softmax across a single dimension

print("Sigmoid output:", sigmoid_output)
print("Softmax output:", softmax_output)

Both of these functions are extremely common in building neural network layers for classification tasks, and learning which one to pick depends on the nature of the classification: binary or multi-class.