Discriminative Learning

Why Conditional Log-Likelihood outperforms standard MLE for Bayesian Network classifiers

Generative vs. discriminative learning

Standard Bayesian Network parameter learning uses Maximum Likelihood Estimation (MLE), which maximizes the joint log-likelihood of the training data. This is the generative objective — it models how data is jointly generated across all variables.

For classification, however, we care about predicting the class Y given features X. Discriminative learning instead maximizes the Conditional Log-Likelihood (CLL):

CLL(θ) = Σᵢ log P(yᵢ | xᵢ, θ)

This directly measures how well the model classifies each observation.

Why MLE falls short for classifiers

Any Bayesian Network structure encodes conditional independence assumptions. When those assumptions don’t hold perfectly in the data, MLE distributes the modeling error across all parameters — including parameters irrelevant to classification. The classification dimension of this error is not minimized.

By optimizing CLL, dplbnDE focuses all parameter adjustments on reducing classification error, even when independence assumptions are violated.

The best generative model (highest joint likelihood) is not necessarily the best classifier. Discriminative learning trades some generative accuracy for better classification performance.

The optimization problem

Given a fixed BN structure, dplbnDE searches for CPT parameters θ that maximize CLL subject to:

• Each CPT row sums to 1: Σⱼ θⱼ = 1
• All parameters are valid probabilities: θⱼ ∈ [0, 1]

Differential Evolution handles this constrained continuous optimization. The reflect function repairs out-of-bounds values, and keepSumFast renormalizes CPT rows after each mutation.

References

• Price, K., Storn, R. (1996). Minimizing the real functions of the ICSI’95 contest by differential evolution. IEEE ICEC. doi:10.1109/ICEC.1996.542711
• Zhang, J., Sanderson, A. (2009). JADE: Adaptive differential evolution with optional external archive. IEEE TEVC. doi:10.1109/TEVC.2009.2014613
• Tanabe, R., Fukunaga, A. (2014). Improving the search performance of SHADE using linear population size reduction. IEEE CEC. doi:10.1109/CEC.2014.6900380