Regularization in Machine Learning: Ridge and Lasso Regression

What Is Regularization in Machine Learning?

Regularization in machine learning is the set of techniques that prevent a model from fitting training data so precisely that it fails on new data. Every time you build a predictive model, you face a core tension: the more perfectly the model explains what it has already seen, the less likely it is to explain what it has not. Regularization manages that tension directly, by adding a penalty to the model’s cost function that discourages excessively large coefficient values.

Think of it like this. A linear regression model fits a line through your data points by minimizing the sum of squared errors. Without any constraint, the model will chase every data point — including the noise. The result is a model that looks excellent on paper but collapses the moment it encounters real-world variation. This is overfitting. Regularization adds a term to the cost function that says: “yes, minimize your errors, but also keep your coefficients small.” That trade-off is the engine behind both Ridge regression and Lasso regression.

For students working on regression analysis assignments, understanding why regularization exists matters as much as knowing how to run it in Python. The concept sits at the intersection of statistics, optimization, and machine learning — and examiners test understanding, not just code output.

Why Does Overfitting Happen?

Overfitting happens when a model has enough parameters to memorize the training dataset rather than learning its underlying pattern. In regression, this shows up as wildly large positive and negative coefficients that cancel out in training but produce absurd predictions on new inputs. The more features your dataset has relative to the number of training observations, the more severe the problem becomes.

This is especially common in high-dimensional data — genomics datasets, text features, financial time series — where you might have thousands of predictors and only hundreds of observations. Simple linear regression collapses under those conditions. Regularization is the fix. It forces the model to be humble about its coefficients by making extreme values expensive.

The Bias-Variance Tradeoff: The Foundation of Regularization

Every regularization decision is, at its core, a decision about the bias-variance tradeoff. Bias measures how much your model systematically misses the true relationship in the data — a high-bias model is too simple and underfits. Variance measures how sensitive your model is to small changes in the training data — a high-variance model is too complex and overfits.

Regularization deliberately introduces a small amount of bias into the model in exchange for a larger reduction in variance. The result is a model that gives up a little accuracy on training data to gain substantially more accuracy on unseen data. This is the trade that makes regularization valuable, and it is why the assumptions of the regression model matter so much when choosing how aggressively to regularize.

According to research in the International Journal of Data Science and Analytics, Lasso and Ridge regression represent two of the most fundamental regularization techniques in modern statistical learning, with applications ranging from genomics to finance.

Ridge Regression: L2 Regularization Explained

Ridge regression is the version of regularized linear regression that adds the squared magnitude of each coefficient to the cost function. It is also called L2 regularization or Tikhonov regularization. The mathematician Andrey Tikhonov introduced the underlying mathematical framework in the 1940s, and it was later adapted into the statistical regression context. In machine learning, Ridge regression is one of the most practical tools for building stable, generalizable predictive models from correlated or noisy features.

The Ridge Regression Cost Function

Standard linear regression minimizes the sum of squared residuals between predictions and actual values. Ridge regression adds a penalty term equal to lambda times the sum of squared coefficients.

When λ = 0, Ridge regression reduces to ordinary least squares — no penalty, no regularization. As λ increases, the penalty on large coefficients grows, forcing the optimizer to shrink all coefficients toward zero. Crucially, Ridge rarely drives any coefficient exactly to zero. It distributes the penalty across all features, keeping every predictor in the model with a reduced weight.

What Ridge Does to Coefficients

This coefficient-shrinking behavior is exactly what you want when you believe most of your features are genuinely useful but their individual contributions need to be moderated. Ridge regression is particularly effective in the presence of multicollinearity — when two or more predictors are highly correlated with each other. In ordinary linear regression, multicollinearity causes coefficient estimates to become wildly unstable: tiny changes in the data can flip a coefficient from large-positive to large-negative. Ridge’s L2 penalty stabilizes this by spreading the weight more evenly across correlated predictors.

Students working on logistic regression and broader regression modeling will recognize this instability — it also shows up in classification settings where correlated features make coefficient interpretation unreliable. Ridge fixes it in both contexts.

Implementing Ridge Regression in Python

# Ridge Regression with scikit-learn
# Always scale features first — Ridge is sensitive to feature magnitude

from sklearn.linear_model import Ridge, RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

# 1. Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 2. Scale features (critical step — do NOT skip)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

# 3. Use cross-validation to find the best alpha (lambda)
alphas = np.logspace(-4, 4, 100)  # search from 0.0001 to 10000
ridge_cv = RidgeCV(alphas=alphas, cv=5)
ridge_cv.fit(X_train_sc, y_train)

print(f"Best alpha: {ridge_cv.best_alpha_:.4f}")

# 4. Fit the final model
ridge = Ridge(alpha=ridge_cv.best_alpha_)
ridge.fit(X_train_sc, y_train)
y_pred = ridge.predict(X_test_sc)

# 5. Evaluate
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")
print(f"R²:   {r2_score(y_test, y_pred):.4f}")
print(f"Coefficients: {ridge.coef_}")

When to Use Ridge Regression

• When you have many features and believe most of them carry some predictive signal
• When predictors are correlated and OLS coefficients are unstable or counterintuitive
• When you need a model that retains every feature but with moderated weights
• When your dataset has more features than observations (p > n problems)

Lasso Regression: L1 Regularization and Feature Selection

Lasso regression stands for Least Absolute Shrinkage and Selection Operator. The statistician Robert Tibshirani introduced it in 1996, and it has since become one of the most widely used tools in both statistics and machine learning. What makes Lasso unique — and often more useful than Ridge in high-dimensional settings — is its ability to drive some coefficients exactly to zero. That is not a side effect; it is the point. Lasso performs automatic feature selection as part of the fitting process.

The Lasso Cost Function

Lasso uses an L1 penalty rather than Ridge’s L2. Instead of penalizing the squared coefficient values, it penalizes their absolute values.

That difference — absolute value instead of square — has a profound geometric consequence. The feasible region created by the L1 constraint is shaped like a diamond (in 2D) or a hypercube in higher dimensions. The optimum often lands exactly at one of the diamond’s corners, where one or more coordinates are zero. That is why Lasso produces sparse solutions: many coefficients end up exactly at zero, meaning those features are fully removed from the model.

Why Lasso Eliminates Features (and Ridge Does Not)

This is one of the most commonly examined questions in machine learning courses. The geometric explanation is the most intuitive. In Ridge regression, the constraint region is a sphere — smooth, with no corners. The optimal solution for the loss function tends to graze the surface of the sphere at a point where all coefficients are small but non-zero. In Lasso, the constraint region has corners. The optimal solution is far more likely to land at a corner, where one or more coefficients are exactly zero.

For students studying factor analysis and data reduction techniques, Lasso’s sparsity-inducing property is particularly relevant — it is a form of dimensionality reduction built into the model fitting itself, rather than applied as a separate preprocessing step.

Implementing Lasso Regression in Python

# Lasso Regression with cross-validated alpha selection

from sklearn.linear_model import Lasso, LassoCV
from sklearn.preprocessing import StandardScaler
import numpy as np

# Scale first (mandatory)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

# LassoCV: tries many alphas with k-fold cross-validation
lasso_cv = LassoCV(alphas=np.logspace(-4, 2, 100),
                   cv=5,
                   max_iter=10000)  # increase max_iter for convergence
lasso_cv.fit(X_train_sc, y_train)

print(f"Best alpha: {lasso_cv.alpha_:.6f}")

# Fit the final Lasso model
lasso = Lasso(alpha=lasso_cv.alpha_, max_iter=10000)
lasso.fit(X_train_sc, y_train)

# Inspect which features survived (non-zero coefficients)
coefs = lasso.coef_
selected = np.where(coefs != 0)[0]
print(f"Features selected: {len(selected)} of {X_train.shape[1]}")
print(f"Non-zero coefficients: {coefs[selected]}")

When to Use Lasso Regression

• When you suspect many features are irrelevant or redundant noise
• When you need an interpretable model with fewer active predictors
• In high-dimensional settings (genomics, text mining, signal processing) where variable selection is essential
• When you want the model to tell you which features matter, rather than specifying this manually

Ridge vs Lasso Regression: The Full Comparison

The question “when do I use Ridge and when do I use Lasso” comes up on virtually every machine learning exam. The answer is not arbitrary — it follows directly from the mathematical difference between the two penalties and what that difference produces in practice. Understanding regularization in machine learning at a level that earns top marks means being able to explain that difference precisely, not just recite which penalty is L1 and which is L2.

The Geometric Intuition Behind the Difference

The most powerful way to understand why Ridge and Lasso behave so differently is geometrically. Think of the coefficient space. Ordinary least squares finds the point in that space that minimizes the sum of squared errors — often a point far from the origin with large coefficient values. Regularization constrains the solution to lie within a region around the origin.

For Ridge, that region is a sphere (or hypersphere in high dimensions). Smooth. Rounded. No edges. When the elliptical contours of the least-squares loss function intersect this sphere, they almost always do so at a point where every coefficient is small but non-zero.

For Lasso, the region is a diamond (or cross-polytope). It has corners, edges, and vertices. When the loss function’s elliptical contours intersect this shape, the corners are the likely intersection points. At the corners of the diamond, one or more coordinates are exactly zero. That is why Lasso produces sparse models. Geometry, not programming magic.

For a deeper mathematical treatment of this geometric interpretation, the classic text by Hastie, Tibshirani, and Friedman — The Elements of Statistical Learning — is the definitive scholarly reference. It is freely available from Stanford University and is cited in virtually every serious treatment of regularization.

Which Regularization Method Should You Choose?

This is a genuine question with a real answer. Start with your domain knowledge about the data. If you are working with a gene expression dataset with 20,000 features and expect that only a few hundred genes are actually relevant, Lasso is the right tool — it will find and keep those genes while zeroing everything else. If you are working with economic indicators where most macroeconomic variables contribute something and you want to stabilize correlated predictors, Ridge is correct.

When you genuinely do not know — which is most of the time in real data science work — use Elastic Net, which combines both penalties and lets the data find the right balance. More on that in the next section. For assignment purposes, always justify your choice with reference to the data’s characteristics, not just convention.

Elastic Net: When Ridge and Lasso Are Both Right

Elastic Net regularization was introduced by Hui Zou and Trevor Hastie in 2005 as a direct response to Lasso’s instability with correlated features. It combines both L1 and L2 penalties in a single cost function, with a mixing parameter that controls the balance between them. The result is a method that inherits Lasso’s feature selection capability while retaining Ridge’s stability under correlation. In practice, Elastic Net outperforms either method alone on many real-world datasets.

The Elastic Net Cost Function

Elastic Net has two hyperparameters: the overall regularization strength lambda and the mixing parameter rho. When rho equals 1, Elastic Net is pure Lasso. When rho equals 0, it is pure Ridge. In between, it combines both. Finding the right combination requires cross-validation across a grid of both parameters — scikit-learn’s ElasticNetCV handles this automatically.

Implementing Elastic Net in Python

from sklearn.linear_model import ElasticNet, ElasticNetCV
import numpy as np

# Cross-validate both alpha and l1_ratio simultaneously
en_cv = ElasticNetCV(
    l1_ratio=[.1, .5, .7, .9, .95, .99, 1],  # test multiple mixes
    alphas=np.logspace(-4, 2, 100),
    cv=5,
    max_iter=10000
)
en_cv.fit(X_train_sc, y_train)

print(f"Best alpha:    {en_cv.alpha_:.6f}")
print(f"Best l1_ratio: {en_cv.l1_ratio_:.2f}")

# Fit final model
en = ElasticNet(alpha=en_cv.alpha_, l1_ratio=en_cv.l1_ratio_, max_iter=10000)
en.fit(X_train_sc, y_train)

# Count surviving features
n_selected = np.sum(en.coef_ != 0)
print(f"{n_selected} features selected out of {X_train.shape[1]}")

When Elastic Net Is the Right Choice

• When you have groups of correlated features and need some of them (not arbitrary selection between them)
• When Lasso selects too few features and Ridge selects too many for your interpretation needs
• In genomics, text classification, and finance — domains with known feature correlation structures
• When you do not have strong prior knowledge about the data’s sparsity level

Elastic Net has become the default regularization method in many modern machine learning pipelines precisely because it degenerates to either Ridge or Lasso when that is what the data calls for — while covering the space between. Students working on cross-validation and bootstrapping assignments will find Elastic Net’s dual hyperparameter grid search a natural extension of the concepts they are already applying.

How to Choose Lambda: Regularization Strength Tuning

Knowing that regularization in machine learning works is not the same as knowing how much regularization to apply. Lambda — called alpha in scikit-learn — is the hyperparameter that controls regularization strength. Set it too low and the model still overfits. Set it too high and the model underfits, ignoring genuine signal. The only defensible way to choose lambda is cross-validation.

What Cross-Validation Does for Lambda

Cross-validation repeatedly splits the training data into smaller train and validation folds, fits the model with a given lambda on the training fold, and measures performance on the validation fold. Repeating this across many lambda values and averaging the validation errors produces a curve. The lambda at the bottom of that curve — the one that minimizes average validation error — is the one you keep.

For students learning hypothesis testing and model evaluation, the cross-validation curve is a direct diagnostic tool: the left side of the curve shows overfitting (low lambda, high variance), and the right side shows underfitting (high lambda, high bias). The optimal lambda sits in the valley between them.

import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LassoCV

alphas = np.logspace(-4, 2, 200)
lasso_cv = LassoCV(alphas=alphas, cv=5, max_iter=10000)
lasso_cv.fit(X_train_sc, y_train)

# Mean CV error per alpha
mean_errors = lasso_cv.mse_path_.mean(axis=1)

plt.figure(figsize=(9, 5))
plt.semilogx(lasso_cv.alphas_, mean_errors, color='#2563EB', lw=2)
plt.axvline(lasso_cv.alpha_, color='#AA4646', ls='--', label=f'Best α = {lasso_cv.alpha_:.4f}')
plt.xlabel('log(Alpha)')
plt.ylabel('Mean CV MSE')
plt.title('Lasso Cross-Validation: Alpha Selection')
plt.legend()
plt.tight_layout()
plt.show()

The Regularization Path: Watching Coefficients Shrink

A regularization path plot shows what happens to every coefficient as lambda increases from zero. For Lasso, you will see coefficients start at their OLS values, then sequentially drop to zero as lambda grows — different features disappear at different lambda values. This visualization is one of the most pedagogically powerful tools in all of machine learning: it makes the feature selection process visible.

According to research by Friedman, Hastie, and Tibshirani (Journal of Statistical Software, 2010), coordinate descent algorithms — the backbone of Lasso and Elastic Net fitting — make regularization path computation fast enough to be practical even for very large datasets, which was a key bottleneck before their work.

Where Regularization in Machine Learning Is Applied

Regularization is not an academic exercise. It appears in production systems across virtually every industry that uses predictive modeling. Understanding where these techniques are applied in the real world helps students connect coursework to practice — and gives them the vocabulary to discuss their knowledge in job interviews and graduate school applications.

Genomics and Bioinformatics

Lasso’s feature selection capability was arguably made for genomics. Gene expression studies routinely involve datasets with tens of thousands of features (genes) and hundreds or fewer observations (patients or samples). Running standard regression on such data is mathematically impossible — the system is massively underdetermined. Lasso identifies the small subset of genes that are actually predictive of a clinical outcome, while setting the remaining thousands of genes to zero. This is how researchers at institutions like the Broad Institute of MIT and Harvard and the Wellcome Sanger Institute in the UK use regularization to identify disease biomarkers from genome-wide association study data.

Financial Modeling and Risk

In finance, regularization addresses the challenge of building return prediction models from large sets of economic indicators. Ridge regression is particularly common in this context because many macroeconomic variables are correlated — inflation, interest rates, growth — and you want all of them in the model with stabilized weights rather than arbitrarily dropping some. Hedge funds and investment banks use Ridge-regularized factor models to predict asset returns while avoiding the instability that plagues unregularized regression on correlated financial data. Students working on data science assignments in finance will frequently encounter these applications.

Natural Language Processing and Text Classification

When you represent text documents as bag-of-words vectors, each document becomes a high-dimensional sparse vector — potentially with tens of thousands of word features, most of them zero for any given document. Lasso regression (and logistic regression with L1 penalty) naturally handles this by selecting the small subset of words that are predictive of a category. Spam detection, sentiment classification, and topic categorization all use this approach. The resulting models are not just accurate — they are interpretable, showing which specific words drive each prediction.

Medical and Clinical Prediction

Clinicians building predictive models for patient outcomes face the same high-dimensionality problem as genomics: many potential predictors, limited observations, and serious consequences for overfitting. A model that overfits training data will fail on the patients it is supposed to help. Regularization — particularly Elastic Net — is standard practice in clinical risk score development. The LASSO-Cox model, which applies L1 regularization to survival analysis, is widely used in oncology research to select prognostic biomarkers from panels of clinical and molecular features. For students in survival analysis, this connection is particularly direct.

Computer Vision and Signal Processing

Compressed sensing — a framework for reconstructing signals from fewer measurements than classical theory requires — depends entirely on sparsity-inducing regularization. Lasso’s L1 penalty is the mathematical engine behind many compressed sensing algorithms. This is why high-quality MRI scans can be reconstructed from far fewer measurements than the Nyquist theorem would suggest: the underlying signal is sparse in some representation, and L1 minimization finds it. JPEG image compression also exploits sparsity — and while the specific algorithm differs, the same mathematical intuition applies.

Assumptions, Common Mistakes, and Model Diagnostics

Understanding regularization in machine learning means knowing not just how to implement it, but where it can go wrong. Ridge and Lasso regression are built on several assumptions, and violating them produces misleading results even when the code runs without errors.

Assumptions Underlying Ridge and Lasso

Both Ridge and Lasso extend linear regression — so they inherit its core assumptions. The relationship between features and target should be approximately linear. Residuals should be independently distributed. The response variable should be continuous (for regression; for classification, use regularized logistic regression). Additionally, because both methods are sensitive to feature scale, feature standardization is a required preprocessing step, not an optional one. This is not an assumption of the math — it is a practical requirement for the penalty to be applied fairly across features with different units.

For a full treatment of regression model assumptions and how to test them, the guide to regression model assumptions provides diagnostic tests including residual plots, VIF scores for multicollinearity, and normality checks — all directly relevant to validating Ridge and Lasso models.

Common Mistakes Students Make

What Is Data Leakage and Why Does It Destroy Regularization?

Data leakage occurs when information from outside the training set bleeds into the model fitting process, producing artificially good validation metrics that do not reflect real-world performance. In the context of regularization, the most common leakage is fitting the StandardScaler on the full dataset (train + test) before splitting. The scaler then uses test data’s mean and variance in normalizing the training data, creating a subtle dependency. In production, this means the model’s reported performance cannot be replicated. Always fit preprocessing steps on training data only.

Students studying model selection with AIC and BIC will find that these information criteria can also be applied to regularized models — though cross-validation remains the most direct approach for selecting lambda.

Model Diagnostics After Fitting

After fitting a regularized model, always check these diagnostics. Inspect the residual plot against fitted values — systematic patterns indicate model misspecification that regularization cannot fix. Compare training and test error: if training error is much lower than test error, increase lambda. If both are high, decrease lambda or add features. For Lasso, examine how many features were retained — if Lasso kept almost all features, it is behaving like Ridge, which might suggest your data is not sparse and you should use Ridge or Elastic Net directly.

Advanced Regularization: Beyond Ridge and Lasso

Once you have solid command of Ridge, Lasso, and Elastic Net, several natural extensions deepen your understanding of regularization in machine learning. These advanced topics appear in graduate-level machine learning courses at universities like MIT, Stanford, Carnegie Mellon, and University College London, and they are increasingly tested in technical interviews at machine learning-intensive companies.

Adaptive Lasso

Standard Lasso applies the same penalty to all coefficients. But what if some features are more likely to be informative than others? The Adaptive Lasso assigns different weights to each coefficient’s penalty, using initial estimates (typically from OLS or Ridge) to determine those weights. Features with initially large coefficients receive a smaller adaptive penalty — making them harder to eliminate — while features with small initial estimates receive a larger penalty, making elimination more likely. This produces an “oracle property”: under certain conditions, Adaptive Lasso selects exactly the right set of features asymptotically.

Group Lasso

In many real problems, features come in natural groups. Think of categorical variables encoded as one-hot dummies, or sets of features derived from the same measurement instrument. Standard Lasso might select some dummies from a group and not others — producing nonsensical models. Group Lasso applies the sparsity-inducing L1 penalty at the group level rather than the individual feature level. Entire groups either enter the model together or are dropped together. This is widely used in genetic association studies where groups of SNPs on the same gene should be treated as a unit.

Regularization in Neural Networks

The principles of Ridge and Lasso extend naturally to neural networks. L2 weight decay in neural network optimization is mathematically identical to Ridge regression’s L2 penalty — it shrinks all weights toward zero during training. L1 weight decay is the neural network analogue of Lasso. Beyond direct penalties, neural networks use additional regularization strategies: dropout (randomly zeroing units during training), early stopping (halting training when validation error stops improving), and batch normalization (normalizing layer activations). All of these serve the same purpose as Ridge and Lasso: preventing overfitting by discouraging overly complex weight configurations.

Students exploring Principal Component Analysis will find that PCA offers an alternative approach to the same high-dimensionality problem that regularization addresses — by reducing the number of features before modeling, rather than penalizing their coefficients during modeling.

Bayesian Interpretation of Regularization

There is a profound connection between regularization and Bayesian statistics. Ridge regression corresponds to maximum a posteriori (MAP) estimation with a Gaussian prior on the coefficients. Lasso corresponds to MAP estimation with a Laplace (double exponential) prior. The width of the prior determines the regularization strength: a tight prior produces strong regularization, equivalent to a large lambda. This Bayesian view explains why regularization “works” — it incorporates prior knowledge that real-world coefficients tend to be small, and updates that prior with the evidence in the data.

For students working on Markov Chain Monte Carlo and Bayesian methods, this connection is the bridge between the frequentist regularization framework covered here and the fully Bayesian regression literature. The research published in the International Journal of Data Science and Analytics (2025) covers these extensions and their theoretical underpinnings in detail.

Regularization at Scale: SGD and Production Systems

For datasets with millions of rows, fitting Ridge and Lasso with their standard solvers becomes computationally expensive. The solution is stochastic gradient descent with a regularization penalty. Scikit-learn’s SGDRegressor with penalty='l2', 'l1', or 'elasticnet' implements exactly this. It processes mini-batches rather than the full dataset at each iteration, scaling linearly with data size. The resulting models are mathematically equivalent to Ridge and Lasso but trainable on datasets that would not fit in memory for standard solvers. Deploying these models in production is extremely fast: inference is a single dot product — O(p) per prediction — making them viable at millions of predictions per second.

Complete Step-by-Step Workflow: Ridge and Lasso in Practice

Theory understood — now execute. The following end-to-end workflow brings together everything covered in this guide into a reproducible process that applies to any regression problem where regularization in machine learning is appropriate. This is also the structure to follow when completing machine learning assignments and course projects.

Related Machine Learning Concepts That Deepen Your Understanding

Regularization in machine learning does not exist in isolation. The following concepts are directly connected to Ridge and Lasso regression — understanding them will strengthen your work on any assignment or exam that touches this topic.

Multicollinearity and Variance Inflation Factor (VIF)

Multicollinearity — when two or more features are linearly correlated — causes ordinary least squares to produce unstable, high-variance coefficient estimates. VIF measures how much the variance of each coefficient is inflated by collinearity. VIF values above 5 to 10 indicate problematic collinearity. Ridge regression is the standard remedy: it stabilizes coefficients by spreading the weight across correlated features rather than assigning arbitrarily large weights to one and near-zero to another. Checking VIF before deciding between Ridge and Lasso is good analytical practice.

Feature Scaling and Normalization

Ridge and Lasso penalize coefficient magnitude. Since coefficient magnitude depends heavily on the scale of the feature it is attached to, unscaled features are not comparable. A feature measured in meters will have a smaller coefficient than the same information measured in millimeters — not because it is less important, but because the units are larger. Standardization (zero mean, unit variance) levels the playing field. This is why understanding data statistics including means and standard deviations is a practical prerequisite for regularized regression.

Cross-Validation and Model Selection

K-fold cross-validation is the gold standard for selecting regularization hyperparameters. It provides an unbiased estimate of generalization error at each lambda value without touching the test set. Five-fold and ten-fold cross-validation are most common. Nested cross-validation — where an outer loop evaluates the final model and an inner loop selects hyperparameters — provides the most unbiased performance estimate but is computationally expensive. For assignment purposes, five-fold with a held-out test set is the expected standard in most university data science programs.

Polynomial Regression and Regularization Interactions

When you add polynomial features to a regression model, the feature space explodes — a degree-3 polynomial of 10 original features generates hundreds of transformed features. Without regularization, this almost always overfits. With Ridge or Lasso, you can fit high-degree polynomial models that still generalize. Polynomial regression combined with regularization is a powerful, flexible modeling strategy that many machine learning courses cover together precisely because the pairing makes polynomial models practical.

Logistic Regression with Regularization

Everything discussed in this guide applies to classification as well. Regularized logistic regression uses the same L1 and L2 penalties on the log-odds coefficients. In scikit-learn’s LogisticRegression class, the regularization parameter is C (the inverse of lambda — higher C means less regularization, the opposite of Ridge and Lasso convention). The penalty parameter accepts ‘l1’, ‘l2’, or ‘elasticnet’. Feature selection with L1 logistic regression is standard practice in clinical risk score development and text classification. For comprehensive coverage, the logistic regression guide covers both the unregularized and regularized variants in detail.

Frequently Asked Questions About Ridge and Lasso Regression