Introduction

Perhaps one of the most underrated yet powerful features that scikit-learn has to offer, pipelines are a great ally for building effective and modular machine learning workflows. They streamline the entire process — from data preparation and feature engineering to modeling, fine-tuning, and validation — while mitigating the risk of data leakage, making the code reproducible, and keeping it cleaner and easier to maintain.

In this article, we describe and exemplify — through concise but intermediate- to advanced-level use cases — five pipeline tricks to level up your ongoing machine learning projects.

Initial Setup

The following code and its elements will be used in several of the examples listed later on; therefore, applying these preparatory steps first is advisable. Note that we will predominantly use the popular Titanic Survivorship dataset hereinafter:

Time to get started with the core of this hands-on article! 

1. ColumnTransformer to Handle Mixed Data Types

In the first example, we will instantiate a ColumnTransformer object to define a robust data preprocessing pipeline. This class allows different transformations to be flexibly applied to different feature subsets in a unified manner, facilitating the processing of mixed data types and missing values without burdensome operations or repetitive code. 

This approach jointly processes numerical features, categorical ones, and missing values, integrating them into an overall pipeline before training the logistic regression classifier.

2. Feature Engineering with a Custom Transformer

Custom transformers in scikit-learn allow us to define our own feature-level transformation steps (be it feature engineering or preprocessing) and inject them directly into a pipeline. One example is TransformerMixin, which requires defining our own fit and transform methods but, in exchange, allows us to use fit_transform() seamlessly.

The following code defines a custom transformer class through inheritance. It applies its logic to map the “age” numerical feature into binary (0 or 1) values indicating whether the passenger is an adult. We then incorporate this custom transformer logic into a ColumnTransformer like the one defined in the previous use case.

3. Hyperparameter Tuning Across the Entire Pipeline

This example demonstrates that hyperparameter tuning—finding the best configuration among many options—is not exclusively related to the machine learning model’s settings. It can also apply to choices made in previous preprocessing steps, as shown in this interesting example:

Note that preprocessor is the preprocessing pipeline defined in example 1, i.e. a ColumnTransformer. The key in this example is the addition of a hyperparameter to the search grid that is related to a preprocessing step, namely the missing value imputation strategy. In other words, multiple model versions are trained not only based on hyperparameters of the model itself but also regarding specific settings in the preprocessing steps.

4. Integrating Feature Selection into a Pipeline

Another powerful technique, especially for datasets with many features, is to dynamically perform feature selection within the pipeline to keep the final model simpler.

This example automatically selects the most informative preprocessed features before training the model by incorporating the call to the SelectKBest class to select the highest-scoring features into the overarching pipeline (once again, we use the same preprocessor instance defined in earlier examples):

The SelectKBest class requires a scoring function or criterion to determine the top-k features to retain for model training. Another possible argument for it could be score_func=chi2, which applies a Chi-squared test to select features and is useful when categorical features dominate.

5. Stacked Pipelines

Our last example shows how to stack multiple pipelines for building an ensemble machine learning solution. Pipelines are a great way to design our own highly customizable ensembles, in cases where different models, sometimes with distinct preprocessing steps, need to be trained and combined without risking data management inconsistencies.

In the example below, two “overarching” pipelines are defined: one to preprocess the data and train a logistic regression classifier, and the other to apply the same preprocessing (for simplicity, it could have been different) but train a decision tree instead.

The two pipelines are then stacked using the StackingClassifier class. This class uses a final estimator to learn the best way to combine the base models’ predictions, yielding a stronger, more generalizable model.

Wrapping Up

This article showed five insightful examples of what we can do with scikit-learn pipelines to turbocharge and make our machine learning workflows more effective, customizable, and, in some cases, better-performing. From custom preprocessing pipelines for mixed data types to extending hyperparameter tuning to preprocessing steps, we revealed several tricks and hacks to take your machine learning modeling projects to the next level.

No comments yet.