MRMR: the most Googled feature selection term. What is it?
MRMR: The Most Googled Feature Selection Method—Explained!
MRMR is a feature selection method used by statisticians in biochemistry and then popularized by Uber to select features for machine learning models.
MRMR stands for Minimum Redundancy Maximum Relevance.
The idea behind this method is to select features that have maximum relevance with the target, and minimum redundancy to other features in the dataset.
In laywoman terms: we want features that are highly related to the target but not related to other features in the dataset. And by related, we usually mean correlated.
The relation with the target is captured by the relevance. The relation to other features is captured by the redundancy. So, we need ways to quantify relevance and redundancy.
Relevance can be determined with statistical tests like ANOVA, correlation, mutual information, or by the feature importance from Random Forests. The first 2 methods are fast. The others, not so much.
Redundancy can be measured with correlation (fast) or mutual information (not so fast).
MRMR selects features with high relevance and low redundancy in an iterative process: at each round it ranks the features through the ratio or difference between their relevance and redundancy and then selects the top-ranking feature. It is iterative, because the redundancy is calculated between the features being examined and the ones already selected. As we add new features to the selected pot, we need to re-establish the mean redundancy of the remaining ones.
The popularity of MRMR was given by its speed. But to be fast, it uses classical statistical tests that measure linear associations. For non-linear associations, we need to use Random Forests and mutual information, and then the algorithm is not so fast anymore.
You can apply MRMR with Feature-engine: https://feature-engine.trainindata.com/en/latest/user_guide/selection/MRMR.html
If you’ve used this method, I want to hear about your experience.
Other than that, to learn more about ways to assign feature importance for selection check out:
🎓 Our course: Feature Selection for Machine Learning
📘 Our book: Feature Selection in Machine Learning
Now that we've covered MRMR, I’ve got some exciting news! Our Forecasting with Machine Learning course just got a code update. Keep reading for details! 👇
Code update for the course Forecasting with Machine Learning!
A few months back SKForecast made a major update to their API: briefly, they simplified their API, so that, with the same recursive or direct forecaster, you can now create both lag and window features.
Before, we needed to use separate forecasters if we wanted to combine lags and windows, or if we wanted to use just lags. That is no longer the case.
In addition, they included their own function to create the windows, so that we don’t need to write NumPy functions anymore.
And that’s not it. We can now extract the features that are being created for the forecasts directly out of the forecaster, so we can always be sure what goes in to make predictions.
That means that we had to update pretty much all the notebooks for the course. And guess what? We did.
So, if you are enrolled already, make sure to head to the repo and re-download the updated version of the code.
And if you haven’t enrolled yet, well, this is a great time to do so. Head over here and find out more about what you’ll learn in the course.
Ready to enhance your data science skills?
Stop aimless internet browsing. Start learning today with meticulously crafted courses offering a robust curriculum, fostering skill development with steadfast focus and efficiency.
Forecasting specialization (course)
Interpreting Machine Learning Models (course)