extracted.txt

# **ML Feature Selection Study**
Comparing RFE vs SHAP

## Objective
The goal of this notebook is to compare two popular feature selection methods:
- Recursive Feature Elimination (RFE)
- SHAP-based Feature Importance

We evaluate which method selects the most effective feature set based on:
- Model performance
- Feature interpretability
- Stability across methods

# **Imports & Setup**

In this cell, we import all the required libraries for:
- Data handling (pandas, numpy)
- Visualization (matplotlib, seaborn)
- Machine learning (scikit-learn)
- Model explainability (SHAP)

This setup ensures we can:
- Train ML models
- Perform feature selection
- Visualize feature importance
- Compare model performance


# **Dataset Loading**

## Dataset
Using the Breast Cancer Wisconsin dataset for binary classification.

### Dataset Loading

We use the **Breast Cancer Wisconsin Dataset**, a standard binary classification dataset.
- Target: Malignant (1) vs Benign (0)
- Features: 30 numeric medical measurements

This dataset is ideal for feature selection experiments because:
- It has many correlated features
- Interpretability is important in healthcare

# **Train–Test Split & Scaling**

### Train–Test Split

The dataset is split into:
- 80% training data
- 20% testing data

This ensures:
- The model is trained on unseen data
- Performance evaluation is unbiased

### Feature Scaling

We apply **StandardScaler** to normalize features.

Why scaling is important:
- Logistic Regression is sensitive to feature magnitude
- Feature selection methods like RFE work better with scaled data

# **Baseline Model (No Feature Selection)**

### Baseline Model (No Feature Selection)

We train a Logistic Regression model using **all features**.
This acts as a reference point to compare:
- RFE-selected features
- SHAP-selected features

Baseline accuracy shows the maximum achievable performance
before feature reduction.

# **Output**

- Baseline Accuracy: 0.973684

- Indicates a strong initial model

# **Feature Selection Using RFE**

Recursive Feature Elimination

### Feature Selection Using RFE

Recursive Feature Elimination (RFE):
- Starts with all features
- Iteratively removes the least important ones
- Retains the top 10 features based on model coefficients

# **Goal**:
Reduce feature count without losing accuracy.


# **Output**

- List of 10 selected features

- Mostly “worst” and “error” related medical measurements


# **RFE Ranking Visualization**

### RFE Feature Ranking Visualization

This bar chart shows the **top 10 features selected by RFE**.

Why this visualization matters:
- Confirms which features the model considers most important
- Helps validate feature selection logic visually

# **Output**

- Bar plot of top 10 RFE features

- Lower rank = higher importance

# **Model Performance with RFE Features**

### Model Performance with RFE Features

We retrain the model using only the 10 RFE-selected features.

Purpose:
- Measure whether performance drops after feature reduction
- Evaluate efficiency vs accuracy tradeoff

# **Output**

RFE Accuracy: 0.973684

Same as baseline despite using fewer features

# **Observation**:
RFE successfully removed redundant features without hurting performance.

# **Feature Selection Using SHAP**

SHAP Explainer

### SHAP Explainer

SHAP (SHapley Additive exPlanations):
- Explains how each feature contributes to predictions
- Based on cooperative game theory
- Focuses on interpretability, not just ranking

We compute SHAP values for the baseline model.

# **Output**

- SHAP values generated internally

- No printed output

# **SHAP Summary Plot**

Why this matters:
SHAP shows how much each feature contributes to predictions—not just rankings.

### SHAP Summary Plot (Global Importance)

This bar plot shows:
- Average absolute contribution of each feature
- Global feature importance across the dataset

Why this matters:
- Explains *why* the model makes decisions
- Useful for regulated or trust-critical domains

# **Output**

- SHAP bar plot showing most influential features

# **Select Top SHAP Features**

### Selecting Top SHAP Features

We select the top 10 features based on:
- Mean absolute SHAP value

These features have the strongest global impact
on model predictions.

# **Output**

- List of top 10 SHAP-selected features

# **Model Performance with SHAP Features**

### Model Performance with SHAP Features

We retrain the model using only SHAP-selected features.

This evaluates:
- Performance after prioritizing interpretability
- Accuracy vs explainability tradeoff

# **Output**

- SHAP Accuracy: 0.964912

# **Observation:**
Slight accuracy drop is expected because SHAP favors stability and interpretability over aggressive optimization.

# **Final Comparison**

## Accuracy Comparison


### Final Accuracy Comparison

This table compares:
- Baseline model
- RFE-based model
- SHAP-based model

It summarizes the impact of feature selection methods.


# **Accuracy Bar Chart**

### Accuracy Comparison Visualization

This bar chart visually compares model accuracy
across different feature selection methods.

Helps quickly communicate results to:
- Recruiters
- Stakeholders
- Reviewers

# **Output**

- Bar chart comparing Baseline, RFE, and SHAP accuracy

# **Final Conclusion**


## Conclusion

- RFE performs well when we want a compact, performance-driven feature set.
- SHAP provides better interpretability and explains feature contribution clearly.
- SHAP-selected features often match domain intuition.
- Best choice depends on:
  - Performance focus → RFE
  - Explainability & trust → SHAP


# **End**