In [1]:
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
# Import dataset
# dataset from: https://www.kaggle.com/datasets/shantanugarg274/heart-prediction-dataset-quantum
df = pd.read_csv("2_HeartPredictionDataset.csv")
#print(df.head())
# count NaN values => No NaN values
print(f"Counting NaN values:\n"
f"{df.isna().sum()}\n")
# count duplicates => No duplicates
print(f"Counting duplicates: {df.duplicated().sum()}\n")
# Detect and handle outliers using IQR (Interquartile Range)
# A value is considered an outlier if it lies outside the range [Q1 − 1.5 × IQR, Q3 + 1.5 × IQR]
# IQR represents the middle 50% of the data. Values beyond this are considered unusually high/low.
# Mild outliers = 1.5 x IQR
# Extreme outliers = 3 x IQR
q1 = df.quantile(0.25)
q3 = df.quantile(0.75)
iqr = q3 - q1
# Count rows & columns
print(f"Counting data in dataset: {df.shape[0]}")
# Remove outliers
df_no_outliers = df[~((df < (q1 - 1.5 * iqr)) | (df > (q3 + 1.5 * iqr))).any(axis=1)]
# See whether outliers have been removed => No outliers
print(f"Counting data in dataset after outlier-removal: {df_no_outliers.shape[0]}\n")
# Split data into features (X) and target (y)
X = df_no_outliers.drop(columns='HeartDisease') # features
y = df_no_outliers['HeartDisease'] # target
# Endcoding categorical values => Gender & HeartDisease already encoded (0 / 1).
# Rescale data between 0 and 1.
# MinMax scaling transforms the data => values of each feature (column) are scaled to fit withing a range of 0 to 1.
# Why: Because we use a distance-based model for classification. The features Age, BloodPressure, Cholesterol, and HeartRate
# have different ranges => Scaling/Standardization helps the models that rely on distance (or gradient-based optimization).
#Fit and transform feature data
x_scaled = MinMaxScaler().fit_transform(X)
# Convert result back into dataframe with same column names
x_scaled = pd.DataFrame(x_scaled, columns=X.columns)
# Create correlation matrix for feature selection (highly correlated features) => No highly correlated features
correlation_matrix = df.corr()
upper_triangle = correlation_matrix.where(abs(correlation_matrix) > 0.9)
print(f"Searching for highly correlated features (which should be removed for better performance):\n"
f"{upper_triangle}")
# Split data into training- and test sets
x_train, x_test, y_train, y_test = train_test_split(x_scaled, y, test_size=0.2, random_state=42)
Counting NaN values:
Age 0
Gender 0
BloodPressure 0
Cholesterol 0
HeartRate 0
QuantumPatternFeature 0
HeartDisease 0
dtype: int64
Counting duplicates: 0
Counting data in dataset: 500
Counting data in dataset after outlier-removal: 500
Searching for highly correlated features (which should be removed for better performance):
Age Gender BloodPressure Cholesterol HeartRate \
Age 1.0 NaN NaN NaN NaN
Gender NaN 1.0 NaN NaN NaN
BloodPressure NaN NaN 1.0 NaN NaN
Cholesterol NaN NaN NaN 1.0 NaN
HeartRate NaN NaN NaN NaN 1.0
QuantumPatternFeature NaN NaN NaN NaN NaN
HeartDisease NaN NaN NaN NaN NaN
QuantumPatternFeature HeartDisease
Age NaN NaN
Gender NaN NaN
BloodPressure NaN NaN
Cholesterol NaN NaN
HeartRate NaN NaN
QuantumPatternFeature 1.0 NaN
HeartDisease NaN 1.0
- Define function to evaluate performance of model
In [2]:
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, matthews_corrcoef
performance_results = []
def evaluate_model(model, model_name, x_train=x_train, x_test=x_test):
model.fit(x_train, y_train)
y_pred = model.predict(x_test)
y_pred_proba = model.predict_proba(x_test)[:, 1] if hasattr(model, "predict_proba") else None
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)
mcc = matthews_corrcoef(y_test, y_pred)
performance_results.append((model_name, accuracy, precision, recall, f1, mcc, model, y_pred, y_pred_proba))
3a) Define custom distance algorithms
In [3]:
import numpy as np
def custom_minkowski(x, y, p):
return np.sum(np.abs(x - y) ** p) ** (1 / p)
3b) Distance metrics demonstration
In this section, we demonstrate how various distance metrics work using a pair of sample vectors.
This helps to highlight the differences when using different metrics.
Explanation:
- Manhattan Distance sums the absolute differences (L1 norm).
- Euclidean Distance gives the straight-line (L2) distance between points.
- Minkowski Distance generalizes these calculations; for p=3, it provides an alternative scaling of the differences.
- Cosine Distance measures the angular difference, which is not influenced by the magnitude of the vectors.
In [4]:
from sklearn.metrics.pairwise import cosine_distances
# Define two sample vectors
vector_x = np.array([1, 2, 3])
vector_y = np.array([4, 5, 6])
# 1. Manhattan Distance (L1 norm)
manhattan_val = custom_minkowski(vector_x, vector_y, 1)
# 2. Euclidean Distance (L2 norm)
euclidean_val = custom_minkowski(vector_x, vector_y, 2)
# 3. Minkowski Distance with p=3 (generalized distance metric)
minkowski_val = custom_minkowski(vector_x, vector_y, 3)
# 4. Cosine Distance (measures the cosine of the angle between the two vectors)
cosine_distance_val = cosine_distances([vector_x], [vector_y])[0][0]
# Print results for comparison
print("Distance Metrics Demonstration on Sample Vectors:")
print("-------------------------------------------------")
print("Vector A:", vector_x)
print("Vector B:", vector_y)
print(f"Manhattan Distance: {manhattan_val:.4f}")
print(f"Euclidean Distance: {euclidean_val:.4f}")
print(f"Minkowski Distance (p=3): {minkowski_val:.4f}")
print(f"Cosine Distance: {cosine_distance_val:.4f}")
Distance Metrics Demonstration on Sample Vectors: ------------------------------------------------- Vector A: [1 2 3] Vector B: [4 5 6] Manhattan Distance: 9.0000 Euclidean Distance: 5.1962 Minkowski Distance (p=3): 4.3267 Cosine Distance: 0.0254
- Define and process different distances for distance-based models
In [5]:
from sklearn.metrics import pairwise_distances
from functools import partial
NEIGHBORS_DISTANCES = [
["manhattan"],
["euclidean"],
[custom_minkowski, 2],
["minkowski", 2],
["minkowski", 3],
["minkowski", 4],
["minkowski", 5],
]
VECTOR_DISTANCES = [
["linear"],
[custom_minkowski, 2],
]
def process_distance_params(model, model_name, distance_info, parameters: dict[str, int | str], invert=False):
metric_keyword = list(parameters.keys())[-1]
if len(distance_info) > 0 and callable(distance_info[0]):
parameters[metric_keyword] = "precomputed"
print_parameters = parameters.copy()
print_parameters[metric_keyword] = distance_info[0].__name__
if len(distance_info) == 2:
custom_metric = partial(distance_info[0], p=distance_info[1])
print_parameters["p"] = distance_info[1]
else:
custom_metric = distance_info[0]
d_train = pairwise_distances(x_train, x_train, metric=custom_metric)
if invert:
d_train = np.exp(-d_train)
d_test = pairwise_distances(x_test, x_train, metric=custom_metric)
if invert:
d_test = np.exp(-d_test)
evaluate_model(model(**parameters), f"{model_name} ({str(print_parameters).strip('{}')})", x_train=d_train,
x_test=d_test)
elif len(distance_info) == 1:
if type(distance_info[0]) == str:
parameters[metric_keyword] = distance_info[0]
evaluate_model(model(**parameters), f"{model_name} ({str(parameters).strip('{}')})")
elif len(distance_info) == 2:
if type(distance_info[0]) == str and type(distance_info[1]) == int:
parameters[metric_keyword] = distance_info[0]
parameters["p"] = distance_info[1]
evaluate_model(model(**parameters), f"{model_name} ({str(parameters).strip('{}')})")
def process_distances(model, model_name):
lower_model_name = model_name.lower()
if "neighbors" in lower_model_name:
for neighbors in range(1, 10):
for distance_info in NEIGHBORS_DISTANCES:
parameters = {
"n_neighbors": neighbors,
"metric": "",
}
process_distance_params(model, model_name, distance_info, parameters)
elif "vector" in lower_model_name:
for distance_info in VECTOR_DISTANCES:
parameters = {
"kernel": "",
}
process_distance_params(model, model_name, distance_info, parameters, True)
elif "random" in lower_model_name:
evaluate_model(model(random_state=42), model_name)
else:
evaluate_model(model(), model_name)
- Define and process different models
In [6]:
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
MODELS = (
("Logistic Regression", LogisticRegression),
("Support Vector Machine", SVC),
("Random Forest", RandomForestClassifier),
("K-Nearest Neighbors", KNeighborsClassifier),
("Gaussian Naive Bayes", GaussianNB),
)
for model_name, model in MODELS:
process_distances(model, model_name)
performance_results_res_only = [result[0:6] for result in performance_results]
sorted_mcc_results = sorted(performance_results_res_only, key=lambda res: res[5], reverse=True)
print('\n'.join(str(result) for result in sorted_mcc_results))
print(f"\nEvaluated {len(sorted_mcc_results)} different model configurations.")
('Gaussian Naive Bayes', 0.94, 0.9655172413793104, 0.9333333333333333, 0.9491525423728814, np.float64(0.8767819450076552))
("Support Vector Machine ('kernel': 'custom_minkowski', 'p': 2)", 0.94, 0.95, 0.95, 0.95, np.float64(0.875))
('Random Forest', 0.93, 1.0, 0.8833333333333333, 0.9380530973451328, np.float64(0.8670484701822558))
('Logistic Regression', 0.92, 0.9193548387096774, 0.95, 0.9344262295081968, np.float64(0.8326680989374551))
("Support Vector Machine ('kernel': 'linear')", 0.91, 0.9473684210526315, 0.9, 0.9230769230769231, np.float64(0.8163716483922342))
("K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'manhattan')", 0.91, 0.9473684210526315, 0.9, 0.9230769230769231, np.float64(0.8163716483922342))
("K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'manhattan')", 0.9, 0.9032258064516129, 0.9333333333333333, 0.9180327868852459, np.float64(0.7906141545466745))
("K-Nearest Neighbors ('n_neighbors': 8, 'metric': 'manhattan')", 0.89, 0.9622641509433962, 0.85, 0.9026548672566371, np.float64(0.7852514446933637))
("K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'minkowski', 'p': 5)", 0.89, 0.9298245614035088, 0.8833333333333333, 0.905982905982906, np.float64(0.7751407570592931))
("K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'euclidean')", 0.88, 0.9, 0.9, 0.9, np.float64(0.75))
("K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'custom_minkowski', 'p': 2)", 0.88, 0.9, 0.9, 0.9, np.float64(0.75))
("K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'minkowski', 'p': 2)", 0.88, 0.9, 0.9, 0.9, np.float64(0.75))
("K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'manhattan')", 0.88, 0.9, 0.9, 0.9, np.float64(0.75))
("K-Nearest Neighbors ('n_neighbors': 4, 'metric': 'minkowski', 'p': 4)", 0.86, 0.96, 0.8, 0.8727272727272727, np.float64(0.7348469228349535))
("K-Nearest Neighbors ('n_neighbors': 6, 'metric': 'manhattan')", 0.87, 0.9122807017543859, 0.8666666666666667, 0.8888888888888888, np.float64(0.733909865726352))
("K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'minkowski', 'p': 3)", 0.87, 0.8983050847457628, 0.8833333333333333, 0.8907563025210085, np.float64(0.7304476778759532))
("K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'minkowski', 'p': 4)", 0.87, 0.8983050847457628, 0.8833333333333333, 0.8907563025210085, np.float64(0.7304476778759532))
("K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'euclidean')", 0.87, 0.8983050847457628, 0.8833333333333333, 0.8907563025210085, np.float64(0.7304476778759532))
("K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'custom_minkowski', 'p': 2)", 0.87, 0.8983050847457628, 0.8833333333333333, 0.8907563025210085, np.float64(0.7304476778759532))
("K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'minkowski', 'p': 2)", 0.87, 0.8983050847457628, 0.8833333333333333, 0.8907563025210085, np.float64(0.7304476778759532))
("K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'minkowski', 'p': 3)", 0.87, 0.8983050847457628, 0.8833333333333333, 0.8907563025210085, np.float64(0.7304476778759532))
("K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'manhattan')", 0.87, 0.8852459016393442, 0.9, 0.8925619834710744, np.float64(0.7281928433801468))
("K-Nearest Neighbors ('n_neighbors': 4, 'metric': 'manhattan')", 0.86, 0.9259259259259259, 0.8333333333333334, 0.8771929824561403, np.float64(0.7208273412017193))
("K-Nearest Neighbors ('n_neighbors': 8, 'metric': 'euclidean')", 0.86, 0.9259259259259259, 0.8333333333333334, 0.8771929824561403, np.float64(0.7208273412017193))
("K-Nearest Neighbors ('n_neighbors': 8, 'metric': 'custom_minkowski', 'p': 2)", 0.86, 0.9259259259259259, 0.8333333333333334, 0.8771929824561403, np.float64(0.7208273412017193))
("K-Nearest Neighbors ('n_neighbors': 8, 'metric': 'minkowski', 'p': 2)", 0.86, 0.9259259259259259, 0.8333333333333334, 0.8771929824561403, np.float64(0.7208273412017193))
("K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'euclidean')", 0.86, 0.896551724137931, 0.8666666666666667, 0.8813559322033898, np.float64(0.7113513893458334))
("K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'custom_minkowski', 'p': 2)", 0.86, 0.896551724137931, 0.8666666666666667, 0.8813559322033898, np.float64(0.7113513893458334))
("K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'minkowski', 'p': 2)", 0.86, 0.896551724137931, 0.8666666666666667, 0.8813559322033898, np.float64(0.7113513893458334))
("K-Nearest Neighbors ('n_neighbors': 4, 'metric': 'minkowski', 'p': 3)", 0.85, 0.9411764705882353, 0.8, 0.8648648648648649, np.float64(0.7104941384475366))
("K-Nearest Neighbors ('n_neighbors': 4, 'metric': 'euclidean')", 0.85, 0.9245283018867925, 0.8166666666666667, 0.8672566371681416, np.float64(0.7034544192044717))
("K-Nearest Neighbors ('n_neighbors': 4, 'metric': 'custom_minkowski', 'p': 2)", 0.85, 0.9245283018867925, 0.8166666666666667, 0.8672566371681416, np.float64(0.7034544192044717))
("K-Nearest Neighbors ('n_neighbors': 4, 'metric': 'minkowski', 'p': 2)", 0.85, 0.9245283018867925, 0.8166666666666667, 0.8672566371681416, np.float64(0.7034544192044717))
("K-Nearest Neighbors ('n_neighbors': 6, 'metric': 'euclidean')", 0.85, 0.9245283018867925, 0.8166666666666667, 0.8672566371681416, np.float64(0.7034544192044717))
("K-Nearest Neighbors ('n_neighbors': 6, 'metric': 'custom_minkowski', 'p': 2)", 0.85, 0.9245283018867925, 0.8166666666666667, 0.8672566371681416, np.float64(0.7034544192044717))
("K-Nearest Neighbors ('n_neighbors': 6, 'metric': 'minkowski', 'p': 2)", 0.85, 0.9245283018867925, 0.8166666666666667, 0.8672566371681416, np.float64(0.7034544192044717))
("K-Nearest Neighbors ('n_neighbors': 6, 'metric': 'minkowski', 'p': 5)", 0.85, 0.9245283018867925, 0.8166666666666667, 0.8672566371681416, np.float64(0.7034544192044717))
("K-Nearest Neighbors ('n_neighbors': 4, 'metric': 'minkowski', 'p': 5)", 0.84, 0.9583333333333334, 0.7666666666666667, 0.8518518518518519, np.float64(0.7027494842451595))
("K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'euclidean')", 0.85, 0.8813559322033898, 0.8666666666666667, 0.8739495798319328, np.float64(0.6889449689057285))
("K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'custom_minkowski', 'p': 2)", 0.85, 0.8813559322033898, 0.8666666666666667, 0.8739495798319328, np.float64(0.6889449689057285))
("K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'minkowski', 'p': 2)", 0.85, 0.8813559322033898, 0.8666666666666667, 0.8739495798319328, np.float64(0.6889449689057285))
("K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'minkowski', 'p': 5)", 0.85, 0.8813559322033898, 0.8666666666666667, 0.8739495798319328, np.float64(0.6889449689057285))
("K-Nearest Neighbors ('n_neighbors': 6, 'metric': 'minkowski', 'p': 3)", 0.84, 0.9230769230769231, 0.8, 0.8571428571428571, np.float64(0.6864064729836441))
("K-Nearest Neighbors ('n_neighbors': 6, 'metric': 'minkowski', 'p': 4)", 0.84, 0.9230769230769231, 0.8, 0.8571428571428571, np.float64(0.6864064729836441))
("K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'euclidean')", 0.84, 0.8928571428571429, 0.8333333333333334, 0.8620689655172413, np.float64(0.6744004873337631))
("K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'custom_minkowski', 'p': 2)", 0.84, 0.8928571428571429, 0.8333333333333334, 0.8620689655172413, np.float64(0.6744004873337631))
("K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'minkowski', 'p': 2)", 0.84, 0.8928571428571429, 0.8333333333333334, 0.8620689655172413, np.float64(0.6744004873337631))
("K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'minkowski', 'p': 3)", 0.84, 0.8928571428571429, 0.8333333333333334, 0.8620689655172413, np.float64(0.6744004873337631))
("K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'minkowski', 'p': 3)", 0.84, 0.8793103448275862, 0.85, 0.864406779661017, np.float64(0.669993750430378))
("K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'minkowski', 'p': 3)", 0.84, 0.8666666666666667, 0.8666666666666667, 0.8666666666666667, np.float64(0.6666666666666666))
("K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'minkowski', 'p': 4)", 0.84, 0.8666666666666667, 0.8666666666666667, 0.8666666666666667, np.float64(0.6666666666666666))
("K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'minkowski', 'p': 5)", 0.84, 0.8666666666666667, 0.8666666666666667, 0.8666666666666667, np.float64(0.6666666666666666))
("K-Nearest Neighbors ('n_neighbors': 8, 'metric': 'minkowski', 'p': 3)", 0.83, 0.8909090909090909, 0.8166666666666667, 0.8521739130434782, np.float64(0.6564879518897745))
("K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'minkowski', 'p': 4)", 0.83, 0.8771929824561403, 0.8333333333333334, 0.8547008547008547, np.float64(0.6514480830604696))
("K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'manhattan')", 0.83, 0.864406779661017, 0.85, 0.8571428571428571, np.float64(0.647442259935504))
("K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'minkowski', 'p': 4)", 0.83, 0.864406779661017, 0.85, 0.8571428571428571, np.float64(0.647442259935504))
("K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'minkowski', 'p': 5)", 0.83, 0.864406779661017, 0.85, 0.8571428571428571, np.float64(0.647442259935504))
("K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'minkowski', 'p': 4)", 0.83, 0.8524590163934426, 0.8666666666666667, 0.859504132231405, np.float64(0.6444925165548426))
("K-Nearest Neighbors ('n_neighbors': 2, 'metric': 'manhattan')", 0.81, 0.9183673469387755, 0.75, 0.8256880733944955, np.float64(0.6369947448150328))
("K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'minkowski', 'p': 5)", 0.82, 0.8387096774193549, 0.8666666666666667, 0.8524590163934426, np.float64(0.6223983769835523))
("K-Nearest Neighbors ('n_neighbors': 8, 'metric': 'minkowski', 'p': 4)", 0.81, 0.8867924528301887, 0.7833333333333333, 0.831858407079646, np.float64(0.6216573937155796))
("K-Nearest Neighbors ('n_neighbors': 8, 'metric': 'minkowski', 'p': 5)", 0.81, 0.8867924528301887, 0.7833333333333333, 0.831858407079646, np.float64(0.6216573937155796))
("K-Nearest Neighbors ('n_neighbors': 2, 'metric': 'euclidean')", 0.77, 0.9111111111111111, 0.6833333333333333, 0.780952380952381, np.float64(0.5744269579035527))
("K-Nearest Neighbors ('n_neighbors': 2, 'metric': 'custom_minkowski', 'p': 2)", 0.77, 0.9111111111111111, 0.6833333333333333, 0.780952380952381, np.float64(0.5744269579035527))
("K-Nearest Neighbors ('n_neighbors': 2, 'metric': 'minkowski', 'p': 2)", 0.77, 0.9111111111111111, 0.6833333333333333, 0.780952380952381, np.float64(0.5744269579035527))
("K-Nearest Neighbors ('n_neighbors': 2, 'metric': 'minkowski', 'p': 3)", 0.76, 0.9090909090909091, 0.6666666666666666, 0.7692307692307693, np.float64(0.5592589407158036))
("K-Nearest Neighbors ('n_neighbors': 2, 'metric': 'minkowski', 'p': 4)", 0.76, 0.9090909090909091, 0.6666666666666666, 0.7692307692307693, np.float64(0.5592589407158036))
("K-Nearest Neighbors ('n_neighbors': 2, 'metric': 'minkowski', 'p': 5)", 0.75, 0.8888888888888888, 0.6666666666666666, 0.7619047619047619, np.float64(0.5333964609104418))
Evaluated 68 different model configurations.
- Compare different models and distances
In [7]:
import matplotlib.pyplot as plt
import seaborn as sns
from matplotlib.gridspec import GridSpec
from sklearn.metrics import confusion_matrix, roc_curve, auc, classification_report
df_results = pd.DataFrame(performance_results_res_only, columns=["Model Name", "Accuracy", "Precision", "Recall", "F1 Score", "MCC"])
df_sorted = df_results.sort_values(by="Recall", ascending=False)
top_models = df_sorted.head(5)
top_30_models = df_sorted.head(30)
# 1. Summary table Top 30
print("=" * 80)
print("HEART DISEASE CLASSIFICATION: MODEL EVALUATION SUMMARY")
print("=" * 80)
print("Top 30 Models by Recall:")
print("(For our case, Recall is probably the most important metric, because it is important to have as few false negatives as possible.\n"
"It would be fatal (in many cases in a literal sense) to not detect heart disease when it is actually present.\n"
"Thats why we sort these models by best recall.)")
styled_df = top_30_models.style.background_gradient(cmap='Blues', subset=['Accuracy', 'Precision', 'Recall', 'F1 Score', 'MCC'])\
.format({'Accuracy': '{:.4f}', 'Precision': '{:.4f}', 'Recall': '{:.4f}', 'F1 Score': '{:.4f}', 'MCC': '{:.4f}'})
display(styled_df)
# 2. Bar chart for top 5 models
print("\n" + "=" * 80)
print("COMPARATIVE VISUALIZATION OF ALL METRICS FOR TOP 5 MODELS")
print("=" * 80)
print("The bar chart compares Accuracy, Precision, Recall, and F1 Score for the top 5 models (by Recall).\n"
"It highlights the strengths and weaknesses of each model across these metrics, helping to identify the best-performing model.\n"
"It should be noted that the y axis starts at 0.5, to make the differences between the models more visible.\n")
plt.figure(figsize=(16, 10))
model_names = [name.split('.')[-1] for name in top_models['Model Name']] # Get just the model name
x = np.arange(0, len(model_names) * 1.5, 1.5)
width = 0.2
plt.bar(x - width*1.5, top_models['Accuracy'], width, label='Accuracy', color='#3498db')
plt.bar(x - width/2, top_models['Precision'], width, label='Precision', color='#2ecc71')
plt.bar(x + width/2, top_models['Recall'], width, label='Recall', color='#e74c3c')
plt.bar(x + width*1.5, top_models['F1 Score'], width, label='F1 Score', color='#f39c12')
plt.bar(x + width*2.5, top_models['MCC'], width, label='MCC', color='#9b59b6')
plt.xlabel('Model Names')
plt.ylabel('Score')
plt.title('Performance Comparison of Top 5 Models')
plt.xticks(x, model_names, rotation=45, ha='right')
plt.ylim(0.5, 1.0)
plt.legend(loc='lower right')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
advanced_metrics = {}
for model_name in top_models['Model Name']:
model_entry = next((entry for entry in performance_results if entry[0] == model_name), None)
if not model_entry:
continue
model = model_entry[6]
y_pred = model_entry[7] if len(model_entry) > 7 else None
y_pred_proba = model_entry[8] if len(model_entry) > 8 else None
cm = confusion_matrix(y_test, y_pred)
metrics_dict = {
"model": model,
"confusion_matrix": cm,
}
# ROC curve and AUC (if the model supports probabilities)
if y_pred_proba is not None:
fpr, tpr, _ = roc_curve(y_test, y_pred_proba)
roc_auc = auc(fpr, tpr)
metrics_dict["roc_auc"] = roc_auc
metrics_dict["fpr"] = fpr
metrics_dict["tpr"] = tpr
advanced_metrics[model_name] = metrics_dict
# 3. ROC Curve comparison (all in one plot)
print("\n" + "=" * 80)
print("ROC Curve")
print("=" * 80)
print("The ROC curve shows the trade-off between True Positive Rate (sensitivity) and False Positive Rate at different thresholds.\n"
"This is not available for all models.")
plt.figure(figsize=(17, 8))
colors = ['darkorange', 'forestgreen', 'royalblue', 'purple', 'brown']
for i, model_name in enumerate(top_models['Model Name']):
if "roc_auc" in advanced_metrics[model_name]:
fpr = advanced_metrics[model_name]["fpr"]
tpr = advanced_metrics[model_name]["tpr"]
roc_auc = advanced_metrics[model_name]["roc_auc"]
display_name = model_name.split(' (')[0] if ' (' in model_name else model_name
plt.plot(fpr, tpr, color=colors[i % len(colors)], lw=2,
label=f'{display_name} (AUC = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve Comparison of Top Models')
plt.legend(loc="lower right")
plt.grid(True, alpha=0.3)
plt.show()
# 5. Confusion Matrix for the best model
print("\n" + "=" * 80)
print("TOP MODEL CONFUSION MATRIX")
print("=" * 80)
print("The confusion matrix visualizes the performance of the best model in terms of true positives, true negatives, false positives, and false negatives.\n"
"The percentage matrix shows the proportion of predictions for each class, helping to understand where the model makes errors.\n")
best_model_name = top_models['Model Name'].iloc[0]
best_model_cm = advanced_metrics[best_model_name]["confusion_matrix"]
best_model_display_name = best_model_name.split(' (')[0] if ' (' in best_model_name else best_model_name
# Get both raw counts and percentages
plt.figure(figsize=(12, 5))
gs = GridSpec(1, 2, width_ratios=[1, 1])
# Raw counts
plt.subplot(gs[0])
sns.heatmap(best_model_cm, annot=True, fmt='d', cmap='Blues',
xticklabels=['No Disease', 'Heart Disease'],
yticklabels=['No Disease', 'Heart Disease'])
plt.title(f'Confusion Matrix - Raw Counts\n{best_model_display_name}')
plt.ylabel('True Label')
plt.xlabel('Predicted Label')
# Percentages (normalize by row)
plt.subplot(gs[1])
cm_percent = best_model_cm.astype('float') / best_model_cm.sum(axis=1)[:, np.newaxis] * 100
sns.heatmap(cm_percent, annot=True, fmt='.1f', cmap='Blues',
xticklabels=['No Disease', 'Heart Disease'],
yticklabels=['No Disease', 'Heart Disease'])
plt.title(f'Confusion Matrix - Percentage\n{best_model_display_name}')
plt.ylabel('True Label')
plt.xlabel('Predicted Label')
plt.tight_layout()
plt.show()
# 6. Conclusion and medical siginificance
print("\n" + "=" * 80)
print("Summary best Model and medical significance")
print("=" * 80)
best_model_entry = next((result for result in performance_results if result[0] == best_model_name), None)
best_accuracy = best_model_entry[1]
best_precision = best_model_entry[2]
best_recall = best_model_entry[3]
best_f1 = best_model_entry[4]
best_mcc = best_model_entry[5]
print(f"Best Model: {best_model_display_name}")
print(f"Accuracy: {best_accuracy:.4f}")
print(f"Precision: {best_precision:.4f}")
print(f"Recall: {best_recall:.4f}")
print(f"F1-Score: {best_f1:.4f}")
print(f"MCC: {best_mcc:.4f}")
if "roc_auc" in advanced_metrics[best_model_name]:
print(f"ROC AUC: {advanced_metrics[best_model_name]['roc_auc']:.4f}")
false_negatives = best_model_cm[1,0] # True class is 1 (disease), predicted 0 (no disease)
false_positives = best_model_cm[0,1] # True class is 0 (no disease), predicted 1 (disease)
fn_rate = false_negatives / best_model_cm[1,:].sum() * 100
fp_rate = false_positives / best_model_cm[0,:].sum() * 100
print("\nMEDICAL INTERPRETATION:")
print(f"- False Negatives (missed heart disease cases): {false_negatives} cases ({fn_rate:.1f}%)")
print(f"- False Positives (unnecessary further testing): {false_positives} cases ({fp_rate:.1f}%)\n")
print("Clinical Significance:")
print(f"1. In heart disease prediction, false negatives (missed diagnoses) can lead to")
print(f" delayed treatment and potentially severe health consequences.")
print(f"2. False positives lead to additional testing which has financial and")
print(f" psychological costs but is generally preferred over missing cases. Thats why we put the Recall over the F1. False positives are bad but do not lead to immediate health risks.")
print(f"3. This model correctly identifies {100-fn_rate:.1f}% of actual heart disease cases,")
print(f"4. For a definitive diagnosis, additional tests would be recommended for positive")
print(f" predictions to rule out false positives.")
================================================================================ HEART DISEASE CLASSIFICATION: MODEL EVALUATION SUMMARY ================================================================================ Top 30 Models by Recall: (For our case, Recall is probably the most important metric, because it is important to have as few false negatives as possible. It would be fatal (in many cases in a literal sense) to not detect heart disease when it is actually present. Thats why we sort these models by best recall.)
| Â | Model Name | Accuracy | Precision | Recall | F1 Score | MCC |
|---|---|---|---|---|---|---|
| 0 | Logistic Regression | 0.9200 | 0.9194 | 0.9500 | 0.9344 | 0.8327 |
| 2 | Support Vector Machine ('kernel': 'custom_minkowski', 'p': 2) | 0.9400 | 0.9500 | 0.9500 | 0.9500 | 0.8750 |
| 32 | K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'manhattan') | 0.9000 | 0.9032 | 0.9333 | 0.9180 | 0.7906 |
| 67 | Gaussian Naive Bayes | 0.9400 | 0.9655 | 0.9333 | 0.9492 | 0.8768 |
| 34 | K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'custom_minkowski', 'p': 2) | 0.8800 | 0.9000 | 0.9000 | 0.9000 | 0.7500 |
| 18 | K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'manhattan') | 0.8700 | 0.8852 | 0.9000 | 0.8926 | 0.7282 |
| 60 | K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'manhattan') | 0.9100 | 0.9474 | 0.9000 | 0.9231 | 0.8164 |
| 46 | K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'manhattan') | 0.8800 | 0.9000 | 0.9000 | 0.9000 | 0.7500 |
| 35 | K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'minkowski', 'p': 2) | 0.8800 | 0.9000 | 0.9000 | 0.9000 | 0.7500 |
| 1 | Support Vector Machine ('kernel': 'linear') | 0.9100 | 0.9474 | 0.9000 | 0.9231 | 0.8164 |
| 33 | K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'euclidean') | 0.8800 | 0.9000 | 0.9000 | 0.9000 | 0.7500 |
| 50 | K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'minkowski', 'p': 3) | 0.8700 | 0.8983 | 0.8833 | 0.8908 | 0.7304 |
| 49 | K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'minkowski', 'p': 2) | 0.8700 | 0.8983 | 0.8833 | 0.8908 | 0.7304 |
| 48 | K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'custom_minkowski', 'p': 2) | 0.8700 | 0.8983 | 0.8833 | 0.8908 | 0.7304 |
| 47 | K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'euclidean') | 0.8700 | 0.8983 | 0.8833 | 0.8908 | 0.7304 |
| 38 | K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'minkowski', 'p': 5) | 0.8900 | 0.9298 | 0.8833 | 0.9060 | 0.7751 |
| 37 | K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'minkowski', 'p': 4) | 0.8700 | 0.8983 | 0.8833 | 0.8908 | 0.7304 |
| 3 | Random Forest | 0.9300 | 1.0000 | 0.8833 | 0.9381 | 0.8670 |
| 36 | K-Nearest Neighbors ('n_neighbors': 5, 'metric': 'minkowski', 'p': 3) | 0.8700 | 0.8983 | 0.8833 | 0.8908 | 0.7304 |
| 19 | K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'euclidean') | 0.8600 | 0.8966 | 0.8667 | 0.8814 | 0.7114 |
| 65 | K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'minkowski', 'p': 4) | 0.8400 | 0.8667 | 0.8667 | 0.8667 | 0.6667 |
| 20 | K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'custom_minkowski', 'p': 2) | 0.8600 | 0.8966 | 0.8667 | 0.8814 | 0.7114 |
| 7 | K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'minkowski', 'p': 2) | 0.8500 | 0.8814 | 0.8667 | 0.8739 | 0.6889 |
| 52 | K-Nearest Neighbors ('n_neighbors': 7, 'metric': 'minkowski', 'p': 5) | 0.8500 | 0.8814 | 0.8667 | 0.8739 | 0.6889 |
| 5 | K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'euclidean') | 0.8500 | 0.8814 | 0.8667 | 0.8739 | 0.6889 |
| 6 | K-Nearest Neighbors ('n_neighbors': 1, 'metric': 'custom_minkowski', 'p': 2) | 0.8500 | 0.8814 | 0.8667 | 0.8739 | 0.6889 |
| 66 | K-Nearest Neighbors ('n_neighbors': 9, 'metric': 'minkowski', 'p': 5) | 0.8400 | 0.8667 | 0.8667 | 0.8667 | 0.6667 |
| 24 | K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'minkowski', 'p': 5) | 0.8200 | 0.8387 | 0.8667 | 0.8525 | 0.6224 |
| 23 | K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'minkowski', 'p': 4) | 0.8300 | 0.8525 | 0.8667 | 0.8595 | 0.6445 |
| 22 | K-Nearest Neighbors ('n_neighbors': 3, 'metric': 'minkowski', 'p': 3) | 0.8400 | 0.8667 | 0.8667 | 0.8667 | 0.6667 |
================================================================================ COMPARATIVE VISUALIZATION OF ALL METRICS FOR TOP 5 MODELS ================================================================================ The bar chart compares Accuracy, Precision, Recall, and F1 Score for the top 5 models (by Recall). It highlights the strengths and weaknesses of each model across these metrics, helping to identify the best-performing model. It should be noted that the y axis starts at 0.5, to make the differences between the models more visible.
================================================================================ ROC Curve ================================================================================ The ROC curve shows the trade-off between True Positive Rate (sensitivity) and False Positive Rate at different thresholds. This is not available for all models.
================================================================================ TOP MODEL CONFUSION MATRIX ================================================================================ The confusion matrix visualizes the performance of the best model in terms of true positives, true negatives, false positives, and false negatives. The percentage matrix shows the proportion of predictions for each class, helping to understand where the model makes errors.
================================================================================ Summary best Model and medical significance ================================================================================ Best Model: Logistic Regression Accuracy: 0.9200 Precision: 0.9194 Recall: 0.9500 F1-Score: 0.9344 MCC: 0.8327 ROC AUC: 0.9858 MEDICAL INTERPRETATION: - False Negatives (missed heart disease cases): 3 cases (5.0%) - False Positives (unnecessary further testing): 5 cases (12.5%) Clinical Significance: 1. In heart disease prediction, false negatives (missed diagnoses) can lead to delayed treatment and potentially severe health consequences. 2. False positives lead to additional testing which has financial and psychological costs but is generally preferred over missing cases. Thats why we put the Recall over the F1. False positives are bad but do not lead to immediate health risks. 3. This model correctly identifies 95.0% of actual heart disease cases, 4. For a definitive diagnosis, additional tests would be recommended for positive predictions to rule out false positives.