Update report

Author: seofernando25

report/sections/3_experiment.tex

@@ -201,8 +201,4 @@ \subsubsection{Metrics}
 
 \subsubsection{Analysis}
 
-To evaluate the performance differences between the optimizers, we will employ both qualitative and quantitative analysis methods.
-
-Qualitatively, we inspect the convergence plots to understand the search trajectory and efficiency of each algorithm over the 50 generations. We will also utilize box plots to visualize the spread of the final fitness values, allowing us to assess the stability and variance of the solutions found across the 10 independent runs.
-
-Quantitatively, we will perform statistical hypothesis testing to determine if observed differences in performance are significant or merely due to random chance. Since the distribution of fitness scores is not guaranteed to be normal, we will opt for the \textbf{Wilcoxon signed-rank test}, a non-parametric pairwise comparison test. We will define our significance level at $\alpha = 0.05$. The null hypothesis ($H_0$) states that the median difference between pairs of observations from two optimizers is zero (i.e., they perform equally).
+Performance differences are assessed qualitatively via convergence plots (search trajectory) and box plots (solution stability across runs). For quantitative comparison, we apply the Wilcoxon signed-rank test ($\alpha=0.05$, non-parametric) to determine if observed differences are significant. The null hypothesis is that the median performance difference between any two optimizers is zero.

report/sections/4_results.tex

@@ -4,7 +4,7 @@ \section{Results}
 
 \subsection{RQ1: Effectiveness and Convergence}
 
-To answer RQ1, we analyzed the final fitness scores and the convergence behavior of each optimizer. As illustrated in Figure \ref{fig:convergence}, all three optimizers demonstrated the ability to improve solutions over the generations, though with varying efficiency.
+All optimizers improved solutions over 50 evaluations, but convergence efficiency varied by model (Figure \ref{fig:convergence}). For DT and KNN, which have relatively small search spaces, GA, PSO, and RS converged rapidly to near-identical optimal configurations. Their final fitness scores were approximately 0.3384 and 0.4308, respectively (Figure \ref{fig:test_perf}).
 
 \begin{figure}[H]
     \centering
@@ -13,18 +13,7 @@ \subsection{RQ1: Effectiveness and Convergence}
     \label{fig:convergence}
 \end{figure}
 
-For the \textbf{Decision Tree} and \textbf{KNN}, the search space was relatively small. Consequently, all three optimizers rapidly converged to near-identical optimal configurations. As shown in the final test performance (Figure \ref{fig:test_perf}), the DT achieved a fitness of $\approx 0.3384$, while the KNN achieved $\approx 0.4308$.
-
-We have also plotted the mean and standard deviation of the current fitness for each $n$ evaluations for each optimizer. The curves are smoothed for readability (with a smoothing factor of 0.5), while the shaded regions reflect the underlying variation. As shown in Figure \ref{fig:evaluations}, the current fitness is generally improving over the evaluations, though with varying efficiency.
-
-\begin{figure}[H]
-    \centering
-    \includegraphics[width=\textwidth]{./figures/evaluations.png}
-    \caption{Current fitness behavior of GA, PSO, and RS across 50 evaluations for all three models.}
-    \label{fig:evaluations}
-\end{figure}
-
-For the \textbf{CNN}, which possesses the largest and most complex search space, we observed distinct behaviors. While GA (orange line in Figure \ref{fig:convergence}) started with lower fitness, it showed steady improvement. Random Search (RS), surprisingly, started with high fitness in several runs, likely due to the efficacy of random sampling in high-dimensional spaces where few parameters dominate performance. Ultimately, all algorithms converged to a test performance of approximately $0.77$ (Figure \ref{fig:test_perf}).
+For the more complex CNN, GA showed steady improvement from a lower initial fitness, while RS occasionally found high-fitness configurations early (Figure \ref{fig:convergence}). All optimizers converged to a similar final performance of approximately 0.77 (Figure \ref{fig:test_perf}).
 
 \begin{figure}[H]
     \centering
@@ -35,7 +24,7 @@ \subsection{RQ1: Effectiveness and Convergence}
 
 \subsection{RQ2: Stability}
 
-Stability was measured by the standard deviation of the final fitness scores across the 10 runs, visualized in the box plots in Figure \ref{fig:boxplot}.
+The stability of final solutions, shown by box plots of fitness across 10 runs (Figure \ref{fig:boxplot}), was high for all models. Variance was minimal for DT and KNN. For CNN, all optimizers produced similar interquartile ranges (approximately 0.79 to 0.83), with minor differences in outlier counts (GA: 1, PSO: 2, RS: 1).
 
 \begin{figure}[H]
     \centering
@@ -44,11 +33,9 @@ \subsection{RQ2: Stability}
     \label{fig:boxplot}
 \end{figure}
 
-The box plots provide a nuanced view of the stability for each model. For the \textbf{CNN}, the distributions are quite similar across optimizers, with whiskers generally falling within the $0.79-0.83$ range. In terms of outliers, GA and RS each exhibit one, while PSO displays two. For the \textbf{Decision Tree (DT)}, Random Search (RS) shows the largest interquartile range and a significant lower whisker; however, the overall variance remains small given the scale ($0.3415-0.3440$). The metaheuristics are more packed at the upper end, though GA includes two outliers, and PSO has a lower whisker extending to approximately $0.34225$. Finally, for the \textbf{KNN}, no outliers are observed for any of the optimizers.
-
 \subsection{Statistical Significance}
 
-We performed the \textbf{Wilcoxon signed-rank test} to determine if there were statistically significant differences between the optimizers. The results are summarized in Table \ref{tab:wilcoxon}.
+Wilcoxon signed-rank tests showed no significant performance differences between any optimizer pairs across all models (p > 0.05; Table \ref{tab:wilcoxon}). The nearest to significance was PSO versus RS on KNN (p = 0.094). The memetic GA variant also showed no improvement over the standard GA.
 
 \begin{table}[H]
 \centering
@@ -66,14 +53,4 @@ \subsection{Statistical Significance}
 GA-Memetic vs RS & -- & 0.432 & 0.688 \\
 \bottomrule
 \end{tabular}
-\end{table}
-
-The analysis reveals:
-\begin{itemize}
-    \item \textbf{No Significant Differences:} All pairwise comparisons yield $p > 0.05$, so we fail to reject the null hypothesis across models.
-    \item \textbf{Identical Performance (DT):} GA-Standard vs PSO remains $p=1.000$, indicating indistinguishable outcomes on Decision Trees.
-    \item \textbf{Near Significance (KNN):} PSO vs RS on KNN is $p=0.094$, hinting PSO may modestly outperform RS, but it does not reach $\alpha=0.05$.
-    \item \textbf{Memetic Variants:} GA-Memetic comparisons (vs GA-Standard, PSO, RS) are all non-significant ($p > 0.05$), showing no measurable improvement over the standard GA under our budget.
-\end{itemize}
-
-Given the GA population size of $30$ and the strict budget of $50$ evaluations, the algorithm is structurally capped at fewer than three full generations: $30$ evaluations are spent on initialization, leaving only $20$ offspring evaluations (about $0.67$ of a generation). Roughly $60\%$ of the budget is therefore consumed by warm-up sampling, so GA behaves similarly to Random Search under this constraint. PSO faces the same tight budget and likewise does not clear a statistically meaningful advantage. Under such sub-$2\times$-population budgets, Random Search is effectively superior because it pays no population-initialization overhead. The value of GA (and possibly PSO) would likely emerge only with larger budgets that permit 10+ generations, where exploitation phases can amortize their startup cost.
+\end{table}

report/sections/5_limitations.tex

@@ -1,8 +1,8 @@
 \section{Limitations}
 
-\paragraph{Insufficient Evolutionary Iterations.} The primary threat to validity is the interaction between the population size ($P=30$) and the fixed evaluation budget ($B=50$). This ratio mathematically restricted the Genetic Algorithm to fewer than two full generations ($N_{gen} \approx 1.67$). Consequently, 60\% of the computational budget was consumed by the random initialization phase (Generation 0), leaving insufficient iterations for the metaheuristic mechanisms (crossover, mutation, and selection) to drive convergence away from the initial random distribution.
+\paragraph{Insufficient Evolutionary Iterations.} The fixed evaluation budget ($B=50$) relative to the population size ($P=30$) restricted the Genetic Algorithm to fewer than two full generations. Consequently, 60\% of the budget was consumed by initial random sampling, leaving insufficient evaluations for crossover, mutation, and selection to drive convergence.
 
-\paragraph{Statistical Power.} The experiment utilized $N=10$ independent runs per optimizer-model pair. While standard for exploratory HPO studies, this sample size limits the statistical power of the Wilcoxon signed-rank test, potentially masking small but consistent effect sizes between the algorithms.
+\paragraph{Statistical Power.} With $N=10$ independent runs per configuration, the Wilcoxon signed-rank test had limited power to detect small but consistent performance differences between algorithms.
 
-\paragraph{Scope Validity.} The evaluation was restricted to grayscale CIFAR-10 and fixed hyperparameter search spaces. These results may not generalize to higher-dimensional search spaces (e.g., full RGB imagery or deeper architectures) where the "curse of dimensionality" might differentiate global search strategies (GA) from local ones (PSO) more effectively.
+\paragraph{Scope Validity.} Experiments were conducted only on grayscale CIFAR-10 with defined hyperparameter spaces. Results may not generalize to higher-dimensional spaces (e.g., RGB images or deeper architectures) where exploration-exploitation trade-offs could differ.
 

report/sections/6_conclusion.tex

@@ -1,18 +1,15 @@
 \section{Conclusion}
 
-This study compared metaheuristic optimizers (GA, PSO) against a Randomized Search (RS) baseline under a strict computational budget. Statistical testing confirmed that GA and PSO failed to significantly outperform RS for any of the tested Decision Tree, KNN, or CNN models (p > 0.05). All optimizers converged to statistically indistinguishable fitness plateaus.
+This study evaluated metaheuristic optimizers (GA, PSO) against a Randomized Search baseline under a strict budget of 50 evaluations. No significant performance difference was found between methods across Decision Tree, KNN, or CNN models ($p > 0.05$), with all reaching similar fitness plateaus.
 
-This null result is attributed to the initialization overhead of population-based methods within a micro-budget regime. The geneetic algorithms for example, expended 60\% of their budget on the initial random population, leaving insufficient evaluations for evolutionary operators to amortize this cost and drive meaningful improvement.
+The result is explained by initialization overhead: with a population of 30, GA used 60\% of its budget on initial sampling, leaving too few evaluations for evolutionary operators to yield improvement. In such micro-budget regimes (budget $< 2 \times$ population), population-based methods behave similarly to Random Search.
 
-First, \textbf{metaheuristics failed to outperform the baseline.} Statistical testing confirmed no significant difference in final solution quality between GA, PSO, and RS across Decision Tree, KNN, or CNN architectures ($p > 0.05$). All methods converged to statistically indistinguishable fitness plateaus (e.g., $\approx 0.77$ for CNN).
-
-We conclude that for micro-budget HPO tasks, where the total evaluation budget is not substantially larger than the population size, the added complexity of metaheuristics is unjustified. In such scenarios, Randomized Search can potentially be a more effective strategy.
+Therefore, for hyperparameter optimization with very limited evaluations, the added complexity of metaheuristics like GA and PSO is not justified. Random Search proved equally effective baseline under these constraints.
 
 Future work should:
-
 \begin{itemize}
-    \item Extend evaluation budgets to observe whether GA/PSO advantages emerge once initialization overhead is amortized.
-    \item Test budget-aware variants in order to shrink startup cost under tight budgets.
-    \item Replicate on different datasets and larger hyperparameter spaces to assess generality.
-    \item Increase run counts to raise test power.
+    \item Increase evaluation budgets to allow amortization of initialization costs.
+    \item Explore adaptive or budget-aware variants of GA and PSO.
+    \item Extend experiments to broader datasets and hyperparameter spaces.
+    \item Use larger run counts to improve statistical power.
 \end{itemize}