Deep reinforcement learning inspired optimization framework using Optuna for brain tumor detection

Abstract

Accurate brain tumor detection is essential for effective clinical diagnosis; however, the performance of deep learning models is highly sensitive to manually selected architectures and hyperparameters. To address this challenge, this paper presents a reinforcement learning–inspired automated optimization framework for brain tumor detection that eliminates manual trial-and-error tuning of hyperparameters. The proposed approach integrates EfficientNetB0 as a fixed feature extractor (base model) with an Optuna-based reinforcement learning strategy to jointly optimize the classifier architecture and key training hyperparameters, including learning rate, batch size, dropout rate, and network depth. Unlike existing studies that rely on static or heuristically tuned models, the proposed framework dynamically adapts model configurations based on validation feedback. Experiments conducted on the BraTS 2020 MRI dataset demonstrate that the optimized model achieves an accuracy of 92%, an F1-score of 92%, and a ROC–AUC of 0.96. Additional evaluations on imbalanced and cross-dataset settings show stable minority-class performance and good generalization. The results confirm that the proposed automated optimization framework offers a robust, scalable, and clinically relevant solution for brain tumor detection, representing a significant advancement over manually tuned deep learning approaches.