Improper baseline return from the previous task-evoked hemodynamic response (HR) can contribute to a large variation in the subsequent HR, affecting the estimation of mental workload in brain-computer interface systems. In this study, we proposed a method using vector phase analysis to detect the baseline state as being optimal or suboptimal. We hypothesize that selecting neuronal-related HR as observed in the optimal-baseline blocks can lead to an improvement in estimating mental workload. Oxygenated and deoxygenated hemoglobin concentration changes were integrated as parts of the vector phase. The proposed method was applied to a block-design functional near-infrared spectroscopy dataset (total blocks = 1384), measured on 24 subjects performing multiple difficulty levels of mental arithmetic task. Significant differences in hemodynamic signal change were observed between the optimal- and suboptimal-baseline blocks detected using the proposed method. This supports the effectiveness of the proposed method in detecting baseline state for better estimation of mental workload. The results further highlight the need of customized recovery duration. In short, the proposed method offers a practical approach to detect task-evoked signals, without the need of extra probes. © 2001-2011 IEEE.