%X A signal subspace approach for extracting visual evoked potentials (VEPs) from the background electroencephalogram (EEG) colored noise without the need for a prewhitening stage is proposed. Linear estimation of the clean signal is performed by minimizing signal distortion while maintaining the residual noise energy below some given threshold. The generalized eigendecomposition of the covariance matrices of a VEP signal and brain background EEG noise is used to transform them jointly to diagonal matrices. The generalized subspace is then decomposed into signal subspace and noise subspace. Enhancement is performed by nulling the components in the noise subspace and retaining the components in the signal subspace. The performance of the proposed algorithm is tested with simulated and real data, and compared with the recently proposed signal subspace techniques. With the simulated data, the algorithms are used to estimate the latencies of P100, P200, and P300 of VEP signals corrupted by additive colored noise at different values of SNR. With the real data, the VEP signals are collected at Selayang Hospital, Kuala Lumpur, Malaysia, and the capability of the proposed algorithm in detecting the latency of P100 is obtained and compared with other subspace techniques. The ensemble averaging technique is used as a baseline for this comparison. The results indicated significant improvement by the proposed technique in terms of better accuracy and less failure rate. © 2006 IEEE. %K Colored noise; Covariance matrices; Diagonal matrices; Ensemble-averaging techniques; Failure rate; Generalized eigendecomposition; Linear estimation; Malaysia; Noise subspace; Nulling; Prewhitening; Residual noise; Signal sub-space; Simulated data; subspace filtering; Subspace techniques; Visual evoked potential; visual evoked potentials (VEPs), Algorithms; Covariance matrix; Electroencephalography; Electrophysiology; Multiuser detection; Signal analysis; White noise, Signal processing, algorithm; article; brain; controlled study; data extraction; diagnostic accuracy; electroencephalogram; evoked visual response; generalized subspace approach; noise; signal detection; signal noise ratio; simulation; subspace based dynamical estimation method; subspace regularization method, Algorithms; Computer Simulation; Electroencephalography; Evoked Potentials, Visual; Humans; Reproducibility of Results; Signal Processing, Computer-Assisted %O cited By 15 %J IEEE Transactions on Biomedical Engineering %L scholars2168 %D 2011 %N 5 %R 10.1109/TBME.2010.2101073 %T Single-trial subspace-based approach for VEP extraction %V 58 %A N. Kamel %A M.Z. Yusoff %A A.F.M. Hani %P 1383-1393