Noise is a common problem in most of the image understanding analysis. Border noise is one of the common document noises that are introduced when scanning and/or capturing thick books and old books. This noise is responsible for non-accurate results for document image analysis system as example OCR engines. In this paper we introducedan algorithm which can effectively remove the border noise from the document image by using X-Y cut and filtering technique based on morphological operations. We evaluate our technique on CBDAR 2007 document dewarping dataset and make comparison with other state-of-art methods. The experiment results show the high performance of the algorithm in removing the border noise (textual and non-textual noise) for a variety of documents with different levels of complexity and different structure. The used documents include diagrams, figures, formulas and different languages (English, Arabic, Chinese and Japanese document). © 2015 by IJIR (CESER PUBLICATIONS).