Computer vision has a great potential to deal with agriculture problems. It is crucial to utilize novel tools and techniques in the agriculture food industry. The focus of current studies is to automate the fruit harvesting, grading of fruits, fruit recognition, and identification of diseases in the agriculture domain using deep learning and computer vision. Integrating deep learning with computer vision facilitates the consistent, speedy and trustworthy classification of fruit and vegetables compared to the traditional machine learning algorithm. However, there are still some challenges, such as the need for expert farmers to develop large-scale datasets to recognize and identify the problems of agriculture production. This survey includes eighty papers relevant to deep learning and computer vision techniques in the agriculture field. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.