Most studies on hot-mix asphalt have used crushed fine aggregate and gravel; research regarding the use of mining sand or marine sand as fine aggregate is limited. This work investigates the effects of fine aggregate physical properties on the engineering properties of hot-mix asphalt. Four types of sand (quarry, river, mining, and marine sand) with two conventional binders (PEN 50/60 and PEN 80/100) and four types of polymer-modified bitumen (PMB) (PM1-82, PM1-76, PM2-82, and PM2-76) were used. Electron microscopy scanning was performed along with sieve analysis and tests for specific gravity and angularity of the fine aggregate on the different types of sand to determine their properties and thereby examine the effects of these properties when the sands were incorporated into a bituminous mixture. Marshall test results showed that fine aggregate physical properties influenced the optimum bitumen contents and engineering properties. The study also revealed that angular and rougher particles of fine aggregate produced lower voids in mineral aggregate, lower optimum bitumen content, higher shear strength, and higher density, which consequently increased the stability of the resulting mixtures. Moreover, a polymer-modified bituminous mixture enhanced stability and stiffness, reduced density, and slightly increased the voids in mineral aggregate.