In this paper, we address the problem of three dimensional human pose tracking and estimation using Particle Swarm Optimization (PSO) with an improved silhouette extraction mechanism. In this work, the tracking problem is formulated as a nonlinear function optimization problem so the main objective is to optimize the fitness function between the 3D human model and the image observations. In order to improve the tracking performance, new shadow detection, removal and a level-set mechanism are applied during silhouette extraction. Both the silhouette and edge likelihood are used in the fitness function. Experiments using HumanEva-II dataset demonstrate that the proposed approach performance is considerably better than baseline algorithm which uses the Annealed Particle Filter (APF). © 2015 SPIE.