Hyperspectral unmixing is an important issue to analyze hyperspectral data. Based on the present mixing models, a new nonlinear unmixing algorithm for hyperspectral imagery was proposed. By introducing the abundance nonnegative constraint, abundance sum-to-one constraint and the bound constraints of nonlinear parameters, the proposed algorithm transforms the hyperspectral unmixing problem into a constrained nonlinear least squares problem. It consists of two sub-problems which obtain alternately the abundance vectors and nonlinear parameters of the observation pixels. Then, the alternating iterative optimization technique was used to solve this problem. The experimental results on synthetic and real hyperspectral dataset demonstrated that the proposed algorithm can effectively overcome the inherent limitations of the linear mixing model. Meanwhile, the proposed algorithm performs well for noisy data, and can also be used as an effective technique for the nonlinear unmixing of hyperspectral imagery.