To further to improve the detection accuracy and real-time performance of infrared small target detection at sea, a new method based on weighted scene priors is introduced. Firstly, using the sparse characteristics of the target and the non-local self-correlation characteristics of the sea background, the target-background separation problem is modeled as a robust low-rank matrix recovery problem. Moreover, the prior information on sea background is added into the model by weighted nuclear norm to accelerate the decomposition of target and background images’ matrix in the algorithm. Finally, the alternating direction method of multipliers (A DMM) is introduced to further to accelerate the iteration speed of the solution. The experimental results show that the proposed algorithm can effectively improve the accuracy of target detection. The real-time performance of the algorithm is improved by 120% compared with the original algorithm.