An algorithm for detecting the small infrared target was researched. We have analysed the morphologic Top-hat operator in the first, and use a method of Otsu to make the threshold , which is used to divide up the image and detect the small infrared target. The Matlab simulation shows that the method can raise the successful rate of detecting the small infrared target in a certain extent, and by the analysing the results, we find this method can not only be applied to single scene, but also can increase the robustness of object detection in some extent. At the same time, we describe a system of infrared image processing based on DSP and FPGA. The system can meet the requirements of timeliness and flexibility in some extent, and have strong versatility and scalability. The paper introduces the overall structure of the system, and gives the components of the system hardware parts, and describes the relation of data transfer in various parts at the same time.