In order to solve the dispersion constraints of the optical fiber communication systems to achieve high-speed, long?鄄distance transmission, we analyzed the main reason that impact optical fiber communication system, and described a fiber channel simulation model. This paper introduced the least-mean-square algorithm (LMS) and recursive least squares algorithm (RLS). Simulate LMS algorithm and RLS algorithm to 10G/s fiber channel performance of adaptive equalization using MATLAB. Simulation results show that LMS algorithm and RLS algorithm are simple, efficient, and can effectively remove inter-symbol interference that caused by fiber dispersion and polarization mode dispersion (PMD), RLS algorithm is slightly more complicated than LMS, but the convergence more quickly, then the two kinds of algorithms are more suitable for high speed requirements of an equalization fiber-optic communication systems.