The invention relates to a cable abrasion testing system, and particularly relates to an abrasion testing system for an optical composite submarine cable and an operating method thereof. Fault character searching and abrasion degree rule discovering for optical fiber composite submarine cables are very difficult at present. The system includes ground anchors, anchor devices, steel wire ropes, tension sensors, abrading parts, a vibration exciter, vibration sensors, winding engines, and a controller connected to the tension sensors, the vibration sensors, the vibration exciter and the winding engines. Two ends of a submarine cable are fixed through the anchor devices, the steel wire ropes, the tension sensors and the ground anchors respectively. The winding engines suspend the middle of the submarine cable through the steel wire ropes. The vibration exciter and the vibration sensors are disposed at the suspended part of the submarine cable. The abrading parts are at two sides of the suspended part of the submarine cable. The middle of the submarine cable is pulled with the fine steel wire ropes, and can swing back and forth under control by the winding engines. The system and the method are simple and feasible. An actual abrasion process of the submarine cable can be stimulated to the utmost. Testing supports are provided for abrasion fault monitoring and damage degree assessment.