Based on observation, in DO Terror ability works very similar to Fear ability with an exception that there is no models "outnumber" check and if a unit affected by the terror fails to pass the Ld test it always flees.\
If both rival units have Terror ability then it is disregarded and does not affect them.
The Ld test formula is the same as for any other Ld test