[Speedz]

Try a crossover cable between server and computer next to it.

(edit: try different network cables to all machines first too, maybe a bad cable?)

A couple issues this could uncover:

Bad router if the speed is fine.
Bad NIC drivers if speed still is bad
Driver conflict if you have more than one NIC in the server. If speed is still bad.
Bad NIC or NIC settings if speed is still bad.

Also is this a motherbaord integrated NIC on the server? If so and these issues are not resolved by a crossover direct link. Try a PCI NIC and disable the onboard.

Is there any noticeable latency in any other real time environment on either the server or client machines?

Give all that a shot, let us know what happens.