The call simulator should be from physical hardware and not a virtual machine.

Virtual machines do time-slicing between the processors that will mess up the timing for call simulations and show high latencies and jitter when the network is fine. The destination location should also be a physical computer and not a virtual machine. It can be a computer, router, or switch, or even phone. Generally, if you want the best (most accurate) results, you should test from a desktop computer to another desktop computer. You will want to make sure that no other processes are running on the computers that could interfere (Outlook for example will check for messages every so often and will cause a hiccough that could show up as higher latency as well as jitter).