Issue with PTP synchronizing and Skydel Simulator

Dear Orolia/Skydel team/community,

We have found a issue/bug in our simulation, related to the simulation time vs. computer time synchronized with PTP

  1. The Orolia PC GSG-8 has been synchronized with a PTP eth card.

  2. The simulation time has been set to “computer time”

  3. There are two instances of Skydel opened, and the second one is attached to the first one as slave. Each instance streaming to a different GPS.

  4. When we press ARM + Start, they synchronize and start RF streaming, but we have realized, and here is the issue, that Skydel has a heavy latency with the computer time with the time that passes when you arm+sync+start. We have seen a delay of 12, 10 and 8 seconds between different simulations. We need skydel’s RF streaming date to be adjusted to PTP time, so our whole system has PTP time synchronized and we need to have all our messages correctly timataged. (And with low latency between them).

Attached is a picture where you can see the 8 seconds delay.

Thank you in advance!

Hi @Baifo,

The computer time feature is currently not designed for your use case, as stated in the User Manual:

8.7.2.2. Current Computer Time

If you select Current Computer Time, your simulation will be synchronized to the computer system time to within approximately 10 seconds. This is due to the time it takes to start the streaming process with the radios. Synchronizing in this way can be useful if you want your simulation to be close to true time. However, for tight synchronization with a time reference, it is necessary to use a timing receiver.

I tried my best to find a way to achieve your goal with the current Skydel API, see the following example script.

Notes:

  • I don’t use computer time nor master/slave
  • The first Skydel instance uses Dektec card 0, second instance uses card 1
  • I have to get the leap second to replicate the computer time behavior
  • You can probably reduce the StartPPS sync time of 5000ms
  • If your time server is not providing the PPS reference to the dektec cards, you will see a constant offset of less than 1 second between the start time and the computer time. You can compensate for this offset by changing the variable ppsOffset. To get the offset between the PPS and the OS time, you can run the second script provided below.
import time
from datetime import datetime
from datetime import timedelta
from skydelsdx import *
from skydelsdx.commands import *

def initSim(id):
  sim = RemoteSimulator(True)
  sim.connect(id=id)
  sim.call(New(True, False))
  sim.call(SetModulationTarget("DTA-2115B", "", str(id), True, "dektec"))
  sim.call(ChangeModulationTargetSignals(0, 12500000, 85000000, "UpperL", "L1CA", 50, True, "dektec", None))
  sim.call(SetDuration(60))
  return sim

ppsOffset = 0

sim = initSim(0)
sim2 = initSim(1)

leapSecond = sim.call(GetOfficialLeapSecond(datetime.utcnow())).leapSecond()

sim.call(ArmPPS())
sim2.call(ArmPPS())

sim.post(WaitAndResetPPS())
sim2.call(WaitAndResetPPS())

time.sleep(ppsOffsetMs / 1000.0)

start = datetime.utcnow()
start += timedelta(0, leapSecond)

sim.call(SetPps0GpsTime(start))
sim2.call(SetPps0GpsTime(start))

sim.post(StartPPS(5000 + ppsOffsetMs))
sim2.call(StartPPS(5000 + ppsOffsetMs))

sim.disconnect()
sim2.disconnect()

The following script is to get the PPS offset with your OS time if the PPS reference is not provided by the time server (such as a secure sync).

Note this offset will drift over a long period of time, and change if you reboot the GSG-8.
You also have to run this script directly on the GSG-8.

import math
from skydelsdx import *
from skydelsdx.commands import *

sim = RemoteSimulator(True)
sim.connect()

sim.call(New(True, False))
sim.call(SetModulationTarget("DTA-2115B", "", "", True, "dektec"))
sim.call(ChangeModulationTargetSignals(0, 12500000, 85000000, "UpperL", "L1CA", 50, True, "dektec", None))

sim.call(ArmPPS())
sim.call(WaitAndResetPPS())
sim.call(StartPPS(5000))

timestamp = sim.call(GetComputerSystemTimeSinceEpochAtPps0()).milliseconds() / 1000.0
ppsOffsetMs = 1000.0 - (timestamp - int(timestamp)) * 1000.0

print(math.ceil(ppsOffsetMs))

sim.stop()
sim.disconnect()

Please keep me updated once you have tried this solution,
Best regards,
Mathieu

1 Like

Hi Mathieu,
thank you for your reply and your effort doing the example.
I could not save the delay with the method that you said, SetPps0GpsTime, I still have around 5 seconds of difference between my simulation time and my computer time.

In my system I have the computer time synchronized with PTP and the RF cards triggered by PPS from the PTP.
Instance 1 uses two DTA-2115B card and Instances 2 another pair of DTA cards. Each instance simulates a different GPS receiver.

My settings for start PPS:
syncDurationMs=2000

I have noted that if I try to force a different value for start, nothing happens. I have the same result without you code, with your code, and with that one trying to add 5 seconds:
start = datetime.utcnow()
start += timedelta(0, leapSecond+5)
sim.call(SetPps0GpsTime(start))
sim.call(StartPPS(2000))

Do you know what else can I do?

Best regards!

Hi @Baifo,

If your PTP server is also providing the PPS reference to the dektec cards, then you don’t need to use the second script I provided to get the offset.

I have noted that if I try to force a different value for start, nothing happens

What do you mean nothing happens? Do you get an error?
If you don’t get an error in the script can you look at Skydel automation tab and make sure all commands were successful.
Note that the True parameter given to RemoteSimulator is to raise exceptions on errors.

Also what version of Skydel are you using? I tested this script on Skydel version 22.7.1.
Note I did find a bug in my script where you can sometimes have a difference of 1 second between the two simulations.

Here’s the updated script:

import time
from datetime import datetime
from datetime import timedelta
from skydelsdx import *
from skydelsdx.commands import *

def initSim(id):
  sim = RemoteSimulator(True)
  sim.connect(id=id)
  sim.call(New(True, False))
  sim.call(SetModulationTarget("DTA-2115B", "", str(id), True, "dektec"))
  sim.call(ChangeModulationTargetSignals(0, 12500000, 85000000, "UpperL", "L1CA", 50, True, "dektec", None))
  return sim

simStartTimeMs = 5000
simDurationSec = 30

sim = initSim(0)
sim2 = initSim(1)

leapSecond = sim.call(GetOfficialLeapSecond(datetime.utcnow())).leapSecond()

sim.call(ArmPPS())
sim2.call(ArmPPS())

try:
  sim.call(WaitAndResetPPS())

  start = datetime.utcnow()
  start += timedelta(0, leapSecond)
  timestamp = time.time()

  sim2.call(WaitAndResetPPS())

  offsetSec = round(time.time() - timestamp)
  offsetMs = offsetSec * 1000

  if offsetMs > simStartTimeMs:
    raise Exception("Increase simStartTimeMs")

  sim.call(SetPps0GpsTime(start))
  sim2.call(SetPps0GpsTime(start + timedelta(0, offsetSec)))

  sim.post(StartPPS(simStartTimeMs))
  sim2.call(StartPPS(simStartTimeMs - offsetMs))

  sim.stop(simDurationSec)
  sim2.stop(simDurationSec)

except:
  sim.stop()
  sim2.stop()
  raise
  
finally:
  sim.disconnect()
  sim2.disconnect()

Best regards,
Mathieu

1 Like

Dear Mathieu,
sorry for my late reply, It has been crazy days at work. We are using Skydel 22.7.0
Reading all your steps again, I managed to modify the simulation time partially, only with the master. What happened to me, is that I did not remove the line “use current computer time” and SetPps0GpsTime was doing nothing. But after removing that line from the code, it works very well on master. But what is happening is that it does not work in slave.
When I try to use that command in Slave, i got a command exception that says: “SetPps0GpsTime failed: Unexpected call to SetPps0GpsTime”.

What can we do?? We really need the master/slave arquitecture for this HIL testing, and we do need to synchronize the PPS0 GPS time after arm and wait and reset pps to use the current computer time in both, master and slave.

Your approach/example was to use two remote simulator independents, but we need to use a master/slave combinations.

I share with you my code to make it more clear. You can use ctrl + F in my code and search for @Mathieu to see the critics code lines

#!/usr/bin/python3

# This Python script illustrates how to send the receiver trajectory in real-time 
# using the hardware-in-the-loop (HIL) feature in Master/Slave mode.
#
# Before running this script, make sure Skydel is running, and the splash screen is closed.
#
# There are two modes of operation with this script:
#
#   1 - You run this script on the same PC as Skydel, and haven't setup time synchronization
#       between the computer system time and the PPS signal driving your radio (or you run in NoneRT).
#       
#       This is the default use case (when the variable isOsTimeSyncWithPPS is False), it exists
#       to allow users to quickly and easily test HIL without having to set up time synchronization.
#       Note that if you use this mode with a radio, the time will drift between this script and 
#       the Skydel's simulation over time.
#
#   2 - You run this script on any PC which has it's time synchronized with the radio PPS signal.
#       
#       This is the recommended use case (when the variable isOsTimeSyncWithPPS is True).
#       We recommend using a time server, such as the SecureSync 2400 to provide the 10Mhz
#       and the PPS reference to the radio. The SecureSync is also a PTP server that can 
#       synchronize your computer system clock with the PPS to a high degree of precision.
#       In this mode, there will be no time drift between the script and the Skydel's simulation.
#
# Additional note: the script doesn't change the Skydel's engine latency by default,
# as this is a system wide preference. To set the preference, you can uncomment the line
# in the script. We recommend you set it back to the default value of 200ms once you are done
# using this script, unless you only plan to do low latency HIL on this machine.

#import sys
import skydelsdx
from datetime import datetime
from datetime import timedelta

from skydelsdx.commands import *
from example_hil_helper import *


def checkMasterConnection(sim, nbSlaveExpected):
  getMasterStatusResult = sim.call(GetMasterStatus())

  # trying to wait few seconds expecting connection may not be set up yet.
  if not getMasterStatusResult.isMaster() or getMasterStatusResult.slaveConnected() != nbSlaveExpected:
    time.sleep(1)
    getMasterStatusResult = sim.call(GetMasterStatus())

  if not getMasterStatusResult.isMaster():
    raise RuntimeError("Simulator is not Master")
  if getMasterStatusResult.slaveConnected() != nbSlaveExpected:
    raise RuntimeError("Only {0} connection, while {1} slave connections were expected".format(getMasterStatusResult.slaveConnected(), nbSlaveExpected))


def checkSlaveConnection(sim):
  getSlaveStatusResult = sim.call(GetSlaveStatus())

  if getSlaveStatusResult.isSlave() == False:
    raise RuntimeError("Simulator is not Slave")
  if getSlaveStatusResult.isConnected() == False:
    raise RuntimeError("Simulator is not Connected")


def setupSimulator(skydelIpAddress, simInstanceID, skydelEngineLatencyMs, hilTjoin, radioConfig):
  sim = skydelsdx.RemoteSimulator()
  sim.setVerbose(True)
  sim.connect(skydelIpAddress, simInstanceID)

  # Check the engine latency (Skydel's system wide preference)
  if sim.call(GetEngineLatency()).latency() != skydelEngineLatencyMs:
    sim.call(SetEngineLatency(skydelEngineLatencyMs))  # Uncomment this line to set the engine latency preference
    #error("Please execute the SetEngineLatency({0}) command or change the skydelEngineLatencyMs value before executing this script.".format(skydelEngineLatencyMs))

  # Check the streaming buffer preference, do not change it from its default value
  if sim.call(GetStreamingBuffer()).size() != 200:
    error("Please do not change the Streaming Buffer preference.")

  # Uncomment these lines if you do very low latency HIL, as these features can impact Skydel's performance (Skydel's system wide preferences)
  # sim.call(ShowMapAnalysis(False))
  # sim.call(SetSpectrumVisible(False))

  # Create new config, ignore the default config if it's set
  sim.call(New(True, False))

  # Change the output
  for radio in radioConfig:
      sim.call(SetModulationTarget(type=radioConfig[radio]["radioType"], path="", address=radioConfig[radio]["radioNumber"], clockIsExternal=True, id=radioConfig[radio]["uniqueRadioId"]))
      sim.call(ChangeModulationTargetSignals(output=0, minRate=12500000, maxRate=85000000, band=radioConfig[radio]["outputType"], signal=radioConfig[radio]["signal"], gain=50, gaussianNoise=True, id=radioConfig[radio]["uniqueRadioId"], centralFrequency=None))
      sim.call(SetGpu(gpuIdx=radioConfig[radio]["GPU"], output=0, id=radioConfig[radio]["uniqueRadioId"]))
      
  # Enable some logging
  sim.call(EnableLogRaw(True))  # You can enable raw logging and compare the logs (the receiver position is especially helpful)
  sim.call(EnableLogHILInput(True))  # This will give you exactly what Skydel has received through the HIL interface

  # Change the vehicle's trajectory to HIL
  sim.call(SetVehicleTrajectory("HIL"))
  sim.call(ImportConstellationParameters("GPS", "/home/skydel/Documents/Skydel-SDX/Almanacs/ANK200TUR_S_20222910000_01D_GN.rnx", None, None))
  
  # ionosperic model "nequick"
  sim.call(SetIonoModel("NeQuick"))
  #sim.call(SetStartTimeMode("Computer")) #We have commented this line to make it work with SetPPs0GPStime, @Mathieu
  
  
  # HIL Tjoin is a volatile parameter that must be set before every HIL simulation
  sim.call(SetHilTjoin(hilTjoin))

  # The streaming check is performed at the end of pushEcefNed. It's recommended to disable this check 
  # and do it asynchronously outside of the while loop when sending positions at high frequencies.
  sim.setHilStreamingCheckEnabled(True)

  return sim


# Change these as required
masterTrajectory = StraightTrajectory(speed=10.0, latDeg=37.21679722, lonDeg=-7.18851111, alt=2000)
slaveTrajectory = StraightTrajectory(speed=10.0, latDeg=37.21679722, lonDeg=-7.18851111, alt=1900)
simDurationMs = 36000000
syncDurationMs = 2000

# If this script isn't running on the same PC as the Skydel instances, set to the Skydel instances IP addresses
masterIpAddress = "127.0.0.1"
slaveIpAddress = "127.0.0.1"
masterInstanceId = 0
slaveInstanceId = 1
masterPort = 4567

# Set to True if the computer which runs this script has it's time synchronized with the output radio PPS
isOsTimeSyncWithPPS = True

if (masterIpAddress != "127.0.0.1" or slaveIpAddress != "127.0.0.1") and not isOsTimeSyncWithPPS:
  error("Can't run this script on a different computer if the OS time isn't in sync with the radios PPS.")

# We suggest these values as a starting point, but they will have to be modified according 
# to your hardware, the configuration of the simulation and your requirements.
# Use the performance graph as well as the HIL graph to monitor Skydel and diagnose issues.
# It is strongly recommended to read the user manual before you try to optimize those settings.
timeBetweenPosMs = 10  # Send receiver position every 15 milliseconds
skydelEngineLatencyMs = 10  # How much in advance can Skydel be versus the radio time
hilTjoin = 20  # This value should be greater than skydelEngineLatencyMs + timeBetweenPosMs + network latency

# Setup master simulator
master_radio_config = {
    "radio_1": {
        "radioType" : "DTA-2115B",
        "radioNumber" : "0",
        "uniqueRadioId": "master-01",
        "outputType": "UpperL",
        "signal": "L1CA",
        "GPU": 1,
        },
    "radio_2": {
        "radioType" : "DTA-2115B",
        "radioNumber" : "1",
        "uniqueRadioId": "master-02",
        "outputType": "LowerL",
        "signal": "L2P",
        "GPU": 1,
        },
    }
simMaster = setupSimulator(masterIpAddress, masterInstanceId, skydelEngineLatencyMs, hilTjoin, master_radio_config)
simMaster.call(SetSyncServer(masterPort))
simMaster.call(EnableMasterPps(True))
simMaster.call(SetConfigBroadcastFilter([ConfigFilter.Radios, ConfigFilter.OutputAndRadios])) # we tried to automation in the broadcast but in the moment that you arm the pps, the broadcast happended. You can not broadcast after the symten is armed. For that reason, the GPSpps0time is not synchronized with the slave @Mathieu
simMaster.call(SetConfigBroadcastOnStart(True)) #@Mathieu same as above line


# Setup slave simulator
slave_radio_config = {
    "radio_1": {
        "radioType" : "DTA-2115B",
        "radioNumber" : "2",
        "uniqueRadioId": "slave-01",
        "outputType": "UpperL",
        "signal": "L1CA",
        "GPU": 0,
        },
    "radio_2": {
        "radioType" : "DTA-2115B",
        "radioNumber" : "3",
        "uniqueRadioId": "slave-02",
        "outputType": "LowerL",
        "signal": "L2P",
        "GPU": 0,
        },
    }
simSlave = setupSimulator(slaveIpAddress, slaveInstanceId, skydelEngineLatencyMs, hilTjoin, slave_radio_config)
simSlave.call(SetSyncClient(masterIpAddress, masterPort))  
simSlave.call(EnableSlavePps(True))

# check connection between slave and master
checkMasterConnection(simMaster, 1)
checkSlaveConnection(simSlave)

# From here we want to make sure to stop the simulation if something goes wrong
try:
  #simMaster.call(BroadcastConfig())
  # Arm the simulator, when this command returns, we can start synchronizing with the PPS
  simMaster.call(ArmPPS())

  # The WaitAndResetPPS command returns immediately after a PPS signal, which is our PPS reference (PPS0)
  simMaster.call(WaitAndResetPPS())
  
  
  
  # If our PC clock is synchronized with the PPS, the nearest rounded second is the PPS0
  if isOsTimeSyncWithPPS:
    pps0TimestampMs = getClosestPpsTimeMs()
    print(pps0TimestampMs)
    

  start = datetime.utcnow()
  start += timedelta(0,18)
  simMaster.call(SetPps0GpsTime(start))
  #simSlave.call(SetPps0GpsTime(start))# @Mathieu we can not execute this line and we got the error that is described in the post, so, our master has good time, and slave has not synchronized its time
  
 
  


  # The command StartPPS will start the simulation at PPS0 + syncDurationMs
  # You can synchronize with your HIL simulation start, by changing the value of syncDurationMs (resolution in milliseconds)
  #simMaster.call(SetPps0GpsTime(datetime.utcnow()))
  simMaster.call(StartPPS(syncDurationMs))
  # If the PC clock is NOT synchronized with the PPS, we can ask Skydel to tell us the PC time corresponding to PPS0
  if not isOsTimeSyncWithPPS:
    pps0TimestampMs = simMaster.call(GetComputerSystemTimeSinceEpochAtPps0()).milliseconds()

  # Compute the timestamp at the beginning of the simulation
  simStartTimestampMs = pps0TimestampMs + syncDurationMs

  # We send the first position outside of the loop, so initialize this variable for the second position
  nextTimestampMs = simStartTimestampMs + timeBetweenPosMs

  # Keep track of the simulation elapsed time in milliseconds
  elapsedMs = 0.0
  
  # Skydel must know the initial position of the receiver for initialization
  masterPosition, masterVelocity = masterTrajectory.generateEcefWithDynamicsGoingEast(elapsedMs)
  slavePosition, slaveVelocity = slaveTrajectory.generateEcefWithDynamicsGoingEast(elapsedMs)
  simMaster.pushEcef(elapsedMs, masterPosition, masterVelocity)
  simSlave.pushEcef(elapsedMs, slavePosition, slaveVelocity)
  #simMaster.post(EnableRFOutputForSV("GPS", 7, False),120)
  #simMaster.post(EnableRFOutputForSV("GPS", 7, True),120*3)
  #simMaster.post(EnableSignalForSV("L1CA", 7, False),20)

  # Send positions in real time until the elapsed time reaches the desired simulation duration
  while elapsedMs <= simDurationMs:

    # Wait for the next position's timestamp
    preciseSleepUntilMs(nextTimestampMs)
    nextTimestampMs += timeBetweenPosMs
    
    # Get the current elapsed time in milliseconds
    elapsedMs = getCurrentTimeMs() - simStartTimestampMs

    
    masterPosition, masterVelocity = masterTrajectory.generateEcefWithDynamicsGoingEast(elapsedMs)
    slavePosition, slaveVelocity = slaveTrajectory.generateEcefWithDynamicsGoingEast(elapsedMs)

    # Push the positions to Skydel
    simMaster.pushEcef(elapsedMs, masterPosition, masterVelocity)
    simSlave.pushEcef(elapsedMs, slavePosition, slaveVelocity)

finally:
  # Stop the simulation
  simMaster.stop()

  # Disconnect from Skydel
  simMaster.disconnect()
  simSlave.disconnect()

Best regards

Hi Baifo,

Today it is not possible to achieve what you want in Master/Slave, this is why I proposed to use two Skydel instances. Why do you need the Master/Slave architecture?

We currently have a bug opened on our side to fix the issue with SetPps0GpsTime not affecting slave instances, it should get fixed in the near future.

Best regards,
Mathieu

Hi Mathieu,
We do need the master/slave configuration because we are using a differential receiver system, and we need that both receivers see the same signal synchronized with PPS.
Pierre-Marie @pmleveel has the details, in case that you need more information about our situation.
When are you planning to fix the issue?

Best regards

Hi Baifo,

While it is more convenient to use Master/Slave, the solution I provided also synchronizes both simulators on the PPS. There should be no difference there. What’s important is that all your radios use the same PPS and 10MHz source.

I can’t give you a date, but we are aiming by the end of the month.

Best regards.

Hi @Baifo,

We just released Skydel 22.7.2, you can get the download here.
The issue with the command SetPps0GpsTime is fixed in this version.

Here’s an example of how to use the command in Master / Slave.
Note that I only send the command to the Master instance.

import time
from datetime import datetime
from datetime import timedelta
from skydelsdx import *
from skydelsdx.commands import *

def initSim(id, type):
  sim = RemoteSimulator(True)
  sim.connect(id=id)
  sim.call(New(True, False))
  sim.call(SetModulationTarget(type, "", str(id), True, "radio"))
  sim.call(ChangeModulationTargetSignals(0, 12500000, 85000000, "UpperL", "L1CA", -1, True, "radio", None))
  return sim

simStartTimeMs = 5000
simDurationSec = 30

sim = initSim(0, "DTA-2115B")
sim2 = initSim(1, "DTA-2115B")

sim.call(EnableMasterPps(True))
sim2.call(EnableSlavePps(True))
time.sleep(3)

leapSecond = sim.call(GetOfficialLeapSecond(datetime.utcnow())).leapSecond()

sim.call(ArmPPS())

try:
  sim.call(WaitAndResetPPS())
  start = datetime.utcnow() + timedelta(0, leapSecond) # UTC to GPS time
  sim.call(SetPps0GpsTime(start))
  sim.post(StartPPS(simStartTimeMs))
  sim.stop(simDurationSec)

except:
  sim.stop()
  raise
  
finally:
  sim.disconnect()
  sim2.disconnect()

Hope this fixes your issue,
Best regards

1 Like

Thank you Mathieu and all the team for the fast support. We are out of office right now, but we will test it ASAP.
Best regards.

2 Likes

Hi @mathieu.favreau , I confirm that with the new update, everything works properly in a master/slave configuration! It works as we expected.
Best regards!

1 Like

I am glad it resolved your issue!
Best regards