Teleop guide (beyond-line-of-sight teleoperation)

⚠️ Not yet released: in-development v1.5.0-SNAPSHOT capability; interface and protocol may change. Do not use in production before the official release.

New in v1.5.0. Teleop = beyond-line-of-sight remote takeover: a remote operator drives the robot's voice interaction over SDK, independent of local visibility. It is the third interaction mode alongside passive / active, using its own event family and the AgentPolicy.TELEOP policy.

When to use

• The robot is unattended or in poor visibility / network conditions and a human operator needs to take over specific interactions.
• The application needs to bypass the LLM and let a human voice-drive a short window (soothing complaints, verifying orders, etc.).
• The operator's audio is uploaded via SDK; the gateway routes it to the robot identified by agentId.

Flow

Strict order: enter → any number of startAudio / appendAudioDelta / doneAudio → exit. keepalive may be called at any frequency during the session (recommended every 10 seconds, see Robot online detection (keepalive)).

              enter                      exit
   ┌──────────────────────┐       ┌──────────────────────┐
   │  TeleopRequest.enter │       │  TeleopRequest.exit  │
   │     (onEnterAck)     │       │     (onExitAck)      │
   └──────────┬───────────┘       └──────────┬───────────┘
              │  any number of times          │
              ▼   (fire-and-forget)           │
   ┌──────────────────────┐                   │
   │  startAudio(param)   │                   │
   │  appendAudioDelta(*) │───────────────────┘
   │  doneAudio           │
   └──────────────────────┘

   ┌──────────────────────┐
   │ TeleopRequest.keepalive │ ← every 10 s recommended; `code != 0` means offline
   │   (onKeepaliveAck)      │
   └──────────────────────┘

• enter / keepalive / exit have ack callbacks. code == 0 = success, code == -1 = server rejected (msg carries errorMsg), code == 1000 = local connection closed (SDK-side error, callback fires synchronously on caller thread).
• startAudio / appendAudioDelta / doneAudio are fire-and-forget (no ACK).

Event family & wire format

Every teleop envelope carries:

{
  "type":      "<event type>",
  "agentId":   "<target robot agentId>",
  "teleopId":  "teleop_<21 random chars>",
  "eventId":   "event_<21 random chars>",
  "agentMode": "teleop"
}

teleopId is generated by IdGenerator.generateTeleopId(). agentMode is always "teleop" (matches AgentPolicy.TELEOP).

startAudio parameter

startAudio(AgentParam) accepts an AgentParam per audio segment for voice / VAD tuning:

Java construction:

AgentParam startParam = AgentParam.create();
startParam.setString("role", "male");
startParam.setInteger("threshold", 0);
String eventId = teleopRequest.startAudio(startParam);

On the wire these travel as param.extParam.role / param.extParam.threshold (matching how other AgentParam are serialised).

Robot online detection (keepalive)

A teleop session usually lasts a while, during which the robot may go offline due to network loss, low battery, or on-site power failure. The SDK exposes TeleopRequest.keepalive(...) as a one-shot probe. Applications should call it periodically (recommended every 10 seconds) to check whether the robot is still online.

Signature & callback

public String keepalive(TeleopKeepaliveCallback keepaliveCallback, long timeout);

public abstract class TeleopKeepaliveCallback {
    public abstract void onKeepaliveAck(String teleopId, String eventId, int code, String msg);
}

• timeout is a local placeholder (< 50 is clamped to 50, same as enter/exit).
• Returns the outgoing message's eventId.
• Each call sends one agentsdk.teleop.keepalive message; the gateway routes it to the robot identified by teleopId; the robot responds with success == true when online.

code semantics (same as enter / exit)

Periodic probe + auto exit on offline

Any code != 0 should be treated as "robot went offline" — stop the timer, exit the session, and unregisterTeleop to release resources, so no further appendAudioDelta is wasted on a dead session.

import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;

ScheduledExecutorService keepaliveScheduler = Executors.newSingleThreadScheduledExecutor(r -> {
    Thread t = new Thread(r, "teleop-keepalive");
    t.setDaemon(true);
    return t;
});
AtomicBoolean robotOnline = new AtomicBoolean(true);

// After enter succeeds, fire a probe every 10 seconds
keepaliveScheduler.scheduleAtFixedRate(() -> {
    if (!robotOnline.get()) return;
    teleopRequest.keepalive(new TeleopKeepaliveCallback() {
        @Override
        public void onKeepaliveAck(String teleopId, String eventId, int code, String msg) {
            if (code == 0) return;
            if (!robotOnline.compareAndSet(true, false)) return;
            log.warn("robot offline => code={}, msg={}", code, msg);
            keepaliveScheduler.shutdown();
            teleopRequest.exit(new TeleopExitCallback() {
                @Override
                public void onExitAck(String teleopId, String eventId, int code, String msg) {
                    agentSdk.unregisterTeleop(teleopId);
                }
            }, 2000);
        }
    }, 2000);
}, 10, 10, TimeUnit.SECONDS);

Also keepaliveScheduler.shutdown() when your business logic invokes exit, to stop further probes.

API cheat-sheet

• Request class: com.agibot.aiem.sdk.teleop.TeleopRequest
• Callback bases: com.agibot.aiem.sdk.teleop.TeleopEnterCallback / TeleopKeepaliveCallback / TeleopExitCallback
• Register: AgentSdk.registerTeleop(TeleopRequest) / AgentSdk.unregisterTeleop(String teleopId)
• ID: IdGenerator.generateTeleopId()

Full signatures: Teleop API. Runnable example: Teleop example.

Lifecycle & cleanup

• The SDK maintains active requests keyed by teleopId. Always registerTeleop(...) before calling enter.
• Every request method call refreshes an internal updateTs; requests idle for 7200 s are GC'd by a background sweeper.
• After exit (or when abandoning the session), call agentSdk.unregisterTeleop(teleopId) to release eagerly.
• AgentSdk.release() clears all teleop requests owned by this instance.

FAQ

1. onEnterAck returns code == 1000 even though I did not close the SDK. The underlying WebSocket is disconnected or has not yet come up. Wait for AgentAuthCallback.onAuthState(code == 0) before enter; if you land on code == 1000, the SDK will reconnect automatically — retry enter after reconnection.
2. How long should each appendAudioDelta chunk be? Same as the passive audio path — 20 ms – 40 ms of PCM/Opus, base64-encoded. Larger chunks may starve the WS frame budget; smaller ones inflate message count.
3. Must I call startAudio immediately after enter? No. enter ack only means the teleop session is ready; you can push audio at any later moment (e.g. after the on-site customer starts talking).
4. How many startAudio / doneAudio pairs may live inside a single teleop session? As many as needed — each spoken turn is startAudio → appendAudioDelta × N → doneAudio. No need to exit and enter again; use a fresh eventId (returned by startAudio) per segment.