AIMA开放平台文档中心Agent SDK (Java)

Passive Interaction Examples

Passive Interaction Examples

The 8 passive examples are grouped into three categories by input/output complexity. Every *Example.java file is a self-contained main; only the register* method and the onRequest signature differ. Everything else — SDK creation, auth registration, shared base callbacks, initialization — is identical.

Shared skeleton (used by every example)

java
String url = "wss://open.agibot.com/api/V1/open-portal/app/wss/agent-sdk";
String appId = "<your appId from the LinkSoul open platform>";
String appKey = "<your appKey from the LinkSoul open platform>";
String appSecret = "<your appSecret from the LinkSoul open platform>";

// Step 1: create the SDK instance (singleton per appId)
AgentSdk agentSdk = AgentSdk.create(url, appId, appKey, appSecret);

// Step 2: register the auth callback
agentSdk.registerAuth(new AgentAuthCallback() {
    @Override
    public void onAuthState(String appId, int code, String msg) {
        log.info("onAuthState => appId: {}, code: {}, msg: {}", appId, code, msg);
    }
});

// Step 3: register your business callback (see each example below)
agentSdk.registerXxx(new XxxCallback(agentSdk) { /* ... */ });

// Step 4: initialize (opens the WebSocket, auto-reconnect)
agentSdk.initialize();

Shared base callbacks (v1.4.0+)

Every *Callback extends PassiveCallback and may optionally override the following methods (they have safe no-op log defaults):

java
@Override
public void onRobotOnline(String agentId, AgentMeta agentMeta) {
    // Robot online — initialize per-robot resources
    log.info("onRobotOnline => agentId: {}, agentMeta: {}", agentId, agentMeta);
}

@Override
public void onRobotOffline(String agentId) {
    // Robot offline — release per-robot resources
    log.info("onRobotOffline => agentId: {}", agentId);
}

@Override
public void onFaceInfo(String agentId, String eventId, AgentParam param) {
    // Face / voiceprint UID detected (opened in v1.4.0)
    log.info("onFaceInfo => agentId {}, eventId {}, param {}", agentId, eventId, param);
}

@Override
public void onVideoFrame(String agentId, String eventId, int flag, byte[] buf,
                         boolean isKeyFrame, long localTs, AgentParam param) {
    // Video passthrough: flag=2 H264, flag=3 image
    log.info("onVideoFrame => agentId {}, eventId {}, flag {}, isKeyFrame {}, localTs {}, param {}",
        agentId, eventId, flag, isKeyFrame, localTs, param);
}

@Override
public void onGreetSignal(String agentId, String eventId, AgentParam param, GreetResponse response) {
    // Greeting signal; reply via response.onGreetVlm*/onGreetTts*
    log.info("onGreetSignal => agentId {}, eventId {}, param {}", agentId, eventId, param);
}

@Override
public void onState(String agentId, String eventId, String stateName, String stateValue) {
    // Robot state push (merged here in v1.4.0; no more separate StateListenCallback)
    log.info("onState => agentId {}, eventId {}, stateName {}, stateValue {}",
        agentId, eventId, stateName, stateValue);
}

@Override
public void onHistories(String agentId, String eventId, List<HistoryInfo> histories) {
    // New in v1.5.0: conversation history forwarded by the gateway.
    // De-duplicate by eventId, sort by timestamp, then feed to the XLM as context.
    log.info("onHistories => agentId {}, eventId {}, histories {}",
        agentId, eventId, histories == null ? "[]" : JSON.toJSONString(histories));
}

onHistories is a v1.5.0 shared base callback; HistoryInfo fields are listed in the Function Call / history section. Each example below shows only the register* call and the body of onRequest. Wrap it with the shared skeleton above to get a runnable program.

1. Pure audio

Audio2LlmExample — audio in → ASR + LLM text out

Source: Audio2LlmExample.java

Scenario: voice conversation with text-only reply (no TTS).

java
agentSdk.registerAudio2Llm(new Audio2LlmCallback(agentSdk) {

    /**
     * @param flag  0 VAD start, 1 audio chunk, 2 VAD end
     * @param buf   audio bytes (valid when flag == 1)
     */
    @Override
    public void onRequest(String agentId, String eventId, int flag,
                          byte[] buf, AgentParam param, Audio2LlmResponse response) {

        // Optional: agent metadata
        AgentMeta agentMeta = agentSdk.getAgentMeta(agentId);
        String defWakeupWord = agentMeta.getWakeupWord();
        String yourParamKey = agentMeta.getString("your_custom_key");

        // 1) ASR by flag
        if (flag == 0) {
            // init ASR client
        } else if (flag == 1) {
            // feed audio chunk to ASR
        } else if (flag == 2) {
            // VAD end
        }
        response.onAsrMiddle(eventId, "today");
        response.onAsrFinal(eventId, "what's the weather like today");

        // 2) 语义理解 / LLM streaming
        String itemId = "itemId_000001";

        // Optional barge-in
        response.onInterrupt(eventId, "chat", null);

        // Optional skill dispatch
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        String textDelta = "the LLM streaming text goes here";
        response.onLlmItemDelta(eventId, itemId, textDelta);
        response.onLlmItemDone(eventId, itemId);
        response.onLlmDone(eventId);

        // Error handling
        // response.onError(eventId, 1010, "ASR call failed");
    }

    // ... shared base callbacks ...
});

Audio2TtsExample — audio in → ASR + LLM + TTS full pipeline

Source: Audio2TtsExample.java

Scenario: most common voice-in → voice-out conversational loop. The onRequest signature carries an extra itemId compared to Audio2Llm.

java
agentSdk.registerAudio2Tts(new Audio2TtsCallback(agentSdk) {

    /**
     * @param itemId for multi-intent replies
     * @param flag   0 VAD start, 1 audio chunk, 2 VAD end
     * @param buf    audio bytes (flag == 1)
     */
    @Override
    public void onRequest(String agentId, String eventId, String itemId, int flag,
                          byte[] buf, AgentParam param, Audio2TtsResponse response) {

        // 1) ASR
        if (flag == 0) {
            // init ASR
        } else if (flag == 1) {
            // feed audio
        } else if (flag == 2) {
            // VAD end
        }
        response.onAsrMiddle(eventId, "today");
        response.onAsrFinal(eventId, "what's the weather like today");

        // 2) LLM streaming
        String textDelta = "the LLM streaming text goes here";
        response.onLlmItemDelta(eventId, itemId, textDelta);
        response.onLlmItemDone(eventId, itemId);
        response.onLlmDone(eventId);

        // 3) Optional barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // 4) TTS streaming
        byte[] audio = new byte[1024];
        response.onTtsItemDelta(eventId, itemId, audio);
        response.onTtsItemDone(eventId, itemId);
        response.onTtsDone(eventId);

        // Error handling
        // response.onError(eventId, 1010, "ASR call failed");
    }

    // ... shared base callbacks ...
});

2. Pure text

Asr2LlmExample — ASR text in → LLM out

Source: Asr2LlmExample.java

Scenario: ASR done upstream; you only need 语义理解/LLM.

java
agentSdk.registerAsr2Llm(new Asr2LlmCallback(agentSdk) {

    @Override
    public void onRequest(String agentId, String eventId, String text,
                          AgentParam param, Asr2LlmResponse response) {

        // Run 语义理解 + LLM on `text`
        response.onInterrupt(eventId, "chat", null);

        String itemId = "itemId_000001";
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        String textDelta = "the LLM streaming text goes here";
        response.onLlmItemDelta(eventId, itemId, textDelta);
        response.onLlmItemDone(eventId, itemId);
        response.onLlmDone(eventId);

        // response.onError(eventId, 1010, "...");
    }

    // ... shared base callbacks ...
});

Asr2TtsExample — ASR text in → LLM + TTS out

Source: Asr2TtsExample.java

Scenario: ASR done upstream; downstream needs voice reply.

java
agentSdk.registerAsr2Tts(new Asr2TtsCallback(agentSdk) {

    @Override
    public void onRequest(String agentId, String eventId, String text,
                          AgentParam param, Asr2TtsResponse response) {

        String itemId = "itemId_000001";

        // LLM
        String textDelta = "the LLM streaming text goes here";
        response.onLlmItemDelta(eventId, itemId, textDelta);
        response.onLlmItemDone(eventId, itemId);
        response.onLlmDone(eventId);

        // Barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // TTS
        byte[] audio = new byte[1024];
        response.onTtsItemDelta(eventId, itemId, audio);
        response.onTtsItemDone(eventId, itemId);
        response.onTtsDone(eventId);
    }

    // ... shared base callbacks ...
});

3. Multimodal (text/audio + video)

AsrVideo2VlmExample — ASR text + video → VLM out

Source: AsrVideo2VlmExample.java

Scenario: visual question answering.

java
agentSdk.registerAsrVideo2Vlm(new AsrVideo2VlmCallback(agentSdk) {

    /**
     * @param flag  3 ASR text, 4 H264, 5 image
     * @param text  valid when flag == 3
     * @param buf   valid when flag == 4 or 5
     */
    @Override
    public void onRequest(String agentId, String eventId, int flag, String text,
                          byte[] buf, AgentParam param, AsrVideo2VlmResponse response) {

        // Run VLM on `text` plus the buffered video frame
        String itemId = "itemId_000001";

        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        String textDelta = "the LLM streaming text goes here";
        response.onVlmItemDelta(eventId, itemId, textDelta);
        response.onVlmItemDone(eventId, itemId);
        response.onVlmDone(eventId);
    }

    // ... shared base callbacks ...
});

AsrVideo2TtsExample — ASR text + video → VLM + TTS out

Source: AsrVideo2TtsExample.java

Scenario: visual QA with spoken reply.

java
agentSdk.registerAsrVideo2Tts(new AsrVideo2TtsCallback(agentSdk) {

    @Override
    public void onRequest(String agentId, String eventId, int flag, String text,
                          byte[] buf, AgentParam param, AsrVideo2TtsResponse response) {

        String itemId = "itemId_000001";
        String textDelta = "the LLM streaming text goes here";

        // 1) VLM
        response.onVlmItemDelta(eventId, itemId, textDelta);
        response.onVlmItemDone(eventId, itemId);
        response.onVlmDone(eventId);

        // 2) Barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // 3) TTS
        byte[] audio = new byte[1024];
        response.onTtsItemDelta(eventId, itemId, audio);
        response.onTtsItemDone(eventId, itemId);
        response.onTtsDone(eventId);
    }

    // ... shared base callbacks ...
});

AudioVideo2VlmExample — audio + video → ASR + VLM out

Source: AudioVideo2VlmExample.java

Scenario: full multimodal understanding, text reply.

java
agentSdk.registerAudioVideo2Vlm(new AudioVideo2VlmCallback(agentSdk) {

    /**
     * @param flag  0 VAD start, 1 audio chunk, 2 VAD end, 3 H264 video frame
     * @param buf   audio bytes (flag=1) or H264 bytes (flag=3)
     */
    @Override
    public void onRequest(String agentId, String eventId, int flag, byte[] buf,
                          AgentParam param, AudioVideo2VlmResponse response) {

        // 1) ASR
        if (flag == 0) {
            // init ASR
        } else if (flag == 1) {
            // feed audio
        } else if (flag == 2) {
            // VAD end
        }
        response.onAsrMiddle(eventId, "today");
        response.onAsrFinal(eventId, "what's the weather like today");

        String itemId = "itemId_000001";

        // 2) Barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // 3) VLM streaming (combined with video frames)
        String textDelta = "the LLM streaming text goes here";
        response.onVlmItemDelta(eventId, itemId, textDelta);
        response.onVlmItemDone(eventId, itemId);
        response.onVlmDone(eventId);
    }

    // ... shared base callbacks ...
});

AudioVideo2TtsExample — audio + video → ASR + VLM + TTS out

Source: AudioVideo2TtsExample.java

Scenario: full multimodal end-to-end — closest to real robot conversations.

java
agentSdk.registerAudioVideo2Tts(new AudioVideo2TtsCallback(agentSdk) {

    /**
     * @param flag  0 VAD start, 1 audio chunk, 2 VAD end, 3 H264 video frame
     * @param buf   audio bytes (flag=1) or H264 bytes (flag=3)
     */
    @Override
    public void onRequest(String agentId, String eventId, int flag, byte[] buf,
                          AgentParam param, AudioVideo2TtsResponse response) {

        // 1) ASR
        if (flag == 0) {
            // init ASR
        } else if (flag == 1) {
            // feed audio
        } else if (flag == 2) {
            // VAD end
        }
        response.onAsrMiddle(eventId, "today");
        response.onAsrFinal(eventId, "what's the weather like today");

        String itemId = "itemId_000001";

        // 2) Barge-in + skill
        response.onInterrupt(eventId, "chat", null);
        AgentParam skillParam = AgentParam.create().setInteger("step", 3);
        response.onSkill(eventId, itemId, "movement", "move_forward", skillParam);

        // 3) VLM streaming
        String textDelta = "the LLM streaming text goes here";
        response.onVlmItemDelta(eventId, itemId, textDelta);
        response.onVlmItemDone(eventId, itemId);
        response.onVlmDone(eventId);

        // 4) TTS streaming
        byte[] audio = new byte[1024];
        response.onTtsItemDelta(eventId, itemId, audio);
        response.onTtsItemDone(eventId, itemId);
        response.onTtsDone(eventId);
    }

    // ... shared base callbacks ...
});

Cross-cutting concerns

Skill dispatch

When the LLM/VLM decides a skill (navigation, grasping, ...) should fire:

java
AgentParam skillParam = AgentParam.create()
    .setInteger("step", 3)
    .setDouble("speed", 0.5)
    .setString("target", "kitchen");

response.onSkill(eventId, itemId, "movement", "walk_forward", skillParam);

Error handling

Report failures back to the robot with response.onError(...):

java
try {
    // business logic ...
} catch (AsrServiceException e) {
    response.onError(eventId, 5001, "ASR service unavailable: " + e.getMessage());
} catch (LlmServiceException e) {
    response.onError(eventId, 5002, "LLM service timeout");
} catch (Exception e) {
    response.onError(eventId, 5999, "unknown error");
}

Function Call dispatch & Arbiter (new in v1.5.0)

v1.5.0 opens Function Call dispatch and Arbiter arbitration on the passive path: the gateway forwards the semantic classification result (v3 classifier + skill router) as a function call (onFunctionCall), and the application decides — based on FunctionCallInfo.policy — whether to delegate to the agent's default cloud pipeline or override and take over itself. All 8 passive callback classes can override onFunctionCall.

java
@Override
public void onFunctionCall(String agentId, String eventId, FunctionCallInfo functionCall, Audio2TtsResponse response) {
    log.info("onFunctionCall => agentId {}, eventId {}, functionCall {}",
        agentId, eventId, functionCall == null ? "{}" : JSON.toJSONString(functionCall));
    if (functionCall == null) {
        return;
    }
    // After the classification result arrives, the developer arbitrates: delegate or override
    if ("body".equals(functionCall.getPolicy())) {
        // Body skill: delegate to the agent's default cloud pipeline
        response.onArbiter(eventId, ArbiterDecision.DELEGATE.getDecision(), null);
    } else if ("cloud".equals(functionCall.getPolicy())) {
        // Cloud-side skill: the developer takes over (override)
        response.onArbiter(eventId, ArbiterDecision.OVERRIDE.getDecision(), null);
    }
}

FunctionCallInfo (com.agibot.aiem.sdk.FunctionCallInfo) fields: source / policy (e.g. body / cloud) / type / value / param (Map<String,Object>). Arbitration decision ArbiterDecision: DELEGATE / OVERRIDE. Full interface in Passive Callbacks Guide · Function Call exposure & Arbiter.

Conversation history (onHistories, new in v1.5.0)

The gateway forwards the current session's history list via onHistories; use it as context for the XLM:

java
@Override
public void onHistories(String agentId, String eventId, List<HistoryInfo> histories) {
    if (histories == null) {
        return;
    }
    // De-duplicate by eventId and sort by timestamp before use
    histories.stream()
        .sorted(Comparator.comparingLong(HistoryInfo::getTimestamp))
        .forEach(h -> log.info("history => ts {}, query {}, answer {}, complete {}",
            h.getTimestamp(), h.getQuery(), h.getAnswer(), h.isComplete()));
}

HistoryInfo (com.agibot.aiem.sdk.HistoryInfo) fields:

FieldTypeDescription
timestamplongtimestamp of this entry (for sorting)
queryStringuser-side input
answerStringagent-side reply
completebooleanwhether this turn finished completely
eventIdStringcorresponding session event ID (for de-duplication)

Which example to pick?

Your input → outputRecommended example
audio → textAudio2LlmExample
audio → speechAudio2TtsExample
text → textAsr2LlmExample
text → speechAsr2TtsExample
text + video → textAsrVideo2VlmExample
text + video → speechAsrVideo2TtsExample
audio + video → textAudioVideo2VlmExample
audio + video → speechAudioVideo2TtsExample