Page 808 - AI for Good Innovate for Impact

AI for Good Innovate for Impact

focus is on proving the feasibility and benefits of distributed learning across agents without
centralizing raw sensor data. Key performance benchmarks will be measured. For example,
using FL, only model parameters or gradients are shared rather than raw images or point
clouds, dramatically reducing communication loads. The initial goal is to achieve near real-
time aggregation intervals and ensure the global model converges to within an acceptable
range of a centrally trained model’s accuracy while significantly minimizing the data sent over
the network. In parallel, foundational privacy-preserving techniques will be applied. A modest
differential privacy guarantee may be introduced to gauge its impact on model performance.
By the end of this phase, the project should demonstrate a working FL prototype where each
vehicle’s sensor data remains local, basic privacy measures are in place, and the system can
learn a simple shared model with measurable efficiency and privacy benefits.

In the 1–2 year timeframe, the project will build on the prototypes to scale up the capability,
robustness, and sophistication of the intelligent mobility ecosystem:

Enhanced Sensor Fusion & Environmental Context: Integrate additional data modalities
and external data feeds into both the vehicle and its digital twin. Mid-term goals include
incorporating V2X inputs from smart infrastructure directly into the vehicle’s decision-making
loop. The digital twin orchestration platform will be extended to model complex urban
environments, including dynamic traffic conditions and weather events, allowing validation
of the system under a wide range of scenarios. By combining on-board sensor fusion with
V2X communication, the autonomous agent will gain the ability to see beyond line-of-sight,
improving safety and efficiency. The digital twin will serve as a testbed for these features,
ensuring that the fusion of vehicle sensors with external inputs is calibrated and reliable before
deployment.

Robust LLM–Agent Collaboration: With a basic LLM-agent interface in place, the mid-term work
will refine and formalize this integration. A unified protocol will be developed in collaboration
with industry standards. The intent interpretation module will be made more sophisticated,
possibly using semantic parsing or intermediate planning languages to accurately capture
the LLM’s intent. For instance, if the LLM suggests a complex maneuver, the system will break
this down into actionable sub-tasks that the agent can execute. Improved action delegation
strategies will be implemented: the autonomous agent will intelligently allocate tasks between
its traditional control stack and the high-level LLM-driven planner. Safety and explainability will
be prioritized. Concurrently, feedback loops between the agent and LLM will be strengthened.
The vehicle agent will continuously feed back state updates to the LLM, enabling the LLM to
adjust its plans in real time. This interactive loop aligns with techniques like grounded decoding
in robotics, wherein LLM plans are dynamically conditioned on the physical environment’s state.
We anticipate that by the end of two years, the LLM-assisted planning system will be capable
of handling complex, multi-step driving tasks in simulation with high success rates, thanks to
a well-defined interface and continuous agent–LLM communication.

Federated Learning at Scale with Privacy Guarantees: The mid-term milestone for federated
learning is to scale up the number of participating vehicles. Effort will be devoted to optimizing
the convergence rate of the federated model. Success will be measured by the global model
reaching a high accuracy within a limited number of communication rounds, and maintaining
stability even as more vehicles join the training. Concretely, the aim is to approach centralized
training accuracy to within a few percent, with minimal increase in training time. Techniques
like personalized federated learning will be explored to accommodate differences between
vehicles. In terms of communication efficiency, the system will incorporate model update

772