Robots in human-centred environments face open-ended instructions, shifting task priorities, and unpredictable constraints. A language-guided approach promises intuitive interaction, priority-aware planning, and resilient adaptation, enabling robots to deliver useful support in complex settings. OFFIS is currently developing an AI-driven solution that enables a simulated mobile robot to interpret spoken commands, autonomously prioritise tasks, plan routes efficiently, and accurately identify and assign packages to rooms using visual information through a multimodal LLM, enabling the system to adapt dynamically to changing constraints defined by the user.

Leverage the OptimAIse tools and framework to implement an AI-driven capability that turns requests into grounded, priority-sensitive action plans, executes them safely, and explains its decisions. To this end, a set of components/exploitable results will be used and validated, particularly: intelligent orchestration, evaluation framework for orchestration, model compression and explainability.

The use of the OptimAIse framework is expected to directly enhance model development by reducing training and tuning times and lowering infrastructure costs through advanced compression and intelligent workload orchestration, while enforcing safety via validated, policy-aligned actions and strengthening trust through transparency and accountability mechanisms. Its scalable simulation and deployment infrastructure ensures that capabilities refined in controlled environments transfer seamlessly to real-world contexts. By uniting data sovereignty, computational efficiency, orchestration intelligence, and explainable decision-making, the framework not only improves performance, security, and usability but also reduces environmental impact, establishing a reusable foundation for intelligent, priority-aware robot support in dynamic, human-centred scenarios.

We couple an LLM with structured world knowledge and simulation, treating user input, scene context, and task constraints as coequal signals for decision-making. The orchestration tools of OptimAIse will help optimise how workloads are distributed across systems, ensuring robust, energy-efficient operation as priorities shift and computational needs vary.

The model synthesises a global strategy – e.g., serving the highest-priority first and arranging remaining tasks by proximity or effort – and continuously reconciles this plan with current conditions. World knowledge is provided through externally maintained scene and map representations, allowing the system to ground intentions in a shared, up-to-date depiction of the environment. A guarded validation layer enforces safety and plausibility before execution, while a unified control loop carries out actions, monitors progress, and triggers replanning when circumstances or user goals change.

Since parts of the AI workloads are planned to run locally on a mobile robotic system, the application of OptimAIse compression tools should drastically reduce the size of these models, enabling efficient deployment on edge devices within robots while maintaining high performance. The system preserves an interpretable trace throughout, ensuring that outcomes and errors are transparent and explainable to operators. Simulation with a flexible robotics stack supports embodiment-agnostic testing across different robot forms and sensor configurations, accelerating iteration without committing to a single hardware platform.