STADT:up – Intuitive and Understandable HMI for Autonomous Vehicles

That is the central question for us in the STADT:up project. In Subproject 2 – “Human Factors”, Ergosign focuses on developing information and interaction concepts that explain the behavior of autonomous vehicles. Only when we understand what’s happening around us and how the vehicle is making decisions can we feel truly safe. Human-centered user experience design is crucial here: solutions must be aligned with people’s needs to unlock their full potential.

To build trust in AI-based systems like autonomous vehicles, we need to demonstrate how they make decisions. We are experimenting with various ways and modalities to communicate AI decisions — via displays, audio signals, or ambient lighting. It’s important that the information is helpful without being overwhelming. We’re aiming for the perfect information balance and exploring situation- and context-dependent interaction modules.

We rigorously test our ideas and concepts through user studies and simulation environments. Some of our key findings so far include:

Information needs: The occupants’ need for information strongly depends on how safe or dangerous the driving situation feels.

• Critical moments: In critical situations, many occupants find it important to know exactly what the vehicle sees and plans. A clear and understandable visualization of the surroundings is very valuable here.

• Multimodal signals: Multimodal signals can optimize the user experience but require careful design to avoid overwhelming or irritating the users. Light and sound cues can enhance UX and build trust in autonomous vehicles if used purposefully.

To improve information delivery, we’re currently exploring the potential of Augmented Reality (AR). With AR, we can project cues and information directly into the user’s field of vision, such as onto the windshield. In summer 2025, we plan to conduct further user studies in Munich and Hamburg to explore this in more detail. Want to take part and help shape the future of mobility? Sign up for our study here.

In addition to the UX for passengers, we are also developing an EngineeringUI for developers of AI software in Subproject 5 – “Autonomous Driving.” As mentioned in our first article on STADT:up, we initially analyzed the usage context with our partners, derived requirements, and translated them into personas. Based on this foundation, we identified opportunity areas where the EngineeringUI can support developers in their work.

The EngineeringUI visualizes vehicle sensor data in a web application and is optimized for mobile use in and around test vehicles. It helps developers record, annotate, and manage data and evaluate and optimize algorithms.

For development, we use the Robot Operating System (ROS) as a foundation — a flexible, open-source framework that handles core robot communication and control. To visualize and analyze the ROS-generated data streams in real-time, we use the open source application Foxglove, also adopted at the start of the project. We have specifically optimized Foxglove for mobile use on tablets and other portable devices and expanded its visualization capabilities with integrated recording and annotation functions that let us mark and log critical events during operation.

In the long term, the UI aims to enable high-performance sensor data visualization and real-time adjustment of visualization or HMI settings. A major challenge is the sheer volume of data transmitted and processed by the test vehicles. To tackle this, we’re working closely with our project partners.

The STADT:up project is heading into its final phase and will conclude in spring 2026. Thanks to collaboration with OEM, suppliers, and research institutions, we’re developing practical, scientifically grounded solutions for understandable autonomous systems. We’re also supporting the next generation of technical and scientific talent and creating valuable synergies between human factors research and technology development. The empirical findings will flow directly into the development of human-centered, understandable, and trustworthy autonomous vehicles.

Through this and many other projects, Ergosign is applying its expertise to shape the mobility of tomorrow. We look forward to sharing more insights and results from STADT:up’s final project year soon!

Together, we design user-friendly and trustworthy automated driving functions and smart mobility concepts. We support you with our UX expertise, human-centered HMI development, and proven methods from research and practice.

This work is a result of the joint research project STADT:up (Förderkennzeichen 19A22006K). The project is supported by the German Federal Ministry for Economic Affairs and Climate Action (BMWK), based on a decision of the German Bundestag.

In recent times, artificial intelligence (AI) has made a significant impact across various sectors. We are confident that the incorporation of AI into both our own and our client's offerings will progress further. Leveraging our extensive experience and diverse UX portfolio, we eagerly anticipate shaping our future alongside these technological advancements.