Page 21 - U4SSC Guiding principles for artificial intelligence in cities

• Incorporation of security during AI systems design rather than as an afterthought, i.e. “Secure
by Design”;
• Auditing AI systems for safety and security on an ongoing basis;

• Implementing resilient AI systems (e.g., fall-back solutions, business continuity, disaster recovery
mechanisms);
• Conducting risk assessments to ensure safety and security of AI systems;

• Designing response to either AI system component or even entire system failure and to
determine corresponding performance levels; and

• Protect and respect personal data in line with privacy rules.

3.2.7 High performing and robust

AI systems in urban contexts are designed to achieve certain targeted performance objectives
(e.g., accuracy, optimality, acceptable resource consumption) and they perform various functions
to automate certain tasks (potentially reducing human involvement and intervention).

Therefore, it is very important to achieve a high level of performance which is acceptable for AI
systems designers and users. These systems include algorithms and data (e.g., training data prior
to actual usage, operational data while the system is in use).

AI systems operate under varying conditions. The conditions contemplated during AI systems
design may differ from the actual conditions encountered during operation. Hence, it is very
important for AI systems to uphold their acceptable performance levels not only during design,
but also during actual operation (which may entail varying conditions). This is commonly referred
to as performance robustness.

Hence, this principle allows cities to develop, deploy, and use high performing and robust AI
systems.

Implementation Considerations: Cities can adopt various mechanisms to achieve high
performance and robustness in AI systems. These mechanisms include:

• Defining performance objectives for AI systems including target and acceptable performance
levels;
• Testing and evaluating AI systems’ performance robustness with respect to parameter changes
(perturbations) in algorithms (e.g., certain algorithms’ performances may be highly sensitive to
algorithm parameters);

• Testing and evaluating AI systems performance robustness with respect to changes
(perturbations) in datasets (e.g., certain algorithms’ performances may be highly sensitive to
changes in datasets);

11