Page 265 - Kaleidoscope Academic Conference Proceedings 2024

Innovation and Digital Transformation for a Sustainable World

3. METHODOLOGY Table 1 – Incident reporting in AIAAIC and AIID

The study adopted the following methodology:
AIAAIC AIID
1. Executed an exhaustive search and literature review to What can be reported Incidents and Real-world
discover AI incident repositories. controversies harms or
driven by and near harms
2. Isolated four potential repositories: AIAAIC, AIID, relating to caused by AI
AILD, and AVID. Given that AILD focuses on AI related AI. systems.
legal aspects and AVID emphasizes identifying AI Incidents reported (as 905 657
system vulnerabilities, shortlisted the two open-access on 05-05-2024)
repositories, AIID and AIAAIC, for further scrutiny. Who can report Anyone Anyone
incidents
3. Examined the policies, scope, reporting procedures, and Submissions reviewed Yes Yes
review mechanisms of the AIID and AIAAIC databases before publishing?
to comprehend their operational frameworks. Nature of reporting Voluntary Voluntary
Incentive for reporting None None
4. Submitted an incident to each database to discern their
reporting protocols and procedural intricacies.
Table 2 – Snapshot of Incidents reported in AIAAIC
5. Retrieved and scrutinized publicly available data from
both databases to evaluate their content and structure. AIAAIC ID# Headline Ref.
AIAAIC1449 Adobe trained Firefly AI model on [28]
6. Investigated the repositories to pinpoint gaps in competitor images
standardization across various dimensions, including: AIAAIC1439 OpenAI scrapes YouTube to train [29]
incident reporting protocols, quality control, data GPT-4
interoperability, comprehensiveness of data, contributor AIAAIC1414 Leonardo AI generates celebrity [30]
and source diversity, sector-specific coverage, non-consensual porn images
geographical coverage, and data sharing protocols. AIAAIC1395 Scientific journals publish papers [31]
with AI-generated introductions
7. Tabulated observations and inferred key insights based AIAAIC1368 Microsoft Copilot generates fake [32]
on the conducted analysis. Putin comments on Navalny death
AIAAIC1356 ChatGPT ’goes crazy’, speaks [33]
8. Formulated recommendations for standardization gibberish
activities to address identified gaps and enhance the
effectiveness of AI incident reporting practices.
4.4 Contributors to the Databases
4. RESULTS
Table 4 lists the top seven submitters of the published
incidents in AIID. They reported more than 70% of all the
This section presents the observations and results of the study.
incidents in AIID. AIAAIC does not have data fields to
The next section analyses and draws inferences from them.
capture this data.
4.1 Incident reporting
4.5 Sources of the reports submitted to the databases
Table 1 provides the basics of incident reporting in AIAAIC
Table 5 provides details of the top seven source domains of
and AIID. Both have similar processes for incident reporting,
the reports submitted to AIID. AIAAIC does not have data
though their scopes are slightly different.
fields to capture this data.
4.2 AI-Incident snapshot 4.6 Sector Coverage

Sample incidents reported in AIAAIC, shortlisted for Table 6 details the top seven sectors of the incidents reported
analysis, are listed in Table 2. These were extracted for in AIAAIC. While AIID does not have data fields to capture
analysis by filtering on the criteria “Occurred” = “2024” and this data, Table 7 provides details of the top seven deployers
“Country(ies)” = “Global”. of the AI systems with incidents reported in AIID.

4.3 Interoperability and data sharing 4.7 Geographical coverage

Table 3 compares the data fields available in the two Table 8 lists the top seven countries related to the geographic
databases. They have different data structures. origin and/or primary extent of the incidents reported in

– 221 –