Page 75 - ITU Journal, ICT Discoveries, Volume 3, No. 1, June 2020 Special issue: The future of video and immersive media

ITU Journal: ICT Discoveries, Vol. 3(1), June 2020

TRENDS AND ADVANCEMENTS IN
DEEP NEURAL NETWORK COMMUNICATION

1
Felix Sattler , Thomas Wiegand 1,2 , Wojciech Samek 1
1 Department of Video Coding & Analytics, Fraunhofer Heinrich Hertz Institute, 10587 Berlin, Germany,
2 Department of Electrical Engineering & Computer Science, Technische Universität Berlin, 10587 Berlin, Germany

Abstract – Due to their great performance and scalability properties neural networks have become ubiquitous building
blocks of many applications. With the rise of mobile and IoT, these models now are also being increasingly applied
in distributed settings, where the owners of the data are separated by limited communication channels and privacy
constraints. To address the challenges of these distributed environments, a wide range of training and evaluation
schemes have been developed, which require the communication of neural network parametrizations. These novel
approaches, which bring the “intelligence to the data” have many advantages over traditional cloud solutions such as
privacy preservation, increased security and device autonomy, communication efficiency and a greater training speed.
This paper gives an overview on the recent advancements and challenges in this new field of research at the intersection
of machine learning and communications.
Keywords – Distributed training, federated learning, model compression, neural networks, on-device inference.

1. INTRODUCTION all the workload for processing the data to the com-
putationally potent server, it also has multiple severe
Neural networks have achieved impressive successes in
a wide variety of areas of computational intelligence drawbacks and limitations, which all arise from the
fact that user data is processed at a centralized location:
such as computer vision [30][83][41], natural language
processing [6][42][65] and speech recognition [26] among Privacy: Data collected by mobile or IoT devices is
many others and, as a result, have become a core build- often of private nature and thus bound to the local de-
ing block of many applications. vice. Medical data, text messages, private pictures or
As mobile and Internet of things (IoT) devices become footage from surveillance cameras are examples of data
ubiquitous parts of our daily lives, neural networks are which cannot be processed in the cloud. New data pro-
also being applied in more and more distributed settings. tection legislations like the European GDPR [73] or the
These distributed devices are getting equipped with ever Cyber Security Law of the People’s Republic of China
more potent sensors and storage capacities and collect [20] enforce strong regulations on data privacy.
vast amounts of personalized data, which is highly valu- Ownership: Attributing and claiming ownership is a
able for processing in machine learning pipelines. difficult task if personal data is transfered to a central
When it comes to the processing of data from distributed location. Cloud ML leaves users in the dark about what
sources, the ”Cloud ML” paradigm [33] has reigned happens with their data or requires cumbersome rights
supreme in the previous decade. In Cloud ML, local management from the cloud service provider.
user data is communicated from the often hardware con- Security: With all data being stored at one central lo-
strained mobile or IoT devices to a computationally po- cation, Cloud ML exposes a single point of failure. Mul-
tent centralized server where it is then processed in a tiple cases of data leakage in recent times have demon-
1
machine learning pipeline (e.g. a prediction is made strated that the centralized processing of data comes
using an existing model or the data is used to train a with an unpredictable security risk for the users.
new model). The result of the processing operation may Efficiency: Transferring large records of data to a cen-
then be sent back to the local device. From a communi- tral compute node often is more expensive in terms of
cation perspective, methods which follow the Cloud ML time and energy than the actual processing of the data.
paradigm make use of centralized intelligence and
For instance, single records of medical image data can
”Bring the data to the model.” already be hundreds of Megabytes in size [71]. If the
local data is large and/or the communication channels
While the Cloud ML paradigm is convenient for the are limited, moving data to the cloud might thus become
clients from a computational perspective, as it moves inefficient or unfeasible.
Autonomy: Many distributed devices need to act fully
This work was partly supported by the German Ministry for Edu-
cation and Research as BIFOLD - Berlin Institute for the Founda- 1 A comprehensive list of documented breaches can be found at
tions of Learning and Data (ref. 01IS18025A and ref 01IS18037A). https://en.wikipedia.org/wiki/List_of_data_breaches .

© International Telecommunication Union, 2020 53