While Machine Learning (ML) models are becoming an essential component of many mobile applications, they also pose relevant challenges, as they require the execution of an operational pipeline including training and inference. Due to the substantial computational and energy burden imposed by such operations, distributed ML has emerged as a paradigm that can enable ML at the network edge, by indeed distributing the computational load across different network nodes and, also, a better privacy preservation whenever data are processed locally. 
In spite of such advantages, also distributed ML may face significant hurdles, namely, (i) slower execution of the ML operational pipeline, due to the limited computing and energy capability of the nodes contributing to that, and (ii) the high bandwidth consumption on the links connecting the participating network nodes, whenever the output of intermediate processing has to be transferred from one node to another. A paradox thus arises due to such hurdles and the fact that the distributed ML paradigm has been introduced to support ML-based services and applications at the edge with low latency towards the end devices and reduced bandwidth consumption to the cloud. In this talk, we discuss such a paradox and how it can be solved by leveraging dynamic structures of ML models which already exists as well as techniques for the compression of ML models and input datasets.