DNN partitioning for inference throughput acceleration at the edge - École polytechnique Access content directly
Journal Articles IEEE Access Year : 2023

DNN partitioning for inference throughput acceleration at the edge

Thomas Feltin
Leo Marche
  • Function : Author
Juan-Antonio Cordero-Fuertes
Frank Brockners
  • Function : Author
Thomas Heide Clausen

Abstract

Deep neural network (DNN) inference on streaming data requires computing resources to satisfy inference throughput requirements. However, latency and privacy sensitive deep learning applications cannot afford to offload computation to remote clouds because of the implied transmission cost and lack of trust in third-party cloud providers. Among solutions to increase performance while keeping computation on a constrained environment, hardware acceleration can be onerous, and model optimization requires extensive design efforts while hindering accuracy. DNN partitioning is a third complementary approach, and consists of distributing the inference workload over several available edge devices, taking into account the edge network properties and the DNN structure, with the objective of maximizing the inference throughput (number of inferences per second). This paper introduces a method to predict inference and transmission latencies for multi-threaded distributed DNN deployments, and defines an optimization process to maximize the inference throughput. A branch and bound solver is then presented and analyzed to quantify the achieved performance and complexity. This analysis has led to the definition of the acceleration region, which describes deterministic conditions on the DNN and network properties under which DNN partitioning is beneficial. Finally, experimental results confirm the simulations and show inference throughput improvements in sample edge deployments.
Fichier principal
Vignette du fichier
IEEE Access.pdf (32.41 Mo) Télécharger le fichier
Origin : Files produced by the author(s)
Licence : CC BY - Attribution

Dates and versions

hal-04008199 , version 1 (28-02-2023)

Licence

Attribution

Identifiers

Cite

Thomas Feltin, Leo Marche, Juan-Antonio Cordero-Fuertes, Frank Brockners, Thomas Heide Clausen. DNN partitioning for inference throughput acceleration at the edge. IEEE Access, In press, pp.1-1. ⟨10.1109/ACCESS.2023.3244497⟩. ⟨hal-04008199⟩

Collections

X IP_PARIS
5 View
17 Download

Altmetric

Share

Gmail Facebook X LinkedIn More