The success of Deep Artificial Neural Networks (DNNs) in many domains created a rich body of research concerned with hardware accelerators for compute-intensive DNN operators. However, implementing such operators efficiently with complex instructions such as matrix multiply is a task not yet automated gracefully. Solving this task often requires complex program and memory layout transformations . First solutions to this problem have been proposed, such as TVM  or ISAMIR, which work on a loop-level representation of operators and rewrite the program before an instruction embedding into the operator is performed . This top-down approach creates a tension between exploration range and search space complexity  .In this work, we propose a new approach to this problem. We have created a bottom-up method that allows the direct generation of implementations based on an accelerator's instruction set . By formulating the embedding as a constraint satisfaction problem over the scalar dataflow, every possible embedding solution is contained in the search space. By adding additional constraints , a solver can produce the subset of preferable solutions. A detailed evaluation using the VTA hardware accelerator with the Baidu DeepBench inference benchmark suite shows that our approach can automatically generate code competitive to reference implementations , and furthermore that memory layout flexibilty can be beneficial for overall performance. While the reference implementation achieves very low hardware utilization due to its fixed embedding strategy, we achieve a geomean speedup of up to x2 .49, while individual operators can improve as much as x238 .
The success of Deep Artificial Neural Networks (DNNs) in many domains created a rich body of research concerned with hardwareaccelerators for compute-intensive DNN operators. However, implementing such operators efficiently with complex hardwareintrinsics such as matrix multiply is a task not yet automated gracefully. Solving this task often requires joint program and data layouttransformations . First solutions to this problem have been proposed, such as TVM , UNIT or ISAMIR, which work on a loop-levelrepresentation of operators and specify data layout and possible program transformations before the embedding into the operator isperformed . This top-down approach creates a tension between exploration range and search space complexity , especially when alsoexploring data layout transformations such as im2col, channel packing or padding .In this work, we propose a new approach to this problem. We created a bottom-up method that allows the joint transformation ofboth compuation and data layout based on the found embedding . By formulating the embedding as a constraint satisfaction problemover the scalar dataflow, every possible embedding solution is contained in the search space. Adding additional constraints andoptmization targets to the solver generates the subset of preferable solutions. An evaluation using the VTA hardware accelerator with the Baidu DeepBench inference benchmark shows that our approach canautomatically generate code competitive to reference implementations . Further, we show that dynamically determining the data layoutbased on intrinsic and workload is beneficial for hardware utilization and performance. In cases where the reference implementationhas low hardware utilization due to its fixed deployment strategy, we achieve a geomean speedup of up to x2 .813, while individualoperators can improve as much as x170 .
R
hardware accelerators
hardwareaccelerators
R
instructions
hardwareintrinsics
R
complex program and memory layout transformations
joint program and data layouttransformations
A
, UNIT
R
loop-level representation
loop-levelrepresentation
R
rewrite the program before an instruction
specify data layout and possible program transformations before the
R
is performed
isperformed
A
, especially when alsoexploring data layout transformations such as im2col, channel packing or padding
D
have
R
direct generation of implementations based on an accelerator's instruction set
joint transformation ofboth compuation and data layout based on the found embedding
R
problem over
problemover
R
By adding additional constraints , a solver can produce
Adding additional constraints andoptmization targets to the solver generates
R
A detailed
An
D
suite
R
can automatically
canautomatically
R
, and furthermore that memory layout flexibilty can be beneficial for overall performance. While the reference implementation achieves very
. Further, we show that dynamically determining the data layoutbased on intrinsic and workload is beneficial for hardware utilization and performance. In cases where the reference implementationhas
R
embedding
deployment
R
.49, while individual operators
.813, while individualoperators
R
x238
x170