Add parallel linking support for BIDIRECTIONAL_SEQUENCE_{LSTM|RNN}
Change-Id: I51396f82de8dd7282db85a26b0a1be84b236b8c1
Fix: 138653129
Test: NNTest_static and VtsHalNeuralnetworksV1_3TargetTest
diff --git a/current.txt b/current.txt
index 67e2818..4a4d379 100644
--- a/current.txt
+++ b/current.txt
@@ -597,7 +597,7 @@
5f6d3097ba84cb63c430787123f4de1b31c11f90b531b98eae9a8623a5ae962a android.hardware.neuralnetworks@1.1::types
fb382e986c10b8fbb797a8546e8f9ea6d1107bfe6f3fb7e57f6bbbf1f807a906 android.hardware.neuralnetworks@1.2::IDevice
40e71cd693de5b832325c5d8f081f2ff20a7ba2b89d401cee5b4b3eb0e241681 android.hardware.neuralnetworks@1.2::IPreparedModel
-6c29d6fdd5445911df5456b3b84b949cdd59fca0c0b5507662f26a5cac0cf5e5 android.hardware.neuralnetworks@1.2::types
+00649d29680f2c47edf60000c3ae7ae906ba638f0616947147e3676a83cf36fa android.hardware.neuralnetworks@1.2::types
a785a57447a81e9c130eef6904c3a5c256076c6a04588c40620ebd6fa2660d77 android.hardware.radio@1.2::types
1a6e2bd289f22931c526b21916910f1d4c436b7acb9556e4243de4ce8e6cc2e4 android.hardware.soundtrigger@2.0::ISoundTriggerHwCallback
fd65298e1e09e0e3c781ab18305920d757dbe55a3b459ce17814ec5cf6dfee99 android.hardware.wifi@1.0::IWifiP2pIface
@@ -676,7 +676,7 @@
2fa3679ad7c94b5e88724adcd560c561041068a4ca565c63830e68101988746a android.hardware.neuralnetworks@1.3::IFencedExecutionCallback
43088ffc71945b463a7279262cfe2e290f6ed2f15d3fd6032798a3be299fb08f android.hardware.neuralnetworks@1.3::IPreparedModel
0439a1fbbec7f16e5e4c653d85ac685d51bfafbae15b8f8cca530acdd7d6a8ce android.hardware.neuralnetworks@1.3::IPreparedModelCallback
-306fda32ac969fd51d75d066352cadcb769944ec4823be4cdd3f86fdb9e97511 android.hardware.neuralnetworks@1.3::types
+dd39887aa4fb60ce60ea9cc043edeadbbae6e922d09d3946311b0b410024ae14 android.hardware.neuralnetworks@1.3::types
3e01d4446cd69fd1c48f8572efd97487bc179564b32bd795800b97bbe10be37b android.hardware.wifi@1.4::IWifi
c67aaf26a7a40d14ea61e70e20afacbd0bb906df1704d585ac8599fbb69dd44b android.hardware.wifi.hostapd@1.2::IHostapd
2b5a7ea572b736030c64a3b4043af244425477c4672301780fe15aba5ed393d9 android.hardware.wifi.hostapd@1.2::types
diff --git a/neuralnetworks/1.2/types.hal b/neuralnetworks/1.2/types.hal
index 993a105..f0fd769 100644
--- a/neuralnetworks/1.2/types.hal
+++ b/neuralnetworks/1.2/types.hal
@@ -2314,7 +2314,38 @@
AXIS_ALIGNED_BBOX_TRANSFORM = 41,
/**
- * Performs a forward LSTM on the input followed by a backward LSTM.
+ * A recurrent neural network layer that applies an LSTM cell to a
+ * sequence of inputs in forward and backward directions.
+ *
+ * The op supports cross-linking via an auxiliary input. Regular cell feeds
+ * one input into the two RNN cells in the following way:
+ *
+ * INPUT (INPUT_REVERSED)
+ * | |
+ * ---------------------
+ * | FW_LSTM BW_LSTM |
+ * ---------------------
+ * | |
+ * FW_OUT BW_OUT
+ *
+ * An op with cross-linking takes two inputs and feeds them into the RNN
+ * cells in the following way:
+ *
+ * AUX_INPUT (AUX_INPUT_REVERSED)
+ * | |
+ * INPUT | (INPUT_R'D.)|
+ * | | | |
+ * -----------------------
+ * | \ / \ / |
+ * | FW_LSTM BW_LSTM |
+ * -----------------------
+ * | |
+ * FW_OUT BW_OUT
+ *
+ * The cross-linking mode is enabled iff auxiliary input and auxiliary
+ * weights are present. While stacking this op on top of itself, this
+ * allows to connect both forward and backward outputs from previous cell
+ * to the next cell's input.
*
* Supported tensor {@link OperandType}:
* * {@link OperandType::TENSOR_FLOAT16}
@@ -2324,7 +2355,6 @@
*
* All input and output tensors must be of the same type.
*
- *
* Inputs:
* * 0: The input.
* A 3-D tensor of shape:
@@ -2533,8 +2563,8 @@
* * “activation” is the function passed as the “fused_activation_function”
* argument (if not “NONE”).
*
- * The op also supports an auxiliary input. Regular cell feeds one input
- * into the two RNN cells in the following way:
+ * The op supports cross-linking via an auxiliary input. Regular cell feeds
+ * one input into the two RNN cells in the following way:
*
* INPUT (INPUT_REVERSED)
* | |
@@ -2544,8 +2574,8 @@
* | |
* FW_OUT BW_OUT
*
- * An op with an auxiliary input takes two inputs and feeds them into the
- * RNN cells in the following way:
+ * An op with cross-linking takes two inputs and feeds them into the RNN
+ * cells in the following way:
*
* AUX_INPUT (AUX_INPUT_REVERSED)
* | |
@@ -2558,9 +2588,10 @@
* | |
* FW_OUT BW_OUT
*
- * While stacking this op on top of itself, this allows to connect both
- * forward and backward outputs from previous cell to the next cell's
- * inputs.
+ * The cross-linking mode is enabled iff auxiliary input and auxiliary
+ * weights are present. While stacking this op on top of itself, this
+ * allows to connect both forward and backward outputs from previous cell
+ * to the next cell's input.
*
* Supported tensor {@link OperandType}:
* * {@link OperandType::TENSOR_FLOAT16}
diff --git a/neuralnetworks/1.3/types.hal b/neuralnetworks/1.3/types.hal
index a808a2e..08d8e6b 100644
--- a/neuralnetworks/1.3/types.hal
+++ b/neuralnetworks/1.3/types.hal
@@ -2364,7 +2364,54 @@
AXIS_ALIGNED_BBOX_TRANSFORM = @1.2::OperationType:AXIS_ALIGNED_BBOX_TRANSFORM,
/**
- * Performs a forward LSTM on the input followed by a backward LSTM.
+ * A recurrent neural network layer that applies an LSTM cell to a
+ * sequence of inputs in forward and backward directions.
+ *
+ * The op supports cross-linking via an auxiliary input. Regular cell feeds
+ * one input into the two RNN cells in the following way:
+ *
+ * INPUT (INPUT_REVERSED)
+ * | |
+ * ---------------------
+ * | FW_LSTM BW_LSTM |
+ * ---------------------
+ * | |
+ * FW_OUT BW_OUT
+ *
+ * An op with cross-linking takes two inputs and feeds them into the RNN
+ * cells in the following way:
+ *
+ * AUX_INPUT (AUX_INPUT_REVERSED)
+ * | |
+ * INPUT | (INPUT_R'D.)|
+ * | | | |
+ * -----------------------
+ * | \ / \ / |
+ * | FW_LSTM BW_LSTM |
+ * -----------------------
+ * | |
+ * FW_OUT BW_OUT
+ *
+ * The cross-linking mode is enabled iff auxiliary input and auxiliary
+ * weights are present. While stacking this op on top of itself, this
+ * allows to connect both forward and backward outputs from previous cell
+ * to the next cell's input.
+ *
+ * Since HAL version 1.3 parallel linking mode is supported. The mode is
+ * enabled if auxiliary input is present but auxiliary weights are omitted.
+ * In this case, the cell feeds inputs into the RNN in the following way:
+ *
+ * INPUT (AUX_INPUT_REVERSED)
+ * | |
+ * ---------------------
+ * | FW_LSTM BW_LSTM |
+ * ---------------------
+ * | |
+ * FW_OUT BW_OUT
+ *
+ * While stacking this op on top of itself, this allows to connect both
+ * forward and backward outputs from previous cell to the next cell's
+ * corresponding inputs.
*
* Supported tensor {@link OperandType}:
* * {@link OperandType::TENSOR_FLOAT16}
@@ -2374,7 +2421,6 @@
*
* All input and output tensors must be of the same type.
*
- *
* Inputs:
* * 0: The input.
* A 3-D tensor of shape:
@@ -2466,25 +2512,34 @@
* * 38: The backward input cell state.
* A 2-D tensor of shape [batch_size, bw_num_units].
* * 39: The auxiliary input. Optional.
- * A 3-D tensor of shape [max_time, batch_size, input_size], where “batch_size”
- * corresponds to the batching dimension, and “input_size” is the size
- * of the input.
- * * 40: The forward auxiliary input-to-input weights. Optional.
- * A 2-D tensor of shape [fw_num_units, input_size].
- * * 41: The forward auxiliary input-to-forget weights. Optional.
- * A 2-D tensor of shape [fw_num_units, input_size].
- * * 42: The forward auxiliary input-to-cell weights. Optional.
- * A 2-D tensor of shape [fw_num_units, input_size].
- * * 43: The forward auxiliary input-to-output weights. Optional.
- * A 2-D tensor of shape [fw_num_units, input_size].
- * * 44: The backward auxiliary input-to-input weights. Optional.
- * A 2-D tensor of shape [bw_num_units, input_size].
- * * 45: The backward auxiliary input-to-forget weights. Optional.
- * A 2-D tensor of shape [bw_num_units, input_size].
- * * 46: The backward auxiliary input-to-cell weights. Optional.
- * A 2-D tensor of shape [bw_num_units, input_size].
- * * 47: The backward auxiliary input-to-output weights. Optional.
- * A 2-D tensor of shape [bw_num_units, input_size].
+ * A 3-D tensor of shape [max_time, batch_size, aux_input_size],
+ * where “batch_size” corresponds to the batching dimension, and
+ * “aux_input_size” is the size of the auxiliary input. Optional. See
+ * the docs above for the usage modes explanation.
+ * * 40: The forward auxiliary input-to-input weights.
+ * Optional. See the docs above for the usage modes explanation.
+ * A 2-D tensor of shape [fw_num_units, aux_input_size].
+ * * 41: The forward auxiliary input-to-forget weights.
+ * Optional. See the docs above for the usage modes explanation.
+ * A 2-D tensor of shape [fw_num_units, aux_input_size].
+ * * 42: The forward auxiliary input-to-cell weights.
+ * Optional. See the docs above for the usage modes explanation.
+ * A 2-D tensor of shape [fw_num_units, aux_input_size].
+ * * 43: The forward auxiliary input-to-output weights.
+ * Optional. See the docs above for the usage modes explanation.
+ * A 2-D tensor of shape [fw_num_units, aux_input_size].
+ * * 44: The backward auxiliary input-to-input weights.
+ * Optional. See the docs above for the usage modes explanation.
+ * A 2-D tensor of shape [bw_num_units, aux_input_size].
+ * * 45: The backward auxiliary input-to-forget weights.
+ * Optional. See the docs above for the usage modes explanation.
+ * A 2-D tensor of shape [bw_num_units, aux_input_size].
+ * * 46: The backward auxiliary input-to-cell weights.
+ * Optional. See the docs above for the usage modes explanation.
+ * A 2-D tensor of shape [bw_num_units, aux_input_size].
+ * * 47: The backward auxiliary input-to-output weights.
+ * Optional. See the docs above for the usage modes explanation.
+ * A 2-D tensor of shape [bw_num_units, aux_input_size].
* * 48: The activation function.
* A value indicating the activation function:
* <ul>
@@ -2607,8 +2662,8 @@
* * “activation” is the function passed as the “fused_activation_function”
* argument (if not “NONE”).
*
- * The op also supports an auxiliary input. Regular cell feeds one input
- * into the two RNN cells in the following way:
+ * The op supports cross-linking via an auxiliary input. Regular cell feeds
+ * one input into the two RNN cells in the following way:
*
* INPUT (INPUT_REVERSED)
* | |
@@ -2618,8 +2673,8 @@
* | |
* FW_OUT BW_OUT
*
- * An op with an auxiliary input takes two inputs and feeds them into the
- * RNN cells in the following way:
+ * An op with cross-linking takes two inputs and feeds them into the RNN
+ * cells in the following way:
*
* AUX_INPUT (AUX_INPUT_REVERSED)
* | |
@@ -2632,9 +2687,26 @@
* | |
* FW_OUT BW_OUT
*
+ * The cross-linking mode is enabled iff auxiliary input and auxiliary
+ * weights are present. While stacking this op on top of itself, this
+ * allows to connect both forward and backward outputs from previous cell
+ * to the next cell's input.
+ *
+ * Since HAL version 1.3 parallel linking mode is supported. The mode is
+ * enabled if auxiliary input is present but auxiliary weights are omitted.
+ * In this case, the cell feeds inputs into the RNN in the following way:
+ *
+ * INPUT (AUX_INPUT_REVERSED)
+ * | |
+ * ---------------------
+ * | FW_RNN BW_RNN |
+ * ---------------------
+ * | |
+ * FW_OUT BW_OUT
+ *
* While stacking this op on top of itself, this allows to connect both
* forward and backward outputs from previous cell to the next cell's
- * inputs.
+ * corresponding inputs.
*
* Supported tensor {@link OperandType}:
* * {@link OperandType::TENSOR_FLOAT16}
@@ -2667,11 +2739,17 @@
* A 2-D tensor of shape [batchSize, bwNumUnits]. Specifies a hidden
* state input for the first time step of the computation.
* * 9: auxInput.
- * A 3-D tensor. The shape is the same as of the input 0.
+ * A 3-D tensor. The shape is defined by the input 6 (timeMajor). If
+ * it is set to true, then the input has a shape [maxTime, batchSize,
+ * auxInputSize], otherwise the input has a shape [batchSize, maxTime,
+ * auxInputSize]. Can be omitted. See the docs above for the usage
+ * modes explanation.
* * 10:fwAuxWeights.
- * A 2-D tensor of shape [fwNumUnits, inputSize].
+ * A 2-D tensor of shape [fwNumUnits, auxInputSize]. Can be omitted.
+ * See the docs above for the usage modes explanation.
* * 11:bwAuxWeights.
- * A 2-D tensor of shape [bwNumUnits, inputSize].
+ * A 2-D tensor of shape [bwNumUnits, auxInputSize]. Can be omitted.
+ * See the docs above for the usage modes explanation.
* * 12:fusedActivationFunction.
* A {@link FusedActivationFunc} value indicating the activation function. If
* “NONE” is specified then it results in a linear activation.