public MHAttention — parametric type
MHAttention(nheads::Integer, qkv_layer, attn_drop, projection)
Multi-head self-attention layer.
Arguments:
nheads: Number of headsqkv_layer: layer to be used for getting the query, key and valueattn_drop: dropout rate after the self-attention layerprojection: projection layer to be used after self-attention
MHAttention(planes::Integer, nheads::Integer = 8; qkv_bias::Bool = false, attn_drop = 0., proj_drop = 0.)
Multi-head self-attention layer.
Arguments:
planes: number of input channelsnheads: number of headsqkv_bias: whether to use bias in the layer to get the query, key and valueattn_drop: dropout rate after the self-attention layerproj_drop: dropout rate after the projection layer