XOP instruction set
The XOP (eXtended Operations) instruction set, announced by AMD on May 1, 2009, is an extension to the 128-bit SSE core instructions in the x86 and AMD64 instruction set for the Bulldozer processor core, which was released on October 12, 2011. However AMD removed support for XOP from Zen (microarchitecture) onward.

The XOP instruction set contains several different types of vector instructions since it was originally intended as a major upgrade to SSE. Most of the instructions are integer instructions, but it also contains floating point permutation and floating point fraction extraction instructions. See the index for a list of instruction types.

History

XOP is a revised subset of what was originally intended as SSE5. It was changed to be similar but not overlapping with AVX, parts that overlapped with AVX were removed or moved to separate standards such as  FMA4 (floating-point vector multiply–accumulate)  and  CVT16 (Half-precision floating-point conversion implemented as F16C by Intel).

All SSE5 instructions that were equivalent or similar to instructions in the AVX and FMA4 instruction sets announced by Intel have been changed to use the coding proposed by Intel. Integer instructions without equivalents in AVX were classified as the XOP extension. The XOP instructions have an opcode byte 8F (hexadecimal), but otherwise almost identical coding scheme as AVX with the 3-byte VEX prefix.

Commentators have seen this as evidence that Intel has not allowed AMD to use any part of the large VEX coding space. AMD has been forced to use different codes in order to avoid using any code combination that Intel might possibly be using in its development pipeline for something else. The XOP coding scheme is as close to the VEX scheme as technically possible without risking that the AMD codes overlap with future Intel codes. This inference is speculative, since no public information is available about negotiations between the two companies on this issue.

The use of the 8F byte requires that the m-bits (see VEX coding scheme) have a value larger than or equal to 8 in order to avoid overlap with existing instructions. The C4 byte used in the VEX scheme has no such restriction. This may prevent the use of the m-bits for other purposes in the future in the XOP scheme, but not in the VEX scheme. Another possible problem is that the pp bits have the value 00 in the XOP scheme, while they have the value 01 in the VEX scheme for instructions that have no legacy equivalent. This may complicate the use of the pp bits for other purposes in the future.

A similar compatibility issue is the difference between the FMA3 and FMA4 instruction sets. Intel initially proposed FMA4 in AVX/FMA specification version 3 to supersede the 3-operand FMA proposed by AMD in SSE5. After AMD adopted FMA4, Intel canceled FMA4 support and reverted to FMA3 in the AVX/FMA specification version 5 (See  FMA history).

In March 2015, AMD explicitly revealed in the description of the patch for the GNU Binutils package that Zen, its third-generation x86-64 architecture in its first iteration (znver1 – Zen, version 1), will not support TBM, FMA4, XOP and LWP instructions developed specifically for the "Bulldozer" family of micro-architectures.

Integer vector multiply–accumulate instructions
These are integer version of the FMA instruction set. These are all four operand instructions similar to FMA4 and they all operate on signed integers.

Integer vector horizontal addition
Horizontal addition instructions adds adjacent values in the input vector to each other. The output size in the instructions below describes how wide the horizontal addition performed is. For instance horizontal byte to word adds two bytes at a time and returns the result as vector of words, but byte to quadword adds eight bytes together at a time and returns the result as vector of quadwords. Six additional horizontal addition and subtraction instructions can be found in SSSE3, but they operate on two input vectors and only does two and two operations.

Integer vector compare
This set of vector compare instructions all take an immediate as an extra argument. The immediate controls what kind of comparison is performed. There are eight comparison possible for each instruction. The vectors are compared and all comparisons that evaluate to true set all corresponding bits in the destination to 1, and false comparisons sets all the same bits to 0. This result can be used directly in VPCMOV instruction for a vectorized conditional move.

Vector conditional move
VPCMOV works as bitwise variant of the blend instructions in SSE4. For each bit in the selector 1 selects the same bit in the first source, and 0 selects the same in the second source. When used together with the XOP vector comparison instructions above this can be used to implement a vectorized ternary move, or if the second input is the same as the destination, a conditional move (CMOV).

Integer vector shift and rotate instructions
The shift instructions here differ from those in SSE2 in that they can shift each unit with a different amount using a vector register interpreted as packed signed integers. The sign indicates the direction of shift or rotate, with positive values causing left shift and negative right shift
Intel has specified a different incompatible set of variable vector shift instructions in AVX2.

Vector permute
VPPERM is a single instruction that combines the SSSE3 instruction PALIGNR and PSHUFB and adds more to both. Some compare it the Altivec instruction VPERM. It takes three registers as input, the first two are source registers and the third the selector register. Each byte in the selector selects one of the bytes in one of the two input registers for the output. The selector can also apply effects on the selected bytes such as setting it to 0, reverse the bit order, and repeating the most-significant bit. All of the effects or the input can in addition be inverted.

The VPERMIL2PD and VPERMIL2PS instructions are two source versions of the VPERMILPD and VPERMILPS  instructions in AVX which means like VPPERM they can select output from any of the fields in the two inputs.

Floating-point fraction extraction
These instructions extracts the fractional part of floating point, that is the part that would be lost in conversion to integer.

CPUs with XOP 

 AMD:
 "Heavy Equipment" processors 
 Bulldozer-based processors, Q4 2011
 Piledriver-based processors, Q4 2012
 Steamroller-based processors, Q1 2014
 Excavator-based processors (including "v2"), 2015

See also
AVX
CVT16
FMA4
SSE5
x86

Notes

References

X86 instructions
SIMD computing
Advanced Micro Devices technologies