Auto vectorize replace_copy, replace_copy_if

[AlexGuteniev]

Work around DevCom-10606350

Benchmark result before:

----------------------------------------------------------
Benchmark                Time             CPU   Iterations
----------------------------------------------------------
rc<uint8_t>           1458 ns         1465 ns       448000
rc<uint16_t>          1647 ns         1650 ns       407273
rc<uint32_t>          1680 ns         1650 ns       407273
rc<uint64_t>          1666 ns         1674 ns       448000
rc_if<uint8_t>        1644 ns         1650 ns       407273
rc_if<uint16_t>       1686 ns         1688 ns       407273
rc_if<uint32_t>       1648 ns         1674 ns       448000
rc_if<uint64_t>       1750 ns         1726 ns       407273

After:

----------------------------------------------------------
Benchmark                Time             CPU   Iterations
----------------------------------------------------------
rc<uint8_t>            159 ns          160 ns      4977778
rc<uint16_t>           272 ns          270 ns      2488889
rc<uint32_t>           505 ns          502 ns      1120000
rc<uint64_t>          1008 ns         1004 ns       746667
rc_if<uint8_t>         166 ns          165 ns      4072727
rc_if<uint16_t>        348 ns          339 ns      2488889
rc_if<uint32_t>        566 ns          572 ns      1120000
rc_if<uint64_t>       1474 ns         1465 ns       448000

After with /arch:AVX2:

----------------------------------------------------------
Benchmark                Time             CPU   Iterations
----------------------------------------------------------
rc<uint8_t>           58.5 ns         59.4 ns     10000000
rc<uint16_t>           102 ns          103 ns      6400000
rc<uint32_t>           187 ns          188 ns      3733333
rc<uint64_t>           559 ns          558 ns      1120000
rc_if<uint8_t>        76.8 ns         76.7 ns     11200000
rc_if<uint16_t>        131 ns          132 ns      4977778
rc_if<uint32_t>        230 ns          230 ns      2986667
rc_if<uint64_t>       1691 ns         1688 ns       407273

The anomality for rc_if<uint64_t> AVX2 case it that compiler generates AVX512 code and checks for AVX512 in __isa_available and I don't have that, so it is scalar, still a bit faster due to branchless. Applies specifically to unsigned 64 bit only.

[AlexGuteniev]

I don't think we need to require the output type to be the same as the source types.

Curiously, the AVX2 auto vectorization does indeed work even for different source and destination element size, this is achieved using vpmovsx* or vpmovzx* to widen and vpshufb to compress.

Surprising the compiles goes that much far, and misses the final piece to vectorize without this PR

[StephanTLavavej]

I'm mirroring this to the MSVC-internal repo - please notify me if any further changes are pushed.

[StephanTLavavej]

Thanks for helping the compiler, said the developer who once gave a talk titled Don't Help The Compiler! 😹 🤪 🚀