Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmark: TurboPFor Integer Compression - Skylake i7-6700 3.4GHz #47

Open
powturbo opened this issue Oct 16, 2019 · 0 comments
Open

Benchmark: TurboPFor Integer Compression - Skylake i7-6700 3.4GHz #47

powturbo opened this issue Oct 16, 2019 · 0 comments

Comments

@powturbo
Copy link
Owner

powturbo commented Oct 16, 2019 •
edited
Loading

TurboPFor: IcApp Integer Compression Benchmark - Skylake i7-6700 3.4GHz

Inverted index testdata

docs: Document Ids

	./icapp test_collection.docs

file: max bits histogram:
00: 0.002%
01:## 1.7%
02:# 0.7%
03: 0.4%
04: 0.2%
05: 0.2%
06:# 0.5%
07:# 1.4%
08:## 1.6%
09:#### 3.6%
10:###### 6.4%
11:######## 8.1%
12:################# 17%
13:############################################# 45%
14:############# 13%

  E MB/s     size     ratio   D MB/s  function (integer size=32 bits) BOLD=Pareto
 1038.82    3017800  21.93%  2661.63 vszenc32         TurboVSimple zigzag
 1139.82    3147043  22.87%  4950.83 p4nzenc128v32    TurboPForV   zigzag
 1076.60    3147043  22.87%  3642.05 p4nzenc32        TurboPFor    zigzag
  964.09    3186374  23.15%  2922.15 p4nzzenc128v32   TurboPFor zzag/delta
  887.67    3190622  23.18%  1954.18 bitshuffleZ+lz   Transpose+zigzag+lz
 1088.78    3228329  23.46%  2693.93 tpnibbleZ+lz     Transpose+zigzag+lz
 1122.80    3239377  23.54%  5056.32 p4nd1enc128v32   TurboPForV   delta1
 1075.09    3239377  23.54%  3680.03 p4nd1enc32       TurboPFor    delta1
 1331.20    3246518  23.59%  6489.07 p4nzenc256v32    TurboPFor256 zigzag
 1034.37    3277695  23.81%  2621.08 tpbyteZ+lz       Transpose+zigzag+lz
 1135.96    3357179  24.39%  5334.62 p4ndenc128v32    TurboPForV   delta
 1075.59    3357179  24.39%  3644.94 p4ndenc32        TurboPFor    delta
 1305.82    3423417  24.87%  6158.08 p4nd1enc256v32   TurboPFor256 delta1
 1346.84    3515568  25.54%  1550.80 vbddenc32        TurboVByte zzag delt
 1311.54    3528896  25.64%  6330.87 p4ndenc256v32    TurboPFor256 delta
  920.32    3531916  25.66%  1058.47 bvzzenc32        bitio zigzag/delta
 1094.85    3673776  26.69%  3244.53 tpnibble+lz      Transpose+lz
 1079.98    3676707  26.71%  3193.34 tpbyte+lz        Transpose+lz
  981.41    3701682  26.90%  2531.42 tpnibbleX+lz     Transpose+xor+lz
  983.37    3738695  27.16%  2506.98 tpbyteX+lz       Transpose+xor+lz
 8699.94    3925193  28.52% 11144.38 v8nd1enc128v32   TByte+TPackV delta1
 9571.15    3945921  28.67% 11644.09 v8ndenc128v32    TByte+TPackV delta
 2644.76    3948726  28.69%  2783.84 vbzenc32         TurboVByte zigzag
 7443.65    4014799  29.17%  9577.81 v8nzenc128v32    TByte+TPackV zigzag
11010.65    4173952  30.33% 12030.87 v8ndenc256v32    TByte+TPackV delta
10286.48    4161012  30.23% 11723.43 v8nd1enc256v32   TByte+TPackV delta1
 3054.44    4180490  30.37%  3489.68 vbd1enc32        TurboVByte delta1
 3230.82    4181163  30.38%  3716.80 vbdenc32         TurboVByte delta
 8755.29    4215033  30.63% 10769.41 v8nzenc256v32    TByte+TPackV zigzag
  867.36    4228629  30.72%  1893.95 bitshuffleX+lz   Transpose+xor+lz
  970.41    4260129  30.95%  2366.86 bitshuffle+lz    Transpose+lz
 1062.80    4329918  31.46%  1271.79 bvzenc32         bitio zigzag
 8738.61    4406850  32.02% 10031.57 bitnzpack128v32  TurboPackV   zigzag
 4255.82    4406850  32.02%  5017.61 bitnzpack32      TurboPack    zigzag
11063.76    4724718  34.33% 10735.81 v8zenc32         TurboByte zigzag
 3235.38    4724718  34.33%  4183.38 streamvbyte zzag StreamVByte zigzag
11926.61    4819041  35.01% 11644.09 bitnzpack256v32  TurboPack256 zigzag
12041.39    4889190  35.52% 12223.19 v8d1enc32        TurboByte delta1
12661.74    4889479  35.53% 12850.90 v8denc32         TurboByte delta
10490.33    4889479  35.53% 14101.75 streamvbyte delt StreamVByte delta
  385.16    5050551  36.70%   423.20 SPDP             SPDP Floating Point
 1931.69    5265005  38.25%  8086.55 fpxenc32         TurboFloat XOR
 1350.93    5265005  38.25%  1524.01 fpfcmenc32       TurboFloat FCM
  997.49    5310026  38.58%  1184.65 fpgenc32         bitio TurboGorilla
 1657.63    5552401  40.34% 11926.61 p4nenc128v32     TurboPForV
 1578.90    5552401  40.34%  7583.09 p4nenc32         TurboPFor
 1883.32    5626136  40.88%  7616.66 FastPFor         FastPFor
   45.75    5626576  40.88%  4884.07 SimdOptPFor      FastPFor SIMD
 2247.07    5626832  40.88% 13818.59 SimdFastPFor     FastPFor SIMD
 1903.38    5638279  40.97% 13045.79 p4nenc256v32     TurboPFor256
 8291.15    5653941  41.08% 15343.71 v8nenc128v32     TurboByte+TbPackV
18400.15    5661003  41.13% 15275.60 bitnpack128v32   TurboPackV
10458.44    5661003  41.13% 10263.47 bitnpack32       TurboPack
 9331.06    5745570  41.75% 15429.72 v8nenc256v32     TurboByte+TbPackV
19115.71    5748842  41.77% 15464.40 bitnpack256v32   TurboPack256
 2062.54    6226583  45.24%  6671.50 vsenc32          TurboVSimple
  777.85    6577162  47.79%  1149.34 fpdfcmenc32      TurboFloat DFCM
  784.99    6658332  48.38%  1150.49 fp2dfcmenc32     TurboFloat DFCM 2D
 2628.09    6682141  48.55%  4124.46 vbenc32          TurboVByte scalar
 3850.95    6705452  48.72%  6507.48 maskeydvbyte     MasedVByte SIMD
10949.33    6790511  49.34% 11527.06 bitnd1pack128v32 TurboPackV   delta1
 6674.74    6790511  49.34%  8212.00 bitnd1pack32     TurboPack    delta1
12169.15    6816223  49.52% 11854.70 bitndpack128v32  TurboPackV   delta
 7004.23    6816223  49.52%  9001.51 bitndpack32      TurboPack    delta
11498.17    7511530  54.58% 14442.09 v8enc32          TurboByte SIMD
10769.41    7511530  54.58% 14321.86 streamvbyte      StreamVByte SIMD
13681.22    8062094  58.58% 10710.75 bitnd1pack256v32 TurboPack256 delta1
14262.50    8076334  58.68% 10854.35 bitndpack256v32  TurboPack256 delta
  475.40    9044107  65.71%  2907.33 lz               lz
 1427.58   10833105  78.71%  5124.09 srlez32          TurboRLE32 ESC zzag
 1681.94   13712109  99.63%  5869.22 srlex32          TurboRLE32 ESC xor
 1869.00   13763312 100.00% 17532.88 srle32           TurboRLE32 ESC
  298.29   13763312 100.00% 17532.88 trle             TurboRLE
  216.18   13763312 100.00% 17532.88 trlex            TurboRLE   xor
  200.00   13763312 100.00% 17532.88 trlez            TurboRLE   zigzag
17334.15   13763312 100.00% 17510.58 memcpy           memcpy
 9151.14   13763312 100.00% 11595.04 tpenc            Byte transpose
 8891.03   13763312 100.00% 11054.87 tp4enc           Nibble transpose
 4135.61   13763312 100.00%  4112.13 bitshuffle       Bit transpose
                                                      * : external library

freqs: Term Frequencies

	./icapp test_collection.freqs

file: max bits histogram:
00: 0.001%
01:########################### 27%
02:############ 12%
03:###### 6.0%
04:### 3.1%
05:## 1.6%
06:# 1.0%
07:# 1.0%
08:# 0.9%
09:## 1.9%
10:### 3.2%
11:#### 4.1%
12:######## 8.5%
13:####################### 23%
14:###### 6.5%

  E MB/s     size     ratio   D MB/s  function (integer size=32 bits)
   91.61    1692992  12.30%  4615.46 SimdOptPFor      FastPFor SIMD
 1204.45    1722849  12.52%  4061.17 vsenc32          TurboVSimple
 1402.56    1730336  12.57%  8397.38 p4nenc128v32     TurboPForV
 1339.76    1730336  12.57%  4979.49 p4nenc32         TurboPFor
 1343.42    1778796  12.92%  6106.17 FastPFor         FastPFor
 1447.85    1779168  12.93% 12169.15 SimdFastPFor     FastPFor SIMD
 1662.23    1780534  12.94% 12603.76 p4nenc256v32     TurboPFor256
 1701.06    1907713  13.86%  3908.92 tpnibble+lz      Transpose+lz
 1084.24    1922544  13.97%  2792.88 vszenc32         TurboVSimple zigzag
 1435.32    1955549  14.21%  2904.88 tpnibbleX+lz     Transpose+xor+lz
 1023.67    1958057  14.23%  2334.74 bitshuffleX+lz   Transpose+xor+lz
 1194.83    1962095  14.26%  3022.91 bitshuffle+lz    Transpose+lz
 1185.37    1996059  14.50%  5303.78 p4nzenc128v32    TurboPForV   zigzag
 1126.29    1996059  14.50%  3584.19 p4nzenc32        TurboPFor    zigzag
 1364.32    2087485  15.17%  7105.47 p4nzenc256v32    TurboPFor256 zigzag
 1081.60    2130744  15.48%  4695.77 fpxenc32         TurboFloat XOR
  877.76    2130744  15.48%  1330.30 fpfcmenc32       TurboFloat FCM
 1326.33    2145784  15.59%  2916.57 tpnibbleZ+lz     Transpose+zigzag+lz
 1293.42    2219153  16.12%  3770.77 tpbyte+lz        Transpose+lz
 1015.97    2259515  16.42%  1055.63 bvzenc32         bitio zigzag
 1003.67    2300817  16.72%  2364.83 bitshuffleZ+lz   Transpose+zigzag+lz
 1128.69    2381528  17.30%  2856.05 tpbyteX+lz       Transpose+xor+lz
 1021.17    2398296  17.43%  2984.24 p4nzzenc128v32   TurboPFor zzag/delta
 1137.28    2407925  17.50%  2936.48 tpbyteZ+lz       Transpose+zigzag+lz
 1004.84    2764046  20.08%  1203.82 bvzzenc32        bitio zigzag/delta
16483.00    2784387  20.23% 17312.33 v8nenc128v32     TurboByte+TbPackV
17488.32    2804702  20.38% 17118.54 bitnpack128v32   TurboPackV
 9268.22    2804702  20.38% 10522.40 bitnpack32       TurboPack
 1670.91    2883341  20.95%  1931.15 vbddenc32        TurboVByte zazg delt
 1110.66    3029119  22.01%  1114.80 fpgenc32         bitio TurboGorilla
 8271.22    3146929  22.86% 10202.60 v8nzenc128v32    TByte+TPackV zigzag
20948.71    3150687  22.89% 16764.07 v8nenc256v32     TurboByte+TbPackV
22824.72    3189005  23.17% 16764.07 bitnpack256v32   TurboPack256
 8591.33    3189535  23.17% 10340.57 bitnzpack128v32  TurboPackV   zigzag
 4135.61    3189535  23.17%  5150.94 bitnzpack32      TurboPack    zigzag
11546.40    3451880  25.08% 11365.24 vbenc32          TurboVByte scalar
 8661.61    3457754  25.12% 16846.15 maskeydvbyte     MasedVByte SIMD
 5340.82    3475286  25.25%  5265.23 vbzenc32         TurboVByte zigzag
10611.65    3502875  25.45% 12959.80 v8nzenc256v32    TByte+TPackV zigzag
11733.42    3589870  26.08% 13349.47 bitnzpack256v32  TurboPack256 zigzag
  810.18    3925934  28.52%  3365.11 lz               lz
14087.31    4307847  31.30% 16562.34 v8enc32          TurboByte SIMD
11517.41    4307847  31.30% 16825.56 streamvbyte      StreamVByte SIMD
11099.44    4323652  31.41% 10744.19 v8zenc32         TurboByte zigzag
 4017.31    4323652  31.41%  4233.56 streamvbyte zzag StreamVByte zigzag
 1048.79    5742202  41.72%  5785.33 p4ndenc128v32    TurboPForV   delta
 1001.91    5742202  41.72%  3535.40 p4ndenc32        TurboPFor    delta
 1258.30    5773355  41.95%  7098.15 p4ndenc256v32    TurboPFor256 delta
11793.75    7227089  52.51% 12432.98 v8denc32         TurboByte delta
10271.12    7227089  52.51% 12062.49 streamvbyte delt StreamVByte delta
 8705.44    7231250  52.54% 11703.49 v8ndenc256v32    TByte+TPackV delta
 8077.06    7234307  52.56% 10949.33 v8ndenc128v32    TByte+TPackV delta
 1408.88    7343213  53.35%  1508.14 vbdenc32         TurboVByte delta
 1277.10   10319134  74.98%  5386.81 srle32           TurboRLE32 ESC
  735.30   10350497  75.20%  1170.55 fpdfcmenc32      TurboFloat DFCM
  731.12   10690707  77.68%  1161.46 fp2dfcmenc32     TurboFloat DFCM 2D
 1124.82   10963622  79.66%  4061.17 srlex32          TurboRLE32 ESC xor
  853.54   10981857  79.79%  4564.94 p4nd1enc128v32   TurboPForV   delta1
  840.30   10981857  79.79%  3156.72 p4nd1enc32       TurboPFor    delta1
 1024.06   11014587  80.03%  5551.96 p4nd1enc256v32   TurboPFor256 delta1
 1348.15   11113500  80.75%  4623.21 srlez32          TurboRLE32 ESC zzag
  169.10   11621965  84.44%   191.26 SPDP             SPDP Floating Point
 7451.71   11667495  84.77%  9261.98 v8nd1enc128v32   TByte+TPackV delta1
 8029.93   11691692  84.95%  9712.99 v8nd1enc256v32   TByte+TPackV delta1
10016.96   11716644  85.13% 10434.65 v8d1enc32        TurboByte delta1
 1137.46   13329244  96.85%  1150.97 vbd1enc32        TurboVByte delta1
11614.60   13727224  99.74% 10458.44 bitndpack128v32  TurboPackV   delta
 8464.52   13727224  99.74%  9054.81 bitndpack32      TurboPack    delta
11527.06   13747047  99.88%  9465.82 bitndpack256v32  TurboPack256 delta
16542.43   13763304 100.00% 16116.28 memcpy           memcpy
  412.58   13763304 100.00% 16116.28 trle             TurboRLE
  246.88   13763304 100.00% 16116.28 trlez            TurboRLE   zigzag
  361.16   13763304 100.00% 16097.43 trlex            TurboRLE   xor
10719.08   13763304 100.00% 11663.82 tpenc            Byte transpose
 8931.41   13763304 100.00% 11001.84 tp4enc           Nibble transpose
 4247.93   13763304 100.00%  5147.08 bitshuffle       Bit transpose
11693.55   13776743 100.10%  8954.65 bitnd1pack256v32 TurboPack256 delta1
11374.63   13790184 100.20% 10450.50 bitnd1pack128v32 TurboPackV   delta1
 8251.38   13790184 100.20%  8925.62 bitnd1pack32     TurboPack    delta1

Synthetic data: zipfian distribution

	./icapp -a1.5 -m0 -M255 -n100M ZIPF

bits histogram:
00:######################################## 40%
01:############## 14%
02:############# 13%
03:########## 10%
04:######## 7.8%
05:###### 5.7%
06:#### 4.1%
07:### 2.9%
08:## 2.1%

  E MB/s     size     ratio   D MB/s  function (integer size=32 bits)
 2368.80   62939886  15.73% 10950.80 p4nenc256v32     TurboPFor256
 1327.02   63392759  15.85%  7702.23 p4nenc128v32     TurboPForV
 1321.43   63392759  15.85%  4556.85 p4nenc32         TurboPFor
   66.39   65060504  16.27%  3077.49 SimdOptPFor      FastPFor SIMD
  606.96   73459928  18.36%  5349.24 FastPFor         FastPFor
  631.90   73469416  18.37% 10197.58 SimdFastPFor     FastPFor SIMD
 1010.83   76345141  19.09%  2866.89 vsenc32          TurboVSimple
 1745.84   79163645  19.79%  3332.50 tpnibble+lz      Transpose+lz
 1464.36   80509600  20.13%  2494.28 tpnibbleX+lz     Transpose+xor+lz
 1097.09   82870974  20.72%  2541.51 bitshuffle+lz    Transpose+lz
  975.88   83384370  20.85%  2073.53 bitshuffleX+lz   Transpose+xor+lz
 1321.55   85243365  21.31%  6370.95 p4nzenc256v32    TurboPFor256 zigzag
 1178.72   85546946  21.39%  4991.20 p4nzenc128v32    TurboPForV   zigzag
 1104.59   85546946  21.39%  3735.66 p4nzenc32        TurboPFor    zigzag
 1334.30   88589767  22.15%  2936.28 tpbyte+lz        Transpose+lz
  902.66   89283071  22.32%  2240.92 vszenc32         TurboVSimple zigzag
 1010.73   94657651  23.66%  4570.28 fpxenc32         TurboFloat XOR
  823.29   94657651  23.66%  1348.86 fpfcmenc32       TurboFloat FCM
 1284.95   95923933  23.98%  2332.20 tpbyteX+lz       Transpose+xor+lz
 1376.65   97552451  24.39%  2321.01 tpnibbleZ+lz     Transpose+zigzag+lz
 1011.41   98127316  24.53%  2129.66 bitshuffleZ+lz   Transpose+zigzag+lz
  888.50   98371167  24.59%   926.12 bvzenc32         bitio zigzag
17298.05   99910930  24.98% 12408.10 v8nenc128v32     TurboByte+TbPackV
17356.59   99910930  24.98% 12362.47 bitnpack128v32   TurboPackV
11693.51   99910930  24.98% 10137.62 bitnpack32       TurboPack
17057.57  100332929  25.08% 11994.72 v8nenc256v32     TurboByte+TbPackV
17076.50  100332929  25.08% 11170.06 bitnpack256v32   TurboPack256
 1013.86  100628462  25.16%  3083.11 p4nzzenc128v32   TurboPFor zzag/delta
11191.32  101015650  25.25% 10332.45 vbenc32          TurboVByte scalar
 6689.75  102074663  25.52%  9524.04 maskeydvbyte     MasedVByte SIMD
 1161.18  103144618  25.79%  1263.64 fpgenc32         bitio TurboGorilla
 1164.71  105990559  26.50%  2202.97 tpbyteZ+lz       Transpose+zigzag+lz
 3857.24  106284616  26.57%  4009.10 vbzenc32         TurboVByte zigzag
 9820.77  112368050  28.09% 10469.01 bitnzpack128v32  TurboPackV   zigzag
 9686.17  112368050  28.09% 10426.44 v8nzenc128v32    TByte+TPackV zigzag
 4189.84  112368050  28.09%  5317.24 bitnzpack32      TurboPack    zigzag
12560.05  112825409  28.21% 11054.92 v8nzenc256v32    TByte+TPackV zigzag
12680.30  112825409  28.21% 11036.61 bitnzpack256v32  TurboPack256 zigzag
 1650.88  116367689  29.09%  1882.50 vbddenc32        TurboVByte zazg delt
  832.48  119294130  29.82%   973.89 bvzzenc32        bitio zigzag/delta
13107.45  125000000  31.25% 12375.09 v8enc32          TurboByte SIMD
11186.62  125000000  31.25% 12123.78 streamvbyte      StreamVByte SIMD
10956.20  128705458  32.18% 10696.90 v8zenc32         TurboByte zigzag
 3208.47  128705458  32.18%  3850.37 streamvbyte zzag StreamVByte zigzag
  656.49  140353625  35.09%  2416.76 lz               lz
 1002.48  231440944  57.86%  5208.47 p4ndenc128v32    TurboPForV   delta
  959.54  231440944  57.86%  3770.10 p4ndenc32        TurboPFor    delta
 1180.67  231486347  57.87%  5985.96 p4ndenc256v32    TurboPFor256 delta
  488.03  239203583  59.80%  2701.28 trle             TurboRLE
10108.16  245851337  61.46% 10481.08 v8denc32         TurboByte delta
 9441.09  245851337  61.46% 10096.17 streamvbyte delt StreamVByte delta
 8420.70  246241959  61.56% 10398.52 v8ndenc256v32    TByte+TPackV delta
 7895.00  246632587  61.66%  9510.45 v8ndenc128v32    TByte+TPackV delta
 1142.53  262025557  65.51%  1198.36 vbdenc32         TurboVByte delta
  436.08  263233919  65.81%  1532.21 trlex            TurboRLE   xor
  315.02  263233919  65.81%  1053.77 trlez            TurboRLE   zigzag
 1085.73  291716271  72.93%  5591.59 p4nd1enc256v32   TurboPFor256 delta1
  918.79  291995716  73.00%  4686.75 p4nd1enc128v32   TurboPForV   delta1
  897.86  291995716  73.00%  3595.60 p4nd1enc32       TurboPFor    delta1
 9353.88  304132355  76.03%  9687.11 v8d1enc32        TurboByte delta1
 7881.46  304522974  76.13%  8913.05 v8nd1enc256v32   TByte+TPackV delta1
 7441.72  304913602  76.23%  8818.73 v8nd1enc128v32   TByte+TPackV delta1
 1026.88  339718007  84.93%  1063.92 vbd1enc32        TurboVByte delta1
  168.67  374829697  93.71%   189.64 SPDP             SPDP Floating Point
 1374.41  384560608  96.14%  9134.30 srle32           TurboRLE32 ESC
  709.62  387715234  96.93%  1186.37 fpdfcmenc32      TurboFloat DFCM
  710.10  388257392  97.06%  1175.15 fp2dfcmenc32     TurboFloat DFCM 2D
 1174.42  392073698  98.02%  5061.88 srlex32          TurboRLE32 ESC xor
 1433.31  393849411  98.46%  5286.18 srlez32          TurboRLE32 ESC zzag
14002.66  400000000 100.00% 14043.46 memcpy           memcpy
 3876.57  400000000 100.00%  3789.60 bitshuffle       Bit transpose
 8518.79  400000000 100.00%  9106.43 tpenc            Byte transpose
 8304.44  400000000 100.00%  9598.08 tp4enc           Nibble transpose
 9278.16  400390622 100.10%  7273.65 bitnd1pack256v32 TurboPack256 delta1
 9303.19  400390622 100.10%  7575.18 bitndpack256v32  TurboPack256 delta
 7950.55  400781247 100.20%  7644.97 bitnd1pack32     TurboPack    delta1
 8069.89  400781247 100.20%  7650.09 bitndpack32      TurboPack    delta
 9284.62  400781247 100.20%  8515.17 bitnd1pack128v32 TurboPackV   delta1
 9294.98  400781247 100.20%  8564.58 bitndpack128v32  TurboPackV   delta
Repository owner locked as resolved and limited conversation to collaborators Oct 20, 2019
Repository owner unlocked this conversation Oct 20, 2019
Repository owner locked as resolved and limited conversation to collaborators Oct 20, 2019
@powturbo powturbo changed the title TurboPFor: IcApp Integer Compression Benchmark - Skylake i7-6700 3.4GHz Benchmark: TurboPFor Integer Compression - Skylake i7-6700 3.4GHz Mar 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@powturbo