SMT-COMP

The International Satisfiability Modulo Theories (SMT) Competition.

GitHub

Home
Introduction
Benchmark Submission
Publications
SMT-LIB
Competitions by Year

SMT-COMP 2020

Rules
Benchmarks
Tools
Specs
Participants
Results
Slides

Largest Contribution Ranking - Single Query Track

Page generated on 2020-07-04 11:47:22 +0000

Winners

Sequential Performance Parallel Performance SAT Performance (parallel) UNSAT Performance (parallel) 24 seconds Performance (parallel)
CVC4 CVC4 CVC4 Yices2 Yices2

Sequential Performance

Solver Correct Score Time Score Division
CVC4 0.01761516 0.01919632 UFNIA
Yices2 0.01118916 0.03718544 QF_NIA
CVC4 0.00497633 0.00170181 UFDT
Vampire 0.00412451 0.00686574 UF
CVC4 0.00342431 0.01713929 QF_NRA
Vampire 0.0025714 0.01935183 AUFDTLIRA
CVC4 0.00181755 0.00569101 LRA
CVC4 0.00151994 0.001461 UFDTLIA
CVC4 0.00147308 0.00373823 LIA
CVC4 0.00142444 0.00446442 UFDTLIRA
CVC4 0.00140509 0.00176024 AUFDTLIA
Bitwuzla 0.00115373 0.05401255 QF_BV
Yices2 0.00106411 0.00586298 QF_IDL
Vampire 0.00100642 0.00170775 UFLIA
CVC4 0.00059623 0.01224757 QF_LIA
Vampire 0.00055049 0.00532236 UFDTNIRA
Vampire 0.00054497 0.00398073 AUFLIA
CVC4 0.00051206 0.00587804 AUFLIRA
Vampire 0.00041205 0.00014581 AUFNIRA
Vampire 0.00040371 0.00062037 AUFDTNIRA
Vampire 0.00026663 0.00071091 NRA
Yices2 0.00021965 0.00025246 QF_UFNRA
COLIBRI 0.00021512 0.00166482 QF_FP
Yices2 0.00021294 0.00396928 QF_UFIDL
CVC4 0.0001959 0.00264385 QF_UFNIA
Yices2 0.00017782 0.000165 QF_AUFBV
OpenSMT 0.00013699 0.00217853 QF_LRA
CVC4 0.00010301 8.163e-05 UFIDL
Bitwuzla 9.105e-05 0.01679047 QF_ABV
CVC4 6.02e-05 0.00014427 NIA
Yices2 5.256e-05 0.00051728 QF_RDL
Bitwuzla 4.542e-05 0.00400155 QF_ABVFP
Bitwuzla 3.399e-05 0.00146214 QF_UFBV
SMTInterpol 2.266e-05 0.00042953 ALIA
Yices2 1.322e-05 3.51e-05 QF_LIRA
SMT-RAT 1.133e-05 1.998e-05 QF_NIRA
Yices2 0.0 0.02462467 QF_UF
Yices2 0.0 0.00783185 QF_AUFLIA
Yices2 0.0 0.00472826 QF_UFLIA
Yices2 0.0 0.00433178 QF_AX
COLIBRI 0.0 0.00254385 QF_BVFP
Yices2 0.0 0.0020264 QF_ALIA
Yices2 0.0 0.00013373 QF_UFLRA
Vampire 0.0 1.505e-05 UFDTNIA
Bitwuzla 0.0 1.447e-05 QF_UFFP
veriT 0.0 0.0 UFLRA

Parallel Performance

Solver Correct Score Time Score Division
CVC4 0.01239715 0.01801826 UFNIA
Yices2 0.01118916 0.0371324 QF_NIA
Vampire 0.00438695 0.00765573 UF
CVC4 0.00422692 0.00154879 UFDT
CVC4 0.00342431 0.01712086 QF_NRA
Vampire 0.0028347 0.02391578 AUFDTLIRA
CVC4 0.00174333 0.00562724 LRA
CVC4 0.00149912 0.00144906 UFDTLIA
CVC4 0.00147308 0.00373823 LIA
CVC4 0.00140509 0.00175325 AUFDTLIA
CVC4 0.00140493 0.00441375 UFDTLIRA
Vampire 0.00110292 0.00259792 AUFDTNIRA
Yices2 0.00106411 0.00585879 QF_IDL
Vampire 0.00104069 0.00177522 UFLIA
Bitwuzla 0.00083114 0.03920012 QF_BV
Vampire 0.00063675 0.00657041 UFDTNIRA
CVC4 0.00059623 0.01212927 QF_LIA
Vampire 0.00057043 0.00420203 AUFLIA
Vampire 0.00046659 0.00015611 AUFNIRA
Vampire 0.00026663 0.00071206 NRA
Yices2 0.00021965 0.00025245 QF_UFNRA
COLIBRI 0.00021512 0.00166441 QF_FP
Yices2 0.00021294 0.00396346 QF_UFIDL
CVC4 0.0001959 0.00264282 QF_UFNIA
Yices2 0.00017782 0.000165 QF_AUFBV
OpenSMT 0.00013699 0.00218286 QF_LRA
CVC4 0.00010301 7.948e-05 UFIDL
Bitwuzla 9.105e-05 0.01678771 QF_ABV
CVC4 6.02e-05 0.00014251 NIA
Yices2 5.256e-05 0.00051622 QF_RDL
CVC4 4.877e-05 0.00262108 AUFLIRA
Bitwuzla 4.542e-05 0.00399917 QF_ABVFP
Bitwuzla 3.399e-05 0.00146204 QF_UFBV
SMTInterpol 2.266e-05 0.00042962 ALIA
Yices2 1.322e-05 3.51e-05 QF_LIRA
SMT-RAT 1.133e-05 1.998e-05 QF_NIRA
Yices2 0.0 0.02304814 QF_UF
Yices2 0.0 0.00719064 QF_AUFLIA
Yices2 0.0 0.00482338 QF_UFLIA
Yices2 0.0 0.00412664 QF_AX
COLIBRI 0.0 0.00255708 QF_BVFP
Yices2 0.0 0.00194862 QF_ALIA
Yices2 0.0 0.00013109 QF_UFLRA
Vampire 0.0 1.504e-05 UFDTNIA
Bitwuzla 0.0 1.473e-05 QF_UFFP
veriT 0.0 0.0 UFLRA

SAT Performance

Solver Correct Score Time Score Division
CVC4 0.04946516 0.00104247 UFDTLIRA
SMTInterpol 0.0309754 6.8e-05 UFLIA
CVC4 0.01596614 0.00091201 UFDT
CVC4 0.00898241 0.00098488 UFNIA
Vampire 0.00869601 0.00463759 UF
CVC4 0.00609043 0.00583108 QF_NIA
CVC4 0.00351328 0.00219681 QF_NRA
CVC4 0.00329266 0.00146807 LRA
CVC4 0.00286683 0.00145841 LIA
CVC4 0.00260422 0.00017337 AUFLIA
CVC4 0.00222095 0.00110921 AUFDTLIA
Bitwuzla 0.00170867 0.0010975 QF_BV
Yices2 0.00155561 0.00237921 QF_IDL
CVC4 0.00102104 0.00148807 QF_LIA
SMTInterpol 0.00043059 2.265e-05 ALIA
Yices2 0.00023796 0.00022496 QF_UFNRA
Yices2 0.00021404 3.053e-05 QF_AUFBV
SMTInterpol 0.00018886 1.939e-05 UFIDL
OpenSMT 0.00014798 0.00019015 QF_LRA
Bitwuzla 0.00012184 2.796e-05 QF_ABVFP
Yices2 9.853e-05 0.00020718 QF_UFNIA
COLIBRI 8.679e-05 0.00013745 QF_FP
Bitwuzla 8.299e-05 0.00027933 QF_ABV
CVC4 8.026e-05 8.523e-05 NIA
Yices2 7.108e-05 9.915e-05 QF_RDL
Yices2 1.906e-05 2.529e-05 QF_UFBV
Yices2 0.0 3.0e-06 QF_ALIA
Yices2 0.0 2.23e-06 QF_UFLIA
Yices2 0.0 1.79e-06 QF_UF
Bitwuzla 0.0 1.54e-06 QF_BVFP
Yices2 0.0 1.36e-06 QF_UFLRA
Yices2 0.0 7.8e-07 QF_AUFLIA
Yices2 0.0 2.1e-07 QF_AX
Yices2 0.0 1.4e-07 QF_UFIDL
Yices2 0.0 1.0e-08 QF_LIRA

UNSAT Performance

Solver Correct Score Time Score Division
Yices2 0.02733943 0.00898651 QF_NIA
CVC4 0.0130312 0.01389215 UFNIA
Vampire 0.00504379 0.01688906 UFDTLIRA
CVC4 0.00334557 0.00302499 QF_NRA
Vampire 0.0028347 0.02391578 AUFDTLIRA
Vampire 0.00177968 0.00084407 UF
CVC4 0.00149912 0.00144906 UFDTLIA
Vampire 0.00129759 0.00053154 UFDT
Vampire 0.00110292 0.00259792 AUFDTNIRA
UltimateEliminator+MathSAT 0.00105242 0.00112037 LRA
Vampire 0.00104445 0.00176732 UFLIA
Vampire 0.00063675 0.00657041 UFDTNIRA
Vampire 0.0006335 0.00223573 AUFLIA
CVC4 0.00061807 0.00023849 QF_UFNIA
Vampire 0.00046659 0.00015611 AUFNIRA
COLIBRI 0.00037246 0.00034997 QF_FP
Yices2 0.00034341 0.00074459 QF_BV
CVC4 0.00031161 0.00034099 LIA
Yices2 0.00028202 0.00087521 QF_UFIDL
Vampire 0.00026663 0.00071206 NRA
SMTInterpol 0.00021071 0.00020919 QF_LIA
CVC4 0.00019385 5.855e-05 QF_IDL
Yices2 0.00015864 0.00010202 QF_AUFBV
OpenSMT 0.00012464 0.00015194 QF_LRA
Bitwuzla 0.00010866 0.00014407 QF_ABV
Vampire 9.632e-05 2.57e-05 NIA
CVC4 8.394e-05 5.808e-05 UFIDL
Bitwuzla 8.383e-05 0.00011364 QF_UFBV
CVC4 4.877e-05 0.00262108 AUFLIRA
Yices2 3.455e-05 4.918e-05 QF_RDL
Bitwuzla 2.791e-05 0.00017301 QF_ABVFP
Yices2 1.586e-05 2.34e-05 QF_LIRA
SMT-RAT 1.133e-05 1.998e-05 QF_NIRA
Vampire 0.0 1.504e-05 UFDTNIA
Bitwuzla 0.0 1.473e-05 QF_UFFP
Yices2 0.0 1.12e-05 QF_ALIA
Yices2 0.0 6.15e-06 QF_UF
Yices2 0.0 5.83e-06 QF_UFLIA
COLIBRI 0.0 4.96e-06 QF_BVFP
Yices2 0.0 1.27e-06 QF_AX
Yices2 0.0 9.3e-07 QF_AUFLIA
CVC4 0.0 6.5e-07 ALIA
Vampire 0.0 2.6e-07 AUFDTLIA
Yices2 0.0 9.0e-08 QF_UFLRA
Yices2 0.0 7.0e-08 QF_UFNRA
veriT 0.0 0.0 UFLRA

24s Performance

Solver Correct Score Time Score Division
Yices2 0.0323688 0.04797 QF_NIA
CVC4 0.02161643 0.01742358 UFNIA
Vampire 0.01830873 0.01037074 UF
Yices2 0.00638302 0.02624168 QF_LIA
Yices2 0.00383084 0.00699209 QF_IDL
Bitwuzla 0.00341951 0.03875487 QF_BV
CVC4 0.00299737 0.0037609 UFLIA
Vampire 0.00293311 0.00139744 UFDTLIA
CVC4 0.0029014 0.01151362 QF_NRA
CVC4 0.00189386 0.00509115 LRA
Vampire 0.00168046 0.00718657 AUFLIA
CVC4 0.00149196 0.00373824 LIA
CVC4 0.00142444 0.00445122 UFDTLIRA
Vampire 0.00139894 0.0004763 UFDT
CVC4 0.00124728 0.00933054 AUFDTLIRA
CVC4 0.00100952 0.00107437 AUFDTLIA
Vampire 0.00090651 0.00021525 AUFNIRA
Yices2 0.00060881 0.00192254 QF_LRA
COLIBRI 0.00058073 0.00131597 QF_FP
Yices2 0.00043314 0.00361585 QF_UFIDL
Bitwuzla 0.00040266 0.01560617 QF_ABV
Yices2 0.00029155 0.00080413 QF_RDL
CVC4 0.00028975 0.00066855 AUFDTNIRA
CVC4 0.00027656 0.00281215 QF_UFNIA
Vampire 0.00024719 0.00064571 NRA
CVC4 0.0002286 0.00226807 UFDTNIRA
Bitwuzla 0.00017042 0.00117606 QF_UFBV
Yices2 0.00016317 3.377e-05 QF_UFNRA
Bitwuzla 0.00012565 0.00412207 QF_ABVFP
Vampire 0.00012461 0.00120435 AUFLIRA
Yices2 0.0001204 8.361e-05 QF_AUFBV
Yices2 0.00011331 0.00217465 QF_ALIA
CVC4 7.554e-05 6.292e-05 UFIDL
Yices2 5.666e-05 0.00558075 QF_UFLIA
CVC4 5.504e-05 0.00011003 NIA
COLIBRI 3.408e-05 0.00324154 QF_BVFP
SMTInterpol 2.266e-05 0.00042962 ALIA
Yices2 0.0 0.00719064 QF_AUFLIA
Yices2 0.0 0.00412664 QF_AX
Yices2 0.0 0.00319586 QF_UF
Yices2 0.0 0.00013109 QF_UFLRA
Vampire 0.0 1.504e-05 UFDTNIA
Bitwuzla 0.0 1.473e-05 QF_UFFP
Yices2 0.0 2.0e-07 QF_LIRA
veriT 0.0 0.0 UFLRA

n Non-competing.