| |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| | |
| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| | |
| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.7854|± |0.0113| | |
| | | |strict-match | 5|exact_match|↑ |0.7597|± |0.0118| | |
| | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| | |
| |---------|------:|------|-----:|--------|---|-----:|---|-----:| | |
| |hellaswag| 1|none | 5|acc |↑ |0.5579|± |0.0050| | |
| | | |none | 5|acc_norm|↑ |0.7380|± |0.0044| | |
| | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| | |
| |----------------|------:|------|-----:|--------|---|-----:|---|------| | |
| |kobest_boolq | 1|none | 0|acc |↑ |0.9195|± |0.0073| | |
| | | |none | 0|f1 |↑ |0.9195|± | N/A| | |
| |kobest_copa | 1|none | 0|acc |↑ |0.6670|± |0.0149| | |
| | | |none | 0|f1 |↑ |0.6664|± | N/A| | |
| |kobest_hellaswag| 1|none | 0|acc |↑ |0.4380|± |0.0222| | |
| | | |none | 0|acc_norm|↑ |0.5600|± |0.0222| | |
| | | |none | 0|f1 |↑ |0.4349|± | N/A| | |
| |kobest_sentineg | 1|none | 0|acc |↑ |0.6650|± |0.0237| | |
| | | |none | 0|f1 |↑ |0.6233|± | N/A| | |
| |kobest_wic | 1|none | 0|acc |↑ |0.6500|± |0.0134| | |
| | | |none | 0|f1 |↑ |0.6479|± | N/A| | |
| | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| | |
| |----------------|------:|------|-----:|--------|---|-----:|---|------| | |
| |kobest_boolq | 1|none | 5|acc |↑ |0.9145|± |0.0075| | |
| | | |none | 5|f1 |↑ |0.9145|± | N/A| | |
| |kobest_copa | 1|none | 5|acc |↑ |0.7050|± |0.0144| | |
| | | |none | 5|f1 |↑ |0.7045|± | N/A| | |
| |kobest_hellaswag| 1|none | 5|acc |↑ |0.4600|± |0.0223| | |
| | | |none | 5|acc_norm|↑ |0.5540|± |0.0223| | |
| | | |none | 5|f1 |↑ |0.4564|± | N/A| | |
| |kobest_sentineg | 1|none | 5|acc |↑ |0.9496|± |0.0110| | |
| | | |none | 5|f1 |↑ |0.9496|± | N/A| | |
| |kobest_wic | 1|none | 5|acc |↑ |0.7294|± |0.0125| | |
| | | |none | 5|f1 |↑ |0.7258|± | N/A| | |
| | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr| | |
| |------------------------------------------------------------|------:|----------|-----:|-----------|---|-----:|---|-----:| | |
| |kmmlu_direct_accounting | 2|none | 5|exact_match|↑ |0.5200|± |0.0502| | |
| |kmmlu_direct_agricultural_sciences | 2|none | 5|exact_match|↑ |0.3700|± |0.0153| | |
| |kmmlu_direct_aviation_engineering_and_maintenance | 2|none | 5|exact_match|↑ |0.4630|± |0.0158| | |
| |kmmlu_direct_biology | 2|none | 5|exact_match|↑ |0.3680|± |0.0153| | |
| |kmmlu_direct_chemical_engineering | 2|none | 5|exact_match|↑ |0.4700|± |0.0158| | |
| |kmmlu_direct_chemistry | 2|none | 5|exact_match|↑ |0.4733|± |0.0204| | |
| |kmmlu_direct_civil_engineering | 2|none | 5|exact_match|↑ |0.3630|± |0.0152| | |
| |kmmlu_direct_computer_science | 2|none | 5|exact_match|↑ |0.7320|± |0.0140| | |
| |kmmlu_direct_construction | 2|none | 5|exact_match|↑ |0.3620|± |0.0152| | |
| |kmmlu_direct_criminal_law | 2|none | 5|exact_match|↑ |0.4050|± |0.0348| | |
| |kmmlu_direct_ecology | 2|none | 5|exact_match|↑ |0.4860|± |0.0158| | |
| |kmmlu_direct_economics | 2|none | 5|exact_match|↑ |0.6077|± |0.0430| | |
| |kmmlu_direct_education | 2|none | 5|exact_match|↑ |0.7100|± |0.0456| | |
| |kmmlu_direct_electrical_engineering | 2|none | 5|exact_match|↑ |0.3210|± |0.0148| | |
| |kmmlu_direct_electronics_engineering | 2|none | 5|exact_match|↑ |0.5450|± |0.0158| | |
| |kmmlu_direct_energy_management | 2|none | 5|exact_match|↑ |0.3920|± |0.0154| | |
| |kmmlu_direct_environmental_science | 2|none | 5|exact_match|↑ |0.3090|± |0.0146| | |
| |kmmlu_direct_fashion | 2|none | 5|exact_match|↑ |0.4800|± |0.0158| | |
| |kmmlu_direct_food_processing | 2|none | 5|exact_match|↑ |0.4400|± |0.0157| | |
| |kmmlu_direct_gas_technology_and_engineering | 2|none | 5|exact_match|↑ |0.3590|± |0.0152| | |
| |kmmlu_direct_geomatics | 2|none | 5|exact_match|↑ |0.3890|± |0.0154| | |
| |kmmlu_direct_health | 2|none | 5|exact_match|↑ |0.6200|± |0.0488| | |
| |kmmlu_direct_industrial_engineer | 2|none | 5|exact_match|↑ |0.4780|± |0.0158| | |
| |kmmlu_direct_information_technology | 2|none | 5|exact_match|↑ |0.7130|± |0.0143| | |
| |kmmlu_direct_interior_architecture_and_design | 2|none | 5|exact_match|↑ |0.5970|± |0.0155| | |
| |kmmlu_direct_korean_history | 2|none | 5|exact_match|↑ |0.3500|± |0.0479| | |
| |kmmlu_direct_law | 2|none | 5|exact_match|↑ |0.4640|± |0.0158| | |
| |kmmlu_direct_machine_design_and_manufacturing | 2|none | 5|exact_match|↑ |0.4800|± |0.0158| | |
| |kmmlu_direct_management | 2|none | 5|exact_match|↑ |0.6190|± |0.0154| | |
| |kmmlu_direct_maritime_engineering | 2|none | 5|exact_match|↑ |0.4750|± |0.0204| | |
| |kmmlu_direct_marketing | 2|none | 5|exact_match|↑ |0.8010|± |0.0126| | |
| |kmmlu_direct_materials_engineering | 2|none | 5|exact_match|↑ |0.4960|± |0.0158| | |
| |kmmlu_direct_math | 2|none | 5|exact_match|↑ |0.3600|± |0.0278| | |
| |kmmlu_direct_mechanical_engineering | 2|none | 5|exact_match|↑ |0.3960|± |0.0155| | |
| |kmmlu_direct_nondestructive_testing | 2|none | 5|exact_match|↑ |0.4500|± |0.0157| | |
| |kmmlu_direct_patent | 2|none | 5|exact_match|↑ |0.4000|± |0.0492| | |
| |kmmlu_direct_political_science_and_sociology | 2|none | 5|exact_match|↑ |0.5600|± |0.0287| | |
| |kmmlu_direct_psychology | 2|none | 5|exact_match|↑ |0.4790|± |0.0158| | |
| |kmmlu_direct_public_safety | 2|none | 5|exact_match|↑ |0.3710|± |0.0153| | |
| |kmmlu_direct_railway_and_automotive_engineering | 2|none | 5|exact_match|↑ |0.3520|± |0.0151| | |
| |kmmlu_direct_real_estate | 2|none | 5|exact_match|↑ |0.4750|± |0.0354| | |
| |kmmlu_direct_refrigerating_machinery | 2|none | 5|exact_match|↑ |0.3710|± |0.0153| | |
| |kmmlu_direct_social_welfare | 2|none | 5|exact_match|↑ |0.6170|± |0.0154| | |
| |kmmlu_direct_taxation | 2|none | 5|exact_match|↑ |0.3950|± |0.0347| | |
| |kmmlu_direct_telecommunications_and_wireless_technology | 2|none | 5|exact_match|↑ |0.6150|± |0.0154| | |
| |kmmlu_hard_accounting | 2|none | 5|acc |↑ |0.2826|± |0.0671| | |
| | | |none | 5|acc_norm |↑ |0.2826|± |0.0671| | |
| |kmmlu_hard_agricultural_sciences | 2|none | 5|acc |↑ |0.1400|± |0.0349| | |
| | | |none | 5|acc_norm |↑ |0.1400|± |0.0349| | |
| |kmmlu_hard_aviation_engineering_and_maintenance | 2|none | 5|acc |↑ |0.2200|± |0.0416| | |
| | | |none | 5|acc_norm |↑ |0.2200|± |0.0416| | |
| |kmmlu_hard_biology | 2|none | 5|acc |↑ |0.1800|± |0.0386| | |
| | | |none | 5|acc_norm |↑ |0.1800|± |0.0386| | |
| |kmmlu_hard_chemical_engineering | 2|none | 5|acc |↑ |0.1500|± |0.0359| | |
| | | |none | 5|acc_norm |↑ |0.1500|± |0.0359| | |
| |kmmlu_hard_chemistry | 2|none | 5|acc |↑ |0.3000|± |0.0461| | |
| | | |none | 5|acc_norm |↑ |0.3000|± |0.0461| | |
| |kmmlu_hard_civil_engineering | 2|none | 5|acc |↑ |0.1800|± |0.0386| | |
| | | |none | 5|acc_norm |↑ |0.1800|± |0.0386| | |
| |kmmlu_hard_computer_science | 2|none | 5|acc |↑ |0.2400|± |0.0429| | |
| | | |none | 5|acc_norm |↑ |0.2400|± |0.0429| | |
| |kmmlu_hard_construction | 2|none | 5|acc |↑ |0.1800|± |0.0386| | |
| | | |none | 5|acc_norm |↑ |0.1800|± |0.0386| | |
| |kmmlu_hard_cot_accounting | 2|get-answer| 5|exact_match|↑ |0.2391|± |0.0636| | |
| |kmmlu_hard_cot_agricultural_sciences | 2|get-answer| 5|exact_match|↑ |0.0900|± |0.0288| | |
| |kmmlu_hard_cot_aviation_engineering_and_maintenance | 2|get-answer| 5|exact_match|↑ |0.2800|± |0.0451| | |
| |kmmlu_hard_cot_biology | 2|get-answer| 5|exact_match|↑ |0.1200|± |0.0327| | |
| |kmmlu_hard_cot_chemical_engineering | 2|get-answer| 5|exact_match|↑ |0.2100|± |0.0409| | |
| |kmmlu_hard_cot_chemistry | 2|get-answer| 5|exact_match|↑ |0.3100|± |0.0465| | |
| |kmmlu_hard_cot_civil_engineering | 2|get-answer| 5|exact_match|↑ |0.2400|± |0.0429| | |
| |kmmlu_hard_cot_computer_science | 2|get-answer| 5|exact_match|↑ |0.2300|± |0.0423| | |
| |kmmlu_hard_cot_construction | 2|get-answer| 5|exact_match|↑ |0.2100|± |0.0409| | |
| |kmmlu_hard_cot_criminal_law | 2|get-answer| 5|exact_match|↑ |0.1100|± |0.0314| | |
| |kmmlu_hard_cot_ecology | 2|get-answer| 5|exact_match|↑ |0.1300|± |0.0338| | |
| |kmmlu_hard_cot_economics | 2|get-answer| 5|exact_match|↑ |0.1429|± |0.0546| | |
| |kmmlu_hard_cot_education | 2|get-answer| 5|exact_match|↑ |0.0435|± |0.0435| | |
| |kmmlu_hard_cot_electrical_engineering | 2|get-answer| 5|exact_match|↑ |0.2200|± |0.0416| | |
| |kmmlu_hard_cot_electronics_engineering | 2|get-answer| 5|exact_match|↑ |0.3500|± |0.0479| | |
| |kmmlu_hard_cot_energy_management | 2|get-answer| 5|exact_match|↑ |0.3300|± |0.0473| | |
| |kmmlu_hard_cot_environmental_science | 2|get-answer| 5|exact_match|↑ |0.1500|± |0.0359| | |
| |kmmlu_hard_cot_fashion | 2|get-answer| 5|exact_match|↑ |0.2200|± |0.0416| | |
| |kmmlu_hard_cot_food_processing | 2|get-answer| 5|exact_match|↑ |0.2100|± |0.0409| | |
| |kmmlu_hard_cot_gas_technology_and_engineering | 2|get-answer| 5|exact_match|↑ |0.3200|± |0.0469| | |
| |kmmlu_hard_cot_geomatics | 2|get-answer| 5|exact_match|↑ |0.1900|± |0.0394| | |
| |kmmlu_hard_cot_health | 2|get-answer| 5|exact_match|↑ |0.2609|± |0.0936| | |
| |kmmlu_hard_cot_industrial_engineer | 2|get-answer| 5|exact_match|↑ |0.1300|± |0.0338| | |
| |kmmlu_hard_cot_information_technology | 2|get-answer| 5|exact_match|↑ |0.3600|± |0.0482| | |
| |kmmlu_hard_cot_interior_architecture_and_design | 2|get-answer| 5|exact_match|↑ |0.1600|± |0.0368| | |
| |kmmlu_hard_cot_korean_history | 2|get-answer| 5|exact_match|↑ |0.1364|± |0.0523| | |
| |kmmlu_hard_cot_law | 2|get-answer| 5|exact_match|↑ |0.1500|± |0.0359| | |
| |kmmlu_hard_cot_machine_design_and_manufacturing | 2|get-answer| 5|exact_match|↑ |0.3000|± |0.0461| | |
| |kmmlu_hard_cot_management | 2|get-answer| 5|exact_match|↑ |0.1900|± |0.0394| | |
| |kmmlu_hard_cot_maritime_engineering | 2|get-answer| 5|exact_match|↑ |0.2000|± |0.0402| | |
| |kmmlu_hard_cot_marketing | 2|get-answer| 5|exact_match|↑ |0.2000|± |0.0402| | |
| |kmmlu_hard_cot_materials_engineering | 2|get-answer| 5|exact_match|↑ |0.1500|± |0.0359| | |
| |kmmlu_hard_cot_math | 2|get-answer| 5|exact_match|↑ |0.3600|± |0.0482| | |
| |kmmlu_hard_cot_mechanical_engineering | 2|get-answer| 5|exact_match|↑ |0.2600|± |0.0441| | |
| |kmmlu_hard_cot_nondestructive_testing | 2|get-answer| 5|exact_match|↑ |0.3000|± |0.0461| | |
| |kmmlu_hard_cot_patent | 2|get-answer| 5|exact_match|↑ |0.2549|± |0.0616| | |
| |kmmlu_hard_cot_political_science_and_sociology | 2|get-answer| 5|exact_match|↑ |0.0889|± |0.0302| | |
| |kmmlu_hard_cot_psychology | 2|get-answer| 5|exact_match|↑ |0.1300|± |0.0338| | |
| |kmmlu_hard_cot_public_safety | 2|get-answer| 5|exact_match|↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_cot_railway_and_automotive_engineering | 2|get-answer| 5|exact_match|↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_cot_real_estate | 2|get-answer| 5|exact_match|↑ |0.2472|± |0.0460| | |
| |kmmlu_hard_cot_refrigerating_machinery | 2|get-answer| 5|exact_match|↑ |0.3000|± |0.0461| | |
| |kmmlu_hard_cot_social_welfare | 2|get-answer| 5|exact_match|↑ |0.3700|± |0.0485| | |
| |kmmlu_hard_cot_taxation | 2|get-answer| 5|exact_match|↑ |0.1979|± |0.0409| | |
| |kmmlu_hard_cot_telecommunications_and_wireless_technology | 2|get-answer| 5|exact_match|↑ |0.2900|± |0.0456| | |
| |kmmlu_hard_criminal_law | 2|none | 5|acc |↑ |0.2400|± |0.0429| | |
| | | |none | 5|acc_norm |↑ |0.2400|± |0.0429| | |
| |kmmlu_hard_direct_accounting | 2|none | 5|exact_match|↑ |0.3043|± |0.0686| | |
| |kmmlu_hard_direct_agricultural_sciences | 2|none | 5|exact_match|↑ |0.1400|± |0.0349| | |
| |kmmlu_hard_direct_aviation_engineering_and_maintenance | 2|none | 5|exact_match|↑ |0.2200|± |0.0416| | |
| |kmmlu_hard_direct_biology | 2|none | 5|exact_match|↑ |0.1900|± |0.0394| | |
| |kmmlu_hard_direct_chemical_engineering | 2|none | 5|exact_match|↑ |0.1600|± |0.0368| | |
| |kmmlu_hard_direct_chemistry | 2|none | 5|exact_match|↑ |0.2900|± |0.0456| | |
| |kmmlu_hard_direct_civil_engineering | 2|none | 5|exact_match|↑ |0.1800|± |0.0386| | |
| |kmmlu_hard_direct_computer_science | 2|none | 5|exact_match|↑ |0.2400|± |0.0429| | |
| |kmmlu_hard_direct_construction | 2|none | 5|exact_match|↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_direct_criminal_law | 2|none | 5|exact_match|↑ |0.2400|± |0.0429| | |
| |kmmlu_hard_direct_ecology | 2|none | 5|exact_match|↑ |0.1500|± |0.0359| | |
| |kmmlu_hard_direct_economics | 2|none | 5|exact_match|↑ |0.2857|± |0.0706| | |
| |kmmlu_hard_direct_education | 2|none | 5|exact_match|↑ |0.3478|± |0.1015| | |
| |kmmlu_hard_direct_electrical_engineering | 2|none | 5|exact_match|↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_direct_electronics_engineering | 2|none | 5|exact_match|↑ |0.2500|± |0.0435| | |
| |kmmlu_hard_direct_energy_management | 2|none | 5|exact_match|↑ |0.2100|± |0.0409| | |
| |kmmlu_hard_direct_environmental_science | 2|none | 5|exact_match|↑ |0.2000|± |0.0402| | |
| |kmmlu_hard_direct_fashion | 2|none | 5|exact_match|↑ |0.2300|± |0.0423| | |
| |kmmlu_hard_direct_food_processing | 2|none | 5|exact_match|↑ |0.1800|± |0.0386| | |
| |kmmlu_hard_direct_gas_technology_and_engineering | 2|none | 5|exact_match|↑ |0.1100|± |0.0314| | |
| |kmmlu_hard_direct_geomatics | 2|none | 5|exact_match|↑ |0.2200|± |0.0416| | |
| |kmmlu_hard_direct_health | 2|none | 5|exact_match|↑ |0.1739|± |0.0808| | |
| |kmmlu_hard_direct_industrial_engineer | 2|none | 5|exact_match|↑ |0.1900|± |0.0394| | |
| |kmmlu_hard_direct_information_technology | 2|none | 5|exact_match|↑ |0.3200|± |0.0469| | |
| |kmmlu_hard_direct_interior_architecture_and_design | 2|none | 5|exact_match|↑ |0.2100|± |0.0409| | |
| |kmmlu_hard_direct_korean_history | 2|none | 5|exact_match|↑ |0.1591|± |0.0558| | |
| |kmmlu_hard_direct_law | 2|none | 5|exact_match|↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_direct_machine_design_and_manufacturing | 2|none | 5|exact_match|↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_direct_management | 2|none | 5|exact_match|↑ |0.3100|± |0.0465| | |
| |kmmlu_hard_direct_maritime_engineering | 2|none | 5|exact_match|↑ |0.2300|± |0.0423| | |
| |kmmlu_hard_direct_marketing | 2|none | 5|exact_match|↑ |0.3300|± |0.0473| | |
| |kmmlu_hard_direct_materials_engineering | 2|none | 5|exact_match|↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_direct_math | 2|none | 5|exact_match|↑ |0.3100|± |0.0465| | |
| |kmmlu_hard_direct_mechanical_engineering | 2|none | 5|exact_match|↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_direct_nondestructive_testing | 2|none | 5|exact_match|↑ |0.1300|± |0.0338| | |
| |kmmlu_hard_direct_patent | 2|none | 5|exact_match|↑ |0.3137|± |0.0656| | |
| |kmmlu_hard_direct_political_science_and_sociology | 2|none | 5|exact_match|↑ |0.2444|± |0.0456| | |
| |kmmlu_hard_direct_psychology | 2|none | 5|exact_match|↑ |0.2600|± |0.0441| | |
| |kmmlu_hard_direct_public_safety | 2|none | 5|exact_match|↑ |0.1500|± |0.0359| | |
| |kmmlu_hard_direct_railway_and_automotive_engineering | 2|none | 5|exact_match|↑ |0.1800|± |0.0386| | |
| |kmmlu_hard_direct_real_estate | 2|none | 5|exact_match|↑ |0.2247|± |0.0445| | |
| |kmmlu_hard_direct_refrigerating_machinery | 2|none | 5|exact_match|↑ |0.1800|± |0.0386| | |
| |kmmlu_hard_direct_social_welfare | 2|none | 5|exact_match|↑ |0.3100|± |0.0465| | |
| |kmmlu_hard_direct_taxation | 2|none | 5|exact_match|↑ |0.1979|± |0.0409| | |
| |kmmlu_hard_direct_telecommunications_and_wireless_technology| 2|none | 5|exact_match|↑ |0.2500|± |0.0435| | |
| |kmmlu_hard_ecology | 2|none | 5|acc |↑ |0.1400|± |0.0349| | |
| | | |none | 5|acc_norm |↑ |0.1400|± |0.0349| | |
| |kmmlu_hard_economics | 2|none | 5|acc |↑ |0.2857|± |0.0706| | |
| | | |none | 5|acc_norm |↑ |0.2857|± |0.0706| | |
| |kmmlu_hard_education | 2|none | 5|acc |↑ |0.3478|± |0.1015| | |
| | | |none | 5|acc_norm |↑ |0.3478|± |0.1015| | |
| |kmmlu_hard_electrical_engineering | 2|none | 5|acc |↑ |0.1600|± |0.0368| | |
| | | |none | 5|acc_norm |↑ |0.1600|± |0.0368| | |
| |kmmlu_hard_electronics_engineering | 2|none | 5|acc |↑ |0.2500|± |0.0435| | |
| | | |none | 5|acc_norm |↑ |0.2500|± |0.0435| | |
| |kmmlu_hard_energy_management | 2|none | 5|acc |↑ |0.2100|± |0.0409| | |
| | | |none | 5|acc_norm |↑ |0.2100|± |0.0409| | |
| |kmmlu_hard_environmental_science | 2|none | 5|acc |↑ |0.2000|± |0.0402| | |
| | | |none | 5|acc_norm |↑ |0.2000|± |0.0402| | |
| |kmmlu_hard_fashion | 2|none | 5|acc |↑ |0.2100|± |0.0409| | |
| | | |none | 5|acc_norm |↑ |0.2100|± |0.0409| | |
| |kmmlu_hard_food_processing | 2|none | 5|acc |↑ |0.1800|± |0.0386| | |
| | | |none | 5|acc_norm |↑ |0.1800|± |0.0386| | |
| |kmmlu_hard_gas_technology_and_engineering | 2|none | 5|acc |↑ |0.1100|± |0.0314| | |
| | | |none | 5|acc_norm |↑ |0.1100|± |0.0314| | |
| |kmmlu_hard_geomatics | 2|none | 5|acc |↑ |0.2200|± |0.0416| | |
| | | |none | 5|acc_norm |↑ |0.2200|± |0.0416| | |
| |kmmlu_hard_health | 2|none | 5|acc |↑ |0.2174|± |0.0879| | |
| | | |none | 5|acc_norm |↑ |0.2174|± |0.0879| | |
| |kmmlu_hard_industrial_engineer | 2|none | 5|acc |↑ |0.1900|± |0.0394| | |
| | | |none | 5|acc_norm |↑ |0.1900|± |0.0394| | |
| |kmmlu_hard_information_technology | 2|none | 5|acc |↑ |0.3200|± |0.0469| | |
| | | |none | 5|acc_norm |↑ |0.3200|± |0.0469| | |
| |kmmlu_hard_interior_architecture_and_design | 2|none | 5|acc |↑ |0.2100|± |0.0409| | |
| | | |none | 5|acc_norm |↑ |0.2100|± |0.0409| | |
| |kmmlu_hard_korean_history | 2|none | 5|acc |↑ |0.1591|± |0.0558| | |
| | | |none | 5|acc_norm |↑ |0.1591|± |0.0558| | |
| |kmmlu_hard_law | 2|none | 5|acc |↑ |0.1700|± |0.0378| | |
| | | |none | 5|acc_norm |↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_machine_design_and_manufacturing | 2|none | 5|acc |↑ |0.1700|± |0.0378| | |
| | | |none | 5|acc_norm |↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_management | 2|none | 5|acc |↑ |0.3100|± |0.0465| | |
| | | |none | 5|acc_norm |↑ |0.3100|± |0.0465| | |
| |kmmlu_hard_maritime_engineering | 2|none | 5|acc |↑ |0.2300|± |0.0423| | |
| | | |none | 5|acc_norm |↑ |0.2300|± |0.0423| | |
| |kmmlu_hard_marketing | 2|none | 5|acc |↑ |0.3200|± |0.0469| | |
| | | |none | 5|acc_norm |↑ |0.3200|± |0.0469| | |
| |kmmlu_hard_materials_engineering | 2|none | 5|acc |↑ |0.1900|± |0.0394| | |
| | | |none | 5|acc_norm |↑ |0.1900|± |0.0394| | |
| |kmmlu_hard_math | 2|none | 5|acc |↑ |0.3100|± |0.0465| | |
| | | |none | 5|acc_norm |↑ |0.3100|± |0.0465| | |
| |kmmlu_hard_mechanical_engineering | 2|none | 5|acc |↑ |0.1700|± |0.0378| | |
| | | |none | 5|acc_norm |↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_nondestructive_testing | 2|none | 5|acc |↑ |0.1300|± |0.0338| | |
| | | |none | 5|acc_norm |↑ |0.1300|± |0.0338| | |
| |kmmlu_hard_patent | 2|none | 5|acc |↑ |0.2941|± |0.0644| | |
| | | |none | 5|acc_norm |↑ |0.2941|± |0.0644| | |
| |kmmlu_hard_political_science_and_sociology | 2|none | 5|acc |↑ |0.2667|± |0.0469| | |
| | | |none | 5|acc_norm |↑ |0.2667|± |0.0469| | |
| |kmmlu_hard_psychology | 2|none | 5|acc |↑ |0.2700|± |0.0446| | |
| | | |none | 5|acc_norm |↑ |0.2700|± |0.0446| | |
| |kmmlu_hard_public_safety | 2|none | 5|acc |↑ |0.1400|± |0.0349| | |
| | | |none | 5|acc_norm |↑ |0.1400|± |0.0349| | |
| |kmmlu_hard_railway_and_automotive_engineering | 2|none | 5|acc |↑ |0.1700|± |0.0378| | |
| | | |none | 5|acc_norm |↑ |0.1700|± |0.0378| | |
| |kmmlu_hard_real_estate | 2|none | 5|acc |↑ |0.2135|± |0.0437| | |
| | | |none | 5|acc_norm |↑ |0.2135|± |0.0437| | |
| |kmmlu_hard_refrigerating_machinery | 2|none | 5|acc |↑ |0.2000|± |0.0402| | |
| | | |none | 5|acc_norm |↑ |0.2000|± |0.0402| | |
| |kmmlu_hard_social_welfare | 2|none | 5|acc |↑ |0.3200|± |0.0469| | |
| | | |none | 5|acc_norm |↑ |0.3200|± |0.0469| | |
| |kmmlu_hard_taxation | 2|none | 5|acc |↑ |0.2083|± |0.0417| | |
| | | |none | 5|acc_norm |↑ |0.2083|± |0.0417| | |
| |kmmlu_hard_telecommunications_and_wireless_technology | 2|none | 5|acc |↑ |0.2600|± |0.0441| | |
| | | |none | 5|acc_norm |↑ |0.2600|± |0.0441| |