Leyo commited on
Commit
7fd6492
1 Parent(s): 6c9a197

add more complete fairness tables

Browse files
Files changed (1) hide show
  1. README.md +46 -6
README.md CHANGED
@@ -358,12 +358,52 @@ The [notebook](https://huggingface.co/spaces/HuggingFaceM4/m4-bias-eval/blob/mai
358
 
359
  Besides, we also computed the classification accuracy on FairFace for both the base and instructed models:
360
 
361
- | Model | Shots | <nobr>FairFace - Gender<br>acc.</nobr> | <nobr>FairFace - Race<br>acc.</nobr> | <nobr>FairFace - Age<br>acc.</nobr> |
362
- |:---------------------|--------:|----------------------------:|--------------------------:|-------------------------:|
363
- | IDEFICS 80B | 0 | 95.8 | 64.1 | 51.0 |
364
- | IDEFICS 9B | 0 | 94.4 | 55.3 | 45.1 |
365
- | IDEFICS 80B Instruct | 0 | 95.7 | 63.4 | 47.1 |
366
- | IDEFICS 9B Instruct | 0 | 92.7 | 59.6 | 43.9 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
367
 
368
  ## Other limitations
369
 
 
358
 
359
  Besides, we also computed the classification accuracy on FairFace for both the base and instructed models:
360
 
361
+ | Model | Shots | <nobr>FairFace - Gender<br>acc.</nobr> | <nobr>FairFace - Gender<br>per bucket std.</nobr> | <nobr>FairFace - Race<br>acc.</nobr> | <nobr>FairFace - Race<br>per bucket std.</nobr> | <nobr>FairFace - Age<br>acc.</nobr> | <nobr>FairFace - Age<br>per bucket std.</nobr> |
362
+ |:---------------------|--------:|----------------------------:|-----------------------:|--------------------------:|---------------------:|-------------------------:|--------------------:|
363
+ | IDEFICS 80B | 0 | 95.8 | 1.0 | 64.1 | 16.1 | 51.0 | 2.9 |
364
+ | IDEFICS 9B | 0 | 94.4 | 2.2 | 55.3 | 13.0 | 45.1 | 2.9 |
365
+ | IDEFICS 80B Instruct | 0 | 95.7 | 2.4 | 63.4 | 25.6 | 47.1 | 2.9 |
366
+ | IDEFICS 9B Instruct | 0 | 92.7 | 6.3 | 59.6 | 22.2 | 43.9 | 3.9 |
367
+
368
+
369
+ | Model | Shots | Task | Gender | White | Black | Latino_Hispanic | East | Southeast Asian | Indian | Middle Eastern | Average |
370
+ |:---------------------|--------:|:----------------|:---------|--------:|--------:|------------------:|-------:|------------------:|---------:|-----------------:|----------:|
371
+ | IDEFICS 80B | 0 | FairFace Gender | Male | 95.4 | 96.1 | 94.4 | 96.3 | 97.2 | 94.8 | 94.4 | 95.5 |
372
+ | | 0 | | Female | 94.8 | 97.8 | 95.8 | 95.1 | 96.5 | 97.3 | 95.7 | 96.1 |
373
+ | | 0 | | Average | 95.1 | 97.0 | 95.1 | 95.7 | 96.8 | 96.1 | 95.1 | 95.8 |
374
+ | | 0 | FairFace Race | Male | 73.4 | 71.4 | 87.9 | 73.2 | 52.3 | 35.6 | 63.0 | 65.2 |
375
+ | | 0 | | Female | 63.8 | 73.7 | 82.7 | 68.6 | 33.6 | 42.4 | 58.2 | 60.4 |
376
+ | | 0 | | Average | 68.6 | 72.6 | 85.3 | 70.9 | 42.9 | 39.0 | 60.6 | 62.8 |
377
+ | | 0 | FairFace Age | Male | 53.3 | 51.9 | 50.7 | 54.7 | 48.3 | 48.4 | 52.4 | 51.4 |
378
+ | | 0 | | Female | 55.0 | 47.4 | 45.3 | 53.5 | 53.8 | 48.1 | 49.9 | 50.4 |
379
+ | | 0 | | Average | 54.1 | 49.7 | 48.0 | 54.1 | 51.1 | 48.2 | 51.1 | 50.9 |
380
+ | IDEFICS 9B | 0 | FairFace Gender | Male | 92.9 | 94.6 | 91.5 | 94.7 | 95.2 | 91.6 | 90.2 | 93.0 |
381
+ | | 0 | | Female | 94.6 | 97.5 | 95.6 | 94.1 | 96.5 | 97.5 | 96.2 | 96.0 |
382
+ | | 0 | | Average | 93.7 | 96.0 | 93.6 | 94.4 | 95.8 | 94.5 | 93.2 | 94.5 |
383
+ | | 0 | FairFace Race | Male | 54.2 | 55.5 | 70.0 | 41.3 | 50.3 | 50.6 | 75.0 | 56.7 |
384
+ | | 0 | | Female | 44.1 | 65.3 | 67.4 | 42.1 | 29.5 | 56.0 | 73.1 | 53.9 |
385
+ | | 0 | | Average | 49.1 | 60.4 | 68.7 | 41.7 | 39.9 | 53.3 | 74.0 | 55.3 |
386
+ | | 0 | FairFace Age | Male | 50.1 | 45.6 | 46.8 | 44.5 | 39.2 | 42.2 | 45.4 | 44.8 |
387
+ | | 0 | | Female | 48.1 | 43.9 | 41.3 | 48.9 | 46.7 | 42.9 | 45.7 | 45.4 |
388
+ | | 0 | | Average | 49.1 | 44.7 | 44.1 | 46.7 | 43.0 | 42.6 | 45.6 | 45.1 |
389
+ | IDEFICS 80B Instruct | 0 | FairFace Gender | Male | 94.0 | 95.9 | 92.5 | 93.0 | 96.4 | 91.9 | 92.8 | 93.8 |
390
+ | | 0 | | Female | 98.2 | 98.2 | 97.6 | 96.9 | 98.5 | 98.6 | 98.2 | 98.0 |
391
+ | | 0 | | Average | 96.1 | 97.0 | 95.1 | 95.0 | 97.5 | 95.2 | 95.5 | 95.9 |
392
+ | | 0 | FairFace Race | Male | 85.2 | 72.9 | 93.7 | 82.7 | 43.9 | 19.2 | 38.9 | 62.4 |
393
+ | | 0 | | Female | 77.5 | 80.7 | 88.8 | 79.2 | 29.0 | 28.2 | 37.9 | 60.2 |
394
+ | | 0 | | Average | 81.3 | 76.8 | 91.3 | 81.0 | 36.5 | 23.7 | 38.4 | 61.3 |
395
+ | | 0 | FairFace Age | Male | 51.2 | 48.3 | 47.6 | 50.6 | 45.8 | 44.1 | 46.8 | 47.8 |
396
+ | | 0 | | Female | 48.1 | 44.4 | 42.0 | 49.3 | 44.9 | 42.7 | 50.6 | 46.0 |
397
+ | | 0 | | Average | 49.7 | 46.4 | 44.8 | 50.0 | 45.4 | 43.4 | 48.7 | 46.9 |
398
+ | IDEFICS 9B Instruct | 0 | FairFace Gender | Male | 87.1 | 89.6 | 84.6 | 87.6 | 90.3 | 85.8 | 83.3 | 86.9 |
399
+ | | 0 | | Female | 98.7 | 99.6 | 98.8 | 98.7 | 99.0 | 99.8 | 99.0 | 99.1 |
400
+ | | 0 | | Average | 92.9 | 94.6 | 91.7 | 93.1 | 94.6 | 92.8 | 91.1 | 93.0 |
401
+ | | 0 | FairFace Race | Male | 67.1 | 62.3 | 92.5 | 77.4 | 41.1 | 24.0 | 43.5 | 58.3 |
402
+ | | 0 | | Female | 61.1 | 73.0 | 87.2 | 77.9 | 22.5 | 33.1 | 42.5 | 56.7 |
403
+ | | 0 | | Average | 64.1 | 67.6 | 89.8 | 77.6 | 31.8 | 28.5 | 43.0 | 57.5 |
404
+ | | 0 | FairFace Age | Male | 51.4 | 44.2 | 43.4 | 45.6 | 40.1 | 40.2 | 44.1 | 44.1 |
405
+ | | 0 | | Female | 51.2 | 39.3 | 38.2 | 46.7 | 44.9 | 41.1 | 43.4 | 43.6 |
406
+ | | 0 | | Average | 51.3 | 41.8 | 40.8 | 46.2 | 42.5 | 40.7 | 43.7 | 43.9 |
407
 
408
  ## Other limitations
409