Xiph's Daala Compression Efficiency Update

Nine months have passed since my last comparisson of the HEVC, VP9 and Daala compression efficiency. Daala was in the development stage and is still being developed. Recently I've updated the codec to see the changes.
My apologies to the haters of PSNR and the lovers of SSIM. I'm still using PSNR.

The two revisions of Daala codec are compared in the following experiments. The newer one is a master commit on July 15, 2014. The older one is a master commit on October 3, 2014.
First of all intra compression efficiency increased. The following table contains Bjontegaard delta rates and delta PSNRs for the JCT-VC test sequences. I've changed command line interface for the means of automatisation process I'm used to. Daala encoder is configured with daala_info structure. The keyframe_rate parameter is set to 1. The experiments are held for video quality parameters of \(vq=2\cdot n,\) where \(n = [2,\:...,\:16]\).
The changes in key-frame compression efficiency of the newer version comparred to the older one are the following:

Table 1: Daala 15-07-2014 compared to Daala 03-10-2013 on intra-only configuration
Sequence            BD-RATE  BD-PSNR-Y  BD-PSNR-UV
Traffic             -23.49      1.71      -0.11
PeopleOnStreet      -21.29      1.74       0.22
Kimono              -51.30      1.98       0.24
ParkScene           -20.44      1.16       0.10
Cactus              -18.06      0.90       0.09
BQTerrace            -8.31      0.64       0.24
BasketballDrive     -32.81      1.15       0.52
RaceHorsesC           2.60     -0.16      -0.05
BQMall               -2.87      0.27       0.00
PartyScene           10.75     -0.94       0.09
BasketballDrill      -8.00      0.52       0.36
RaceHorses           -8.39     -0.88      -0.01
BQSquare              1.48     -18.2     -19.45
BlowingBubbles       -7.36      0.59       0.30
BasketballPass      -13.00      1.09       0.08
Vidyo1              -28.04      1.98       0.89
Vidyo3              -25.47      1.85       1.51
Vidyo4              -28.21      1.77       1.34
BasketballDrillText  -6.19      0.44       0.49
ChinaSpeed           23.83     -2.12       0.41
SlideEditing         22.02     -2.80      -0.07
SlideShow            -8.60      1.09       0.16

Generally the compression efficiency increase by 20-30% for the majority of test sequences. However, the compression efficiency of ChinaSpeed and SlideEditing test sequences decresed by 20% (in BD-Rate).

For inter-frame coding efficiency the key frame is set to be only the first one. The rest configuration is the same. And the results follow.
Table 2: Daala 15-07-2014 compared to Daala 03-10-2013 on inter frame configuration
Sequence            BD-RATE  BD-PSNR-Y  BD-PSNR-UV
Traffic             -65.10      3.47       1.79
PeopleOnStreet      -53.88      3.55       2.58
Kimono              -83.29      3.57       1.12
ParkScene           -41.18      1.71       1.00
Cactus              -43.62      1.64       0.98
BQTerrace           -20.72      0.93       1.65
BasketballDrive     -64.49      2.24       1.09
RaceHorsesC         -24.20      1.53       1.88
BQMall              -39.31      2.14       2.73
PartyScene          -18.15      1.31       2.37
BasketballDrill     -43.91      2.15       2.21
RaceHorses          -29.16      2.00       2.45
BQSquare            -29.08      2.36       3.33
BlowingBubbles      -26.17      1.59       2.20
BasketballPass      -57.43      3.83       4.16
Vidyo1              -82.65      4.49       1.97
Vidyo3              -75.76      4.01       1.63
Vidyo4              -78.23      3.95       1.77
BasketballDrillText -45.90      2.41       3.07
ChinaSpeed          -36.41      2.90       3.23
SlideEditing        -76.28     14.97      13.09
SlideShow           -69.87      9.95       8.07

There is a significant compression efficiency gain of 20-80%. About 83% bitrate reduction achieved on Kimono and Vidyo1 test sequences.
However, comparred to the JM reference encoder Daala's performance is still rather poor.
Updated Daala's keyframe compression efficiency against JM v.18.4 "Intra HE" configuration:

Table 3: Daala 15-07-2014 intra compared to JM v.18.4 "Intra HE"
Sequence            BD-RATE  BD-PSNR-Y  BD-PSNR-UV
Traffic              17.65     -0.96      -2.13
PeopleOnStreet       20.53     -1.21      -1.22
Kimono               -1.60      0.04      -1.03
ParkScene            17.04     -0.77      -1.27
Cactus               30.30     -1.13      -1.17
BQTerrace            35.12     -2.08      -1.16
BasketballDrive      52.44     -1.13      -2.03
RaceHorsesC          40.90     -2.56      -1.59
BQMall               50.21     -2.83      -1.63
PartyScene           49.14     -3.48      -1.45
BasketballDrill      39.64     -1.81      -2.42
RaceHorses           30.13     -2.99      -2.17
BQSquare             55.50     -9.56      -7.00
BlowingBubbles       37.50     -2.18      -1.77
BasketballPass       39.72     -2.24      -3.13
Vidyo1               42.59     -1.91      -1.42
Vidyo3               59.39     -2.74      -0.80
Vidyo4               43.91     -1.79      -1.25
BasketballDrillText  47.43     -2.25      -3.05
ChinaSpeed           91.41     -6.21      -2.84
SlideEditing        121.43    -10.46      -3.95
SlideShow           270.12     -9.48     -10.55

Obviously Daala provides 1.5 times higher bitrate compared to the JM encoder. Moreover, on the SlideEditing and SlideShow test sequences the bitrate is 2-3 times higher. Intra compression efficiency of the Daala coder seems to remain not efficient enough.

Table 4: Daala 15-07-2014 inter compared to JM v.18.4 "RA B HE"
Sequence            BD-RATE  BD-PSNR-Y  BD-PSNR-UV
Traffic             149.34     -3.12      -3.51
PeopleOnStreet       68.13     -2.46      -2.33
Kimono                3.37     -0.13      -2.48
ParkScene           149.09     -3.15      -3.25
Cactus              199.99     -2.43      -2.04
BQTerrace           297.19     -3.23      -2.48
BasketballDrive      87.29     -1.29      -2.78
RaceHorsesC         110.55     -3.60      -2.70
BQMall              200.56     -4.70      -3.10
PartyScene          188.62     -5.70      -3.32
BasketballDrill     173.82     -4.06      -4.43
RaceHorses          113.98     -4.32      -3.73
BQSquare            369.81     -7.43      -2.80
BlowingBubbles      172.57     -4.56      -3.91
BasketballPass      151.03     -4.59      -4.43
Vidyo1              302.75     -3.61      -2.92
Vidyo3              243.38     -3.62      -2.57
Vidyo4              241.21     -3.06      -3.27
BasketballDrillText 193.53     -4.59      -5.34
ChinaSpeed          287.76     -9.32      -5.97
SlideEditing        378.40    -21.32     -11.34
SlideShow           382.95    -14.45     -18.80

The inter-frame compression performance of Daala is compared to the JM v.18.4 in "Random Access B HE" configuration. The overall compression efficiency of the Daala video encoder is 2-5 times lower than the compression efficiency of the JM reference encoder.