Contributed by David Fibush, January 4, 1999
This group is responsible for recommendations relating to subjective and objective measurements of audio, video and audiovisual quality. At the WP2 meeting documents are summarized, work assigned, and reports approved. Each question has a rapporteur who holds ad-hoc meetings for technical discussion and document drafting.
At the opening plenary of ITU-T SG12 WP2 there was a discussion of the objective picture quality measurement recommendation being prepared by ITU-T SG9 (cable television). The draft recommendation is planned to include four methods: KDD (Japan), Quovadis, Tektronix PQA, and the output of VQEG. The chairman of WP2 was quite concerned that such efforts were being divided between two study groups of ITU-T and requested a reaffirmation that the work should continue in SG12. There is strong support for the work to continue. Coverage of low bit rate applications was mentioned as being important however high quality video is clearly of interest as well. Therefore there is much overlap between ITU-T SG9, SG12 and ITU-R SG11 (JWP 10-11Q) with SG9 appearing to go their own way. It was reported that SG9 believes things are moving too slowly and a recommendation is needed by early next year. The four methods will be in appendices and are informative (not required by the recommendation) not normative, however many engineers in the industry may not appreciate that fact. Since SG9 appears to be accepting multiple methods, KPN is expected to provide a contribution.
1. Draft recommendation P.911, "Subjective Audiovisual Quality Assessment Methods for Multimedia Applications" was reviewed prior to submission for approval. Based on a discussion of my proposed modification to BT.500 (presented discussed at T1A1.5 but not contributed to SG12) it was agreed to add a statement regarding adjustment of operational parameters for the source and processed video.
"Operational parameters, such as signal level, for the test sequences shall match those of the alignment signals used to verify the viewing and listening conditions. Any operational adjustments performed so that source or processed sequences meet this requirement should be reported."
Various other editorial changes were agreed. One rating method, absolute category rating (ACR), uses the traditional 5-level quality scale and a 9-level quality scale. It was agreed the method isn’t actually absolute, as the viewer will have some implied reference based on his/her television viewing experience. A statement indicating there is an implied reference was added. The word "absolute" appears to have a long history hence will not be changed.
Based on a contribution by France Telecom, Com 12-61, "Study of the Influence of Experimental Context on the Relationship Between Audio, Video and Audiovisual Subjective Qualities", Appendix 1 (informative), was proposed for promotion to an annex (normative). The appendix states that various laboratories have shown that independent subjective audio and video assessments can be used to predict combined audiovisual quality. It also states that video dominates when the distortions are of the same comparable level. Prior to the experiments described in Com 12-61, France Telecom had objected to annex status of this information. Another appendix is intended to list audiovisual test sequences. None are presently available. If some are located, SMPTE will be contacted as a possible distributor.
2. Proposed modifications to P.910, "Subjective Video Quality Assessment Methods for Multimedia Applications" were discussed. This is essentially the SG12 version of ITU-R BT.500. Video classes as defined at VQEG and adopted for P.911 are added. The highest class considered by this document is TV3, that includes broadcast emission to the home. Viewing distance has a wide range from 1 to 8 picture heights to be selected based on the application. Other viewing conditions were also changed to match P.911. The above, P.911, statement on operational video adjustments was also added. Two new methods were proposed as appendices (informative). One is a double stimulus version of the ITU-R single stimulus continuous quality evaluation (SSCQE). The other is an object oriented method based on work from MPEG-4. The source documents for these methods, D86 and D87, will be communicated to JWP 10-11Q by some means. Perhaps that will help harmonize any recommendations based on these new methods.
3. Future work to be considered may include applications of the present recommendations to evaluate results of specific applications such as wide band mobile and IP communication. Evaluation of acceptable video quality with transmission errors may make sense for these applications. Document D64, "Testing the Quality of Connections Having Time Varying Impairments" on audio may have some ideas that will help for video.
4. ITU-T SG9 contributed a liaison (TD-6) presenting their subjective testing document. The response from SG12 Questions 10 and 11, encouraged continued cooperative work but did not specifically mention the SG9 objective work. A liaison statement is not allowed to comment on information obtained informally. Part of the liaison is intended to emphasize the SG12 position on objective measurements.
"Our work on measures of video quality is progressing in cooperation with ITU-R JWP 10-11Q via VQEG. A validation test is underway including ten proponents who submitted computer code by the VQEG deadline. It is the intent of VQEG to evaluate the proposed systems noting that many of them are designed for similar applications. The goal is to choose the most appropriate method to be included in ITU recommendations in order to provide a consistent approach to objective video quality measurement. It is believed this work should be of interest to SG9."
5. ITU-R WP11E (now JWP 10-11Q) contributed a liaison statement indicating that the methodologies in P.911 are identical to those of Rec. 500. In response, SG12 stated that a new recommendation was required due to the nature of joint assessment of audio and video. Specific examples were given with emphasis on the audio aspects that have no equivalence in Rec.500.
Question 11, Objective Testing
1. Minor editorial changes were made to draft recommendation P.931, "Multimedia Communications Delay, Synchronization, and Frame Rate Measurements." This does include some NTIA intellectual property (as do many of the proposed objective picture quality measurement methods.)
2. Com 12-60 is the latest version of the VQEG objective test plan. There were no comments by SG12 and no open issues within VQEG.
3. TD-27 is the latest version of the VQEG subjective test plan. The use of the same or different sets of viewers for the high and low quality processed sequences was discussed. Those present indicated a preference for two groups of viewers, however they did not believe the decision would make much difference either way.
Concern was expressed that 20/20 visual acuity may be too stringent and that there should be agreement on application of the Snellen test. Some subjective testing labs use 20/30 however if the VQEG labs are happy with 20/20, precision is always a good thing.
Subject number (section 6.1) implies the session presentation ordering based on the table in section 2.4. There would be a lack of necessary information if more than 18 viewers were used. It is suggested the hundreds column of the subject number be used for this purpose.
Completion of the processed test sequences for VQEG is being delayed by problems with the coder being used for sequences with errors. It is considered important to maintain the erred sequences. One view is that TM5 should be used as it also provides rather low quality video that may be more appropriate for systems where errors are acceptable. Editing is an acceptable way to get around the problem.
4. Planned future work:
P.RSQ, "Reduced source bandwidth double-ended objective video quality measurements." This class of measurement is needed when the source and compressed video are not available at the same location. [We call this the feature extraction method.]
"Objective video quality measurements for low bit rate applications, 1.5 Mb/s and below." This study item will cover low bit rate videoconferencing and multimedia applications.
P.TRQ, "Objective video quality measurements with transmission impairments on packet, mobile and other networks." This study item will investigate objective measurement of the effects of transmission impairments on video and audiovisual quality.
A priority was not set on these items awaiting potential contributions. The WP2 chairman indicated that the second item should be considered high priority. New Subjective evaluation methods will be required to support some of this work.
Future Meetings (in Geneva unless otherwise specified)
| VQEG: | February or March in Rome |
| ITU-T SG9 WP 1/9 and WP 2/9: | 22 – 26 March, location to be determined |
| ITU-R JWP 10-11Q: | 17 – 21 May |
| ITU-R WP11B: | 24 – 28 May |
| ITU-T SG9: | 13 – 17 September |
| ITU-T SG12: | 21 September – 1 October |
"Com" documents, also known as white documents, are distributed by hard copy to member organizations before the meeting. They may be referenced in a recommendation whereas this is not true of delayed documents. http://www.itu.ch/itudoc/itu-t/com12/contr.html contains a complete list of SG12 white documents. Soft copies are available to ITU members. Listed below are those related to this report.
[COM12-50] Rapporteur for Q.10/12: Draft new Recommendation P.911 – Subjective audiovisual quality assessment methods for multimedia applications
[COM12-61] France Telecom/CNET: Study of the influence of experiment context on the relationship between audio, video and audiovisual subjective qualities
[COM12-64] USA: Results of an audiovisual desktop video teleconferencing subjective experiment
[COM12-67] CSELT: VGEG Subjective test plan
"Delayed" documents are available for discussion and action at the meeting. Generally they have been received as late as one week before the meeting. http://www.itu.ch/itudoc/itu-t/com12/dcontr/dc-nov98.html contains a complete list of SG12 delayed documents for this meeting. Soft copies are available to ITU members. Listed below are those related to this report.
[COM12-D64] AT&T (Q14/12): Testing the quality of connections having time varying impairments
[COM12-D67] AT&T (Q11/12): Editorial changes for draft new Recommendation P.931
[COM12-D85] CSELT (Italy), (Q10/12): Proposed modifications to Recommendation P.910
[COM12-D86] CSELT (Italy), (Q10/12): Proposal for object quality evaluation
[COM12-D87] CSELT (Italy), (Q10/12): Proposal for a new test method for error robustness evaluation in video communication
"Temporary" documents are only available at the meeting and only in hard copy. They are the working documents for the meeting. Often they are based on one of the white or delayed contributions. Many become output documents either as draft recommendations or as annexes to the chairman’s report. Listed below are those related to this report.
TD-4, Response to VQEG, liaison from ITU-R WP11E
TD-5, Comments on VQEG plans and draft new recommendation P.avq, liaison from ITU-R WP11E
TD-6, Newly approved ITU-T Recommendation J.140 (J.svq), Subjective Picture Quality Assessment for Digital Cable Television Systems
TD-26, Progress report of Question 11, Objective methods for evaluating audiovisual quality in multimedia services.
TD-27, Updated VQEG Subjective Test Plan
TD-30, Progress report of Question 10, Subjective methods for evaluating audiovisual quality in multimedia services
TD-22 (GEN), Proposed modifications to Recommendation P.910 [revises COM12-D85]
TD-23 (GEN), Revised draft new Recommendation P.911, "Subjective Audiovisual Quality Assessment Methods for Multimedia Applications."
TD-25 (PLEN), Report of Working Party 2/12 [includes latest version of TD-26 and TD-30]