HDL 14 July 1999 4:10 PM
G-2.1.6/96

DRAFT MEETING RECORD

Video Compression Measurements Subcommittee G-2.1.6

Audio Video Techniques Committee G-2.1

Broadcast Technology Society

Institute of Electrical and Electronics Engineers

Twelfth Meeting

U.S. Department of Commerce
Boulder Labs
325 Broadway
Boulder, CO 80303

May 3, 1999

The meeting was called to order by Interim Chairman Alan Godber at 10:28 AM.

The agenda was approved as presented.

One typographical error was fixed, changing the wording in Item 5, paragraph 4 to "aren't as powerful as methods… ".

With this correction, the minutes were approved as submitted.

Arthur Webster reported that ITU Study Group 9, at its meeting in Boston, wanted an objective video quality standard before the VQEG work was completed. Study Group 9 decided to list all of the video quality measurement proposals submitted to the Group in an informative annex. This would not include an independent validation, but would be useful as a catalog of what was available. Study Group 9 will hold off including more information until the September meeting, awaiting a VQEG decision. It was noted that none of the four proponents had provided a full algorithm and unless algorithms were included, Study Group 9 would not make annex normative. To be normative, someone would have to be able to duplicate it.

Arthur Webster will attempt to obtain a draft copy of the document [ACTION ITEM]. It was suggested a copy of the 01 May, 1999 email reporting on the Study Group 9 meeting, from Dr. Andries Hekstra of KPN Research, be included with these minutes. See Appendix C.

Reporting on VQEG progress, Arthur Webster stated all HRCs have been run and normalized by Tektronix, put on computer tapes and distributed to the proponents and objective labs. Subjective and objective testing should be done by the end of May. June 15 is the deadline for releasing all the results, coded for anonymity. While this should make them available for discussion in August, vacation schedules in Europe may make this difficult.

It was agreed that the test data should not be manipulated before it is analyzed by the statisticians. There was a discussion about the possibility of ending up with different results if different people analyze the data. If ten or twelve people analyze the data, we shouldn't end up with ten or twelve different results, otherwise, it was noted, we didn't write the test plan properly. However, nothing in the plan says what will be standardized, so there may not be a winner. There was concern that declaring a winner could prevent future development of video quality measurements. It was suggested that if there is a clear winner, we say it. Otherwise, if there are more, we say they meet criteria "X" (to be decided). There was agreement that VQEG data should be released only for people to use for their own internal methods, not to tweak and promote their system as better than the VQEG systems.

Discussion returned to the pending Study Group 9 decision. It was indicated that it might be possible for G-2.1.6 to have input from VQEG on the test results at the July G-2.1.6 meeting. Alan Godber made an official request to VQEG for this information. Arthur Webster asked Alan Godber to send an email to the VQEG reflector officially requesting data for the next meeting of IEEE G-2.1.6 and noting that we want to give input to VQEG. [ACTION ITEM] It was also suggested that this group send liaison to Study Group 9, possibly supporting VQEG. There was a discussion about the earliest date the next VQEG meeting could be held. It was agreed that it would be difficult to get the VQEG recommendation written by September, since it had to include more than a statement that a particular algorithm was the one to use. It must also say how to use those algorithms. Arthur Webster added that he needs an editor for Study Group 12. The planned document would have a set of principles, subjective/objective test plans and recommended methods.

When additional recommendations for VQEG were discussed, it was agreed to wait on this until the next meeting and allow VQEG to keep its focus on existing work.

Arthur Webster reported there was now a VQEG web site at www.crc.ca/vqeg. Doug Lung will provide a link to it on the G-2.1.6 web site. [ACTION ITEM]

Leon Stanger distributed Requirements for A JND Based Video Quality Measurement Method, Leon Stanger, 29 April 1999 (IEEE Doc. G-2.1.6/95).

The discussion that followed covered subjective test conditions, the number of scenes to be used, the number of artifacts to be studied and methods for generating the degraded images.

While broadcast related measurements have traditionally used a single screen, ITU Rec. 500 and all VQEG work is based on a single screen, the measurements here are visual threshold based. Fine differences are better seen on a split screen. It was also noted that allowing the viewer to switch back and forth between single screen scenes works well in astronomy studies. After discussion, most agreed a single screen sequences would work best, since it would be easier to relate to other data such as that from VQEG. The only variable should be the room lighting and the screening of people. Leon Stanger said his group would establish a precise methodology for determining JND in this narrow application. Rec. 500 is not specific enough for this test.

There was a request for more different scenes per artifact. However, for this initial proof of principal, we should see if the method works with three scenes. Some felt doing different impairments was more important, others felt more scenes were more important. Interim Chairman Alan Godber suggested we add some of these points as additional items for future tests. Prove the concept first, then develop it further. We need to make this clear in the document.

Methods for generating the degraded sequences were discussed. John Libert explained a series of gradations could be created by taking a weighted sum of material encoded at 4 and 15 Mbps. He used the public domain test model 5 MPEG encoder for encoding the sequences. This should make it possible for anyone to reproduce the results. There was concern that sequences generated this way wouldn't have a pure impairment. In this end, this was determined not to be important, so long as we say, for example, that the impairment is "predominantly" blocking, not "pure" blocking. It was also noted that real encoders don't create pure impairments. It was suggested that we don't specify impairments, but rather say we use test model 5 with these scenes, because in these scenes one artifact predominates.

The use of split screens versus sequences for determining JND was debated again. It was emphasized that we need to prove there is no degradation in the split screen process and be careful about introducing a left-right bias. After much discussion, there was no agreement. Task Force chairman Leon Stanger said, if anyone felt strongly enough about the comments covered here they should change the text to exactly what they want and submit it back to the task force.

Leon Stanger was complimented on his work on this project and received a round of applause from committee members. The first round of changes will be released May 10, the document will be resubmitted June 1, with a goal of closing it in June. Leon Stanger will send out an email with the time line. [ACTION ITEM]

John Libert presented the results of his tests on interpolating degraded video by taking a weighted sum of material encoded at 4 and 15 Mbps, as described in Item 6. Video sequences with degradations created using this method were shown to the committee members present.

After the presentation, discussion continued on the next course of action. Obtaining scenes will be a problem. While synthetic scenes are an option, natural scenes are important. A recording medium is needed, preferably something that can record full resolution high definition. Current work uses SGI workstations with Exabyte or DAT data tapes. There was agreement that 8-bit data width was all that was needed. It was mentioned that it would be necessary to borrow a video recorder. Compression may be allowable, if it is lossless. Possible sources for recorders were Jay Ballard or Phil Livingston at Panasonic. Doug Lung offered to see if Sony Pictures could help. Interim Chairman Alan Godber stated we need a write-up from John Libert's Task Force showing what we are doing before we approach manufacturers for help.

When looking for source material, it was noted that since the first tests were proof of principle, not everything needed to be perfect to begin with. For testing alone, we are not limited to public-domain material. It was also suggested that synthesized images might be better than real objects for these tests. They can be duplicated. A synthesized scene for blockiness can define blockiness much better than what is out there today.

The time frame for this project was discussed. It was noted that NIST would be busy with VQEG in July. While VQEG is the first priority, Interim Chairman Alan Godber stated we need to make progress before September. The first step will be John Libert's report, which is needed to solicit help. It may be possible to do this by July. John Libert will create the report and a list of what we need, within the next ten days. Other items will be out in a month or so. [ACTION ITEM]

There was no further discussion.

There was no additional business.

Due to shorter T1A1.5 meetings, if G-2.1.6 wants to hold its meeting on an adjacent day it may be necessary to move the G-2.1.6 meeting to Tuesday from Monday. The next meeting will be on either July 26 (Monday) or July 27 (Tuesday), to be determined.

The Committee thanked the Department of Commerce and NTIA/ITS for providing facilities for the meeting.

There was a motion to adjourn, which was seconded. The meeting was adjourned at 5:13 PM.

Submitted by:
H. Douglas Lung
Secretary

Draft Agenda - IEEE Compression and Processing Subcommittee G-2.1.6, Twelfth Meeting, Monday, May 3, 1999, Alan Godber, Chairman. (Word 2.0 file 216m12an.doc)

Draft Meeting Record, G-2.1.6, Compression and Processing Subcommittee, Eleventh Meeting, January 4, 1999, Eatontown Sheraton Hotel, Eatontown NJ, Doug Lung, Secretary, IEEE Doc. G-2.1.6/94, 3 May 1999.

Requirements for A JND Based Video Quality Measurement Method, Leon Stanger, 29 April 1999, IEEE Doc. G-2.1.6/95, 3 May 1999.

Date Sat, 01 May 1999 184950 +0100
From "Hekstra, A.P." A.P.Hekstra@research.kpn.com
Subject Meeting WP1/SG9 on ovq for cable industry
To "'VQEG Reflector'" ituvidq@massive.its.bldrdoc.gov
Message-id <802140B1D018D2118EB000A02461E5B6D42EBE@ntl11.research.kpn.com>
MIME-version 1.0
X-Mailer Internet Mail Service (5.5.2448.0)
Content-type text/plain; charset="iso-8859-1"