HDL 3 May 1999 3:09 PM
G-2.1.6/94

APPROVED MEETING RECORD

Video Compression Measurements Subcommittee G-2.1.6

Audio Video Techniques Committee G-2.1

Broadcast Technology Society

Institute of Electrical and Electronics Engineers

Eleventh Meeting

Eatontown Sheraton Hotel
Eatontown, NJ

January 4, 1999

Interim Chairman Alan Godber called the meeting to order at 10:35 AM.

An item submitted by David Fibush on a proposed change to ITU-R BT.500 (Rec. 500) was added under Item 9, Other Business. With this addition, the agenda was approved.

The Draft Meeting Record, G-2.1.6, Compression and Processing Subcommittee, Tenth Meeting, November 2, 1998, Kissimmee, FL, IEEE Doc. G-2.1.6/90, was reviewed and approved by those present

There were no matters arising from the minutes.

David Fibush reported on the ITU-T Study Group 12 (SG12) Meetings in Geneva, November 30 - December 3, 1998.

SG12 reviewed open issues and made some suggestions. Study Group 9 (cable TV) is writing their own objective standard. It is an appendix now, but there are plans to promote it to an annex, which would make it part of the standard. The objective measurement methods include the KDD method from Japan (an adaptive SNR method), the QUOVADIS method (traditional feature extraction, using some of the original ANSI standard), the Tektronix product and finally, whatever method VQEG chooses.

SG12 has many subjective testing standards. These were updated and improved. See Agenda Item 9.

SG12 added statements that work on video quality measurements is progressing and that the goal is to choose the most appropriate method. The statements supported the VQEG work. Items approved as future work for SG12 include "Reduced source bandwidth double-ended objective video quality measurements." David Fibush commented that VQEG was not covering this, because most people agree these methods aren't as powerful as methods that have the full source available. Objective video quality measurements for low bit rate (1.5 Mbps and below) applications will also be considered.

David Fibush explained that Study Group 10 has been working on an objective measurement method for high-quality audio and focuses on broadcast applications. SG12 focuses on quality for broadband digital networks and Study Group 9 focuses on cable TV. While each group thinks their needs are unique, they really aren't.

David Fibush said that because people in ITU do not often come to SMPTE or IEEE meetings, our input to ITU is important. This group (G-2.1.6) needs to continue and support VQEG. By the next meeting, he expected to be able to say whether there is something useful here, or not. Work on the calibration tapes is important. One problem is a large number of different machines and a small number of customers.

There was a comment that there is a perception that VQEG is behind schedule. David Fibush noted that there was a problem with the software encoder for segments with errors. Also, there were problems with the sudden transition from the alignment strips to program material.

David Fibush will give the Committee a brief report on the Study Group 12 meetings. Doug Lung will post in on the G-2.1.6 web site. [Action Item - completed] See Submission to G-2.1.6 - Report on Items of Interest at the ITU-T SG12 Meetings Held in Geneva, November 30 - December 3,.David Fibush, January 4, 1999, IEEE Doc. G-2.1.6/91.

Chairman Godber asked for further thoughts on the VQEG discussion.

David Fibush said the biggest problem in the testing was the two quality ranges - low and high. The reference is the same for both. The question was, do you use one lab and one set of viewers for both or a separate lab for each one? Another issue: after a viewer has been through one set of tests, is the viewer now an "expert"? Another problem is that people have a reference based on experience, whether they see a specific reference or not. It was noted that in previous tests there was considerable variation among experts on subjective quality, even in open discussions. This is why a very large panel is needed. David Fibush remarked that Joint Working Party 10-11Q has agreed to work on the issue of expert viewers versus home viewers for more critical measurements. Subjective measurements can change from day to day.

There was a suggestion we come up with something that uses a reference sequence and compares a compressed image with that. David Fibush agreed that we need a standard set of scenes to check measurement system results. VQEG requires monotonicity.

David Fibush reported that once the VQEG test data is available, there will be a meeting to evaluate it.

Leon Stanger was not present at the meeting. Alan Godber distributed Progress Report of Task Force to Define a Unit of Measure and Means of Calibration for Video Quality Analysis, submission to G-2.1.6, January 1, 1999, IEEE Doc. G-2.1.6/92.

Items were organized into discussion points.

Discussion Point 6 - Difference vs. Preference: When comparing images, it was noted that the difference should not include the subjective judgement needed to determine a preference. Some people may prefer a degraded picture.

Discussion Point 8 - Definition of One JND: There was a discussion of using a 75% probability that a difference was observed as the definition of one JND. It was noted that the question of how many people must see the difference gets into statistics. Another option suggested was 50%, which was used in another experiment. It was agreed we need to define a level, but this will be difficult. The viewers vision will affect results. David Fibush stated an optometrist said over 90% of the people who have their eyes examined on a regular basis have corrected 20/20 vision and this is an objective measurement. Alan Godber will ask his optometrist for his opinion. [Action Item] Doug Lung will post Leon Stanger's Progress Report on the G-2.1.6 web site. [Action Item - completed]

Discussion Point 1 - Two Screens vs. One: It was stated that 10/11Q defaulted to Rec. 500, but Rec. 500 doesn't include all the characteristics of monitors. It is a complicated issue. With one monitor, there is nothing, except for someone's internal reference, to determine if something is wrong with the monitor. Since we want an absolute value, monitors must be as close as possible. Having two monitors is one way to see if there is a difference between them. If you have only one monitor, the question becomes how good is your memory. Concerns about using two monitors include the need for design changes to the viewing room and questions about the area of focus. If you use two monitors, you must have repetition.

Discussion continued on the methods for comparing scenes. It was suggested the viewing be done in the critical viewing mode, where the scenes could be played over-and-over. This would require limiting viewing to one person at a time, but that was not seen as a problem for this first trial. We can do one test, see how it works, and try different things if needed. We do need to set rules for monitor setup. It was also suggested we need to set a target date for meeting at NIST to review these tests.

Discussion Point 2 - Fractional JND: This could be done either by using a different percentage of observers or through mathematical extrapolation. It was noted that if we have an objective method that tracks through multiple JNDs, we should be able to track it down to zero, if it is linear.

Discussion Point 4 - Loss-less Distribution Format: There were comments that the whole system has to be lossless, not just the tape format. The display system must be loss-less. There is also the issue of using or converting between 8 or 10 bit data. Different formats may be appropriate for different uses - DAT or Exabyte tape for labs and manufacturers and D5 or even D1 for broadcast end users. Computer display colorimetry is a problem. The Committee must confirm the storage and display system is lossless.

Discussion Point 5 - Observers Should Have Corrected Vision: A variety of vision tests were discussed, including the conventional eye chart, contrast sensitivity chart and a computer/CRT based contrast and acuity test. We need more input to select an appropriate test. It was suggested we ask Leon Stanger to come up with some questions for optometrists.

Discussion Point 7 - Size of Screen: Various distances were discussed. It was noted that if viewers are too far away, they wouldn't be able to see mosquito noise. A distance of 3H was suggested for HDTV, to appreciate the detail, while 6H may be good enough for SDTV.

Discussion Point 9 - 10 JNDs: Terms such as VND for Very Noticeable Difference and HND for Highly Noticeable Difference were mentioned.

Discussion Point 10 - More Experts: According to the Progress Report, John Libert was going to attempt to contact the people in a list of references identified by Beau Watson. John Libert was not present to discuss this.

It was recommended we do some experiments, then come back and discuss them to see if we can come up with some practical results. A lot of definitions and setups will be needed. One possible source for data on how to selected people for the tests was the guidelines for the quality of service tests conducted by Bell Laboratories. Bellcore has the original databases, including how the population was defined, for these tests. Barry Lerich and Al Morton were mentioned as people who may be able to help us locate this information.

John Libert was not present. Doug Lung listed open issues from the last meeting, including definitions, VIRIS, and methods for merging impaired video and reference video.

The method of degrading the video for the testing was discussed. This is the most important point at this time. Apparently there is an ANSI standard that defines this, but it was unknown if there is software available to implement it. ITU notes indicate this standard was not validated. David Fibush will research this. [Action Item]

There was general agreement a real-time artifact generator was needed. One idea was for someone to be able to turn a knob to introduce degradation until it is seen. The VIRIS program needs to be located and the synthetic impairment method selected needs to be validated. The Princeton engine was mentioned as an alternative, but there seem to be some problem with it as it is based on old technology.

One goal that was stated for the May meeting was to define, and, better yet, demonstrate, a real-time artifact generator.

Alan Godber summarized the discussion to these points:

1. What is the approach to generating artifacts? What is the fall back if VIRIS doesn't work.
2. By May 3^rd, our minimum achievement should be to have a method and equipment for generating degraded sequences.
3. The ideal achievement would be to have demonstrated the generation of artifacts to the committee.
4. We need to look for additional people for the two groups working on this project. Should this be done by the chairs?

John Grigg will ask about the Bellcore source materials for Leon Stanger's group. [Action Item] It was agreed that Arthur Webster should be on both groups' lists. David Fibush offered Tektronix as a source for CD-ROMs that have the test sequences stored in a compressed, loss-less format. He will look into this and determine if they will be useful [Action Item]

There was a report on three items from Study Group 12 (see Item 5) - low bit rate applications, low bandwidth source extraction test methods and transmission impairments (in packets over a network). Broadcasters may be interested in these topics.

Al Morton reported that he had reviewed a paper on vision measurements for the IEEE Transactions on Communications. The author removed original frames and repeated frames from the film sequence pulled up to 30 frames per second (fps). However, the conclusions on perceived quality below 24 fps seemed suspect. The study showed 15 fps was acceptable to subjects. It was noted, however, that this may be due to a safety margin already built into the film by Hollywood.

David Fibush stated that ITU Rec. 500 only tells you how to set up a monitor. It does not tell you the input video has to be correct. To cover this deficiency, two items are being submitted as additions to Rec. 500. These are covered in Adjustment of video sequences to be used in conjunction with ITU-R BT.500, David Fibush, no date, IEEE Doc. G-2.1.6/93.

This came out of a discussion at the VQEG meeting, where there was some resistance to it. At Study Group 12, there was no opposition.

Al Morton made a request on behalf of ITU-R that the Joint Working Party routinely send the most recent version of Rec. 500-X after they meet. The IEEE group never received it. Some people in SG-12 didn't have the benefit of having the latest version. Non-ITU members cannot join the TIES data service to obtain these documents on-line. The current version is -8, which came out in the summer of 1998.

Chairman Alan Godber asked if we could request a copy of it for use by Committee members. Al Morton replied that since ITU has established relationships with groups with less standing than the IEEE, IEEE might already have a standing relationship. If not, it should be able to be established. David Fibush will check with the chairman of 10/11Q. We need to get this information in a timely manner. A solution would be to obtain permission to give our people the latest thing on TIES. David Fibush will look into it. [Action Item]

The next T1A1.5 meeting is May 3-7 in Boulder Colorado. It was agreed to set the G-2.1.6 meeting for May 3 in Boulder Colorado. The exact location is to be confirmed. The following T1A1.5 meeting is July 26 in Minneapolis Minnesota.

There was a motion to adjourn, which was seconded and approved. The meeting ended at 2:25 PM.

Submitted by:
H. Douglas Lung
Secretary

Draft Agenda - IEEE Compression and Processing Subcommittee G-2.1.6, Eleventh Meeting, Monday, January 4, 1999, Alan Godber, Chairman. (Word 2.0 file 216m11an.doc)

Draft Meeting Record, G-2.1.6, Compression and Processing Subcommittee, Tenth Meeting, November 2, 1998, Ramada Inn, Kissimmee FL, Doug Lung, Secretary, IEEE Doc. G-2.1.6/90, 4 January 1999.

Report on Items of Interest at the ITU-T SG12 Meetings Held in Geneva, November 30 - December 3, Submission to G-2.1.6, David Fibush, January 4, 1999, IEEE Doc. G-2.1.6/91, 4 January 1999.

Progress Report of Task Force to define A Unit of Measure and Means of Calibration for Video Quality Analysis, Leon Stanger, January 1, 1999, IEEE Doc. G-2.1.6/92, 4 January 1999.

Adjustment of video sequences to be used in conjunction with ITU-R BT.500, David Fibush, December 25, 1998, IEEE Doc. G-2.1.6/93, 4 January 1999.