G-2.1.6/45

The fourth meeting of G-2.1.6, Audio Video Techniques Committee, the Compression and Processing Subcommittee, G-2.1.6 was brought to order by Mr. Alan Godber, Chair of G-2.1 at 8:15 AM in Room 1107 of the NTIA/ITS building at 325 Broadway, Boulder, Colorado. Attendees were introduced.

Alan Godber presented the Draft Agenda for this meeting and suggested moving Item 8, Report of Task Force on Compression Measurement Methodologies, earlier in the meeting, after item 5, Report of Task Force on Compression Measurements Information Gathering. David Fibush recommended that the ITU report be done first, then, after that, the report on the Task Force and after that, Item 7, on Scope. These changes were approved by those present.

The minutes of the last meeting were reviewed. Al Morton offered one correction to the second paragraph in Item 5 of the Scope discussion on "Baseline Assumptions". The wording was corrected to read: "Al Morton concurred, noting that the experience of T1A1, in three different laboratories, was a happy one. Correlation between labs was high."

Alan Godber assigned the document number "25" to the minutes of the January 20^th, 1997 meeting. Those present approved the minutes, as amended, as a satisfactory record of the meeting.

There were no matters arising from the Minutes.

Arthur Webster presented a report on the ITU-T and ITU-R meetings. The report referenced T1A1 document 97-108. Arthur Webster is Rapporteur for Question 11. He said the group tried to include the WP11E representative in the Joint Rapporteurs Group, but ITU would not allow that at this time. WP11E produced Recommendation 500 for subjective video testing.

David Fibush emphasized the importance of the Joint Rapporteurs Group (JRG). He observed that Study Group 12 has taken the lead in the JRG because that was where most of the work has taken place. He is involved with WP11E and wants to get them included in the JRG. He also commented that this IEEE committee's work should be in cooperation with the ITU work.

Al Morton mentioned that WP11A has work in progress on audio-video synchronization and while they haven't begun to standardize methods yet, they have gone on to specify limits. In reference to video quality measurements, David Fibush said that within SG11 there is a division of efforts. The document that specifies scenes is believed to be in 11B. He pointed out the Telecommunication people haven't managed to converge yet, largely due to the difficult in getting ITU-R and ITU-T together.

Arthur Webster said that 11E wanted to be involved in the video quality measurements work, as did 10. However, he said that if too many people start to participate it will become unmanageable. Currently only Study Groups 9 and 12 are officially involved.

David Fibush pointed out what is key is that the right people get involved. Since Study Group 11 works with sound, they got involved with the addition of audio/visual applications. He was concerned, however, that the number of liaison statements may become greater than the number of documents produced.

Eric Hauch asked if there is an agenda item to make sure we don't overlap on the work. David Fibush replied that this will be discussed in the Task Force report and during Future Business.

David Fibush outlined "Objective Picture Quality Measurement Methods". His presentation covered Video Quality Measurement Terminology, Types of Objective Measurements, Measurement Methods, System Requirements, Test Materials, the Validation Plan as discussed at ITU-T WP2 Q10/11 ad-hoc meetings, and Standards Committees working on Objective Picture Quality Measurements. The slides used in this presentation are available on the G-2.1.6 web site at http://stdsbbs.ieee.org/groups/videocomp/ .

During the presentation there was additional discussion of the KPN Research measurement method. Al Morton commented that the KPN method is one of the first to work in color space. Steve Wolf added that KPN is a mixture of different systems, combining feature extraction and matrix methods. KPN sees feature extraction as a continuum. David Fibush cautioned that the KPN method requires spatial alignment.

After presenting the list of committees working on objective picture quality measurements of compressed video, David Fibush observed that the various committees we know about should be working in cooperation with each other instead of by themselves.

Steve Wolf said the validation step is a problem. To replace subjective experiments you need to know how accurate the subjective measurements are. David Fibush commented that John Beerends said there is no way to get an objective number to match a subjective result. However, it can be compared with a series of subjective measurements. Even if you have good correlation, you can still have some outlying data that will have to be dealt with. Steve Wolf replied that the problem is pulling out subjective measurements that are repeatable. David Fibush answered the subjective repeatable may be an oxymoron.

Doug Lung distributed the Summary and Report on the May 4^th meeting of the G-2.1.6 Task Force on Compression Measurement Methodologies. This Task Force meeting included a report by David Fibush on the ITU-R and ITU-T meetings between April 8^th and April 17^th, a report by Arthur Webster on ITU-R and ITU-T Study Group Meetings in April and the presentation of a Tektronix paper by David Fibush titled "Overview of Picture Quality Measurement Methods". The Task Force also reviewed other documents on objective video quality measurement of compressed video signals and discussed additional work and scope. David Fibush suggested we should have a report from the Joint Rapporteurs Group to IEEE/ANSI at their meetings in Kansas City. The Task Force decided no additional meetings were required before the August G-2.1.6 subcommittee meeting. Refer to the Summary and Report for more details.

David Fibush noted there are some restrictions in the ITU work plan. Input and output will be CCIR-601 and gain changes will be taken out. A lot of items won't be covered in the validation of the measurements. That doesn't mean they aren't important.

Arthur Webster said the goal of ITU Study Group 12 was to narrow it down to a system which measures compression effects only – pure video quality measurement – no shifts, no gain changes. These other effects won't be dealt with now.

Alan Godber stated that a lot of these things are important to broadcasters. Arthur Webster suggested we try to work on these in parallel. Al Morton added that they were trying to simplify the problem as much as possible.

Steve Wolf asked who is going to calibrate all this data? Al Morton requested clarification. Steve Wolf explained that weird things can happen in CCIR-601. David Fibush answered that we may be able to help with sorting out things like gain shifts. If the output isn't matched, then it gets pre-matched and we make the measurements. He also added that we don't speak dropped frames. We need to know which frames are which to be able to match gain. Arthur Webster added that systems that swap frames won't be included. David Fibush mentioned that stretching won't work with any matrix method.

Al Morton observed that this illustrates the distinction between broadcast systems, which don't have variable frame rates and alignment, and teleconferencing. These effects are an inextricable part of low bit rate multimedia systems. There is a difficult temporal alignment problem that has to be dealt with. He added that out of Study Group 12, they decided to work on systems where they could easily do the alignments. The initial thoughts were to produce methods that would work primarily for the broadcast community. However, other people in Study Group 12 said variable alignments had to be included. He suggested we simplify six classes from ITU Study Group 9 down to two – broadcast and multimedia..

Warner Johnston said we are interested in at least two or three, starting at 1.2 gbs, then something in the middle and something for the home. David Fibush said that to begin with, we can try to believe one measurement method will be useful for all of them. David Fibush said it looked like Study Group 9 was considering a different method for each of the six classes. We don't want to do that. Warner Johnston commented that a loss-less compression may require a completely different measurement system. David Fibush replied that indirect measurements will work with loss-less compression - we already have the methods.

Leon Stanger cautioned that a matrix is forming that is becoming so unfocused it might be difficult to do anything. All areas are important, perhaps including teleconferencing. He recommended setting a center line, focusing on standard definition 525/625 video as delivered to consumers. Given the matrix, we could spend years working on the all encompassing system or stay focused on 525 line interlaced and apply those results to 525 progressive and other definitions. To him this looked like a new round of red-herrings. Drop frames are not an issue. Time shift? Perhaps the system could give a rating but report the shift.

Bill Meeker agreed, saying this makes sense. David Fibush also agreed. We can only work with 525 now, that is all we have. He did add that people should know the total scope we will have to look at eventually. Alan Godber partially agreed, commented that we would be thought to be "a bit head in the sand" if we say were are only going to do 525 interlaced, based on recent FCC DTV developments. If we don't say now what we intend to do, people may say we aren't looking to the future. David Fibush replied that we do the simple thing first.

Steve Wolf raised the issue of whether or not we are going to include transmission errors. This has an effect on whether these systems will drop frames. He questioned if we want to work on codec performance only or do we want to include transmission performance? Leon Stanger answered that the consensus was we would not do error rate measurement unless it showed up in the picture. David Fibush said that if we measure the system in its normal performance mode, this is important. That is basic. If someone puts a signal from a VHS recorder in it, it will look poor but we won't measure that degradation. He added that while we might want to measure the results of rain fades on a satellite stream, there is no point in having a quality measurement if half the packets are going away.

Steve Wolf offered Internet video as an example where packets are dropped. Warner Johnston replied that broadcasters want 99.9999% reliability. Leon Stanger noted that this is measurable with other systems and suggested we stay focused on picture quality. If quality drops, it drops! John Grigg said we want to include the ramifications of transmission errors in terms of picture quality. How it gets through the various grades isn't important. We want to be able to quantify picture impairment. If we can include drop frames or other errors, it becomes more meaningful.

Warner Johnston observed that in broadcasting, the defining document is SMPTE 170M. If you go to a production house and give them the FCC rules limiting video levels, they will quote SMPTE specs that allow 131 IRE unit video. He commented that the transmission and compression system impairments have to fold together, otherwise you have a tool which he understands, but which he would have difficulty enforcing. Al Morton commented that these channels should have few impairments for broadcasters, but that multimedia channels can be worse. He recommended that this group look at what the broadcast channel impairments will be.

Eric Hauch said that in this group we should work on what we think the broadcast impaired channels should be. Later, T1A1.5 should work on what channel impairments should be for multimedia applications.

Steve Wolf remarked that transmission impairments are localized and they affect quality. We must make sure our systems will handle those types of impairments. David Fibush replied that spatial shift is not considered a picture impairment. It is something you have to take out and notify the user about, but you don't consider it an impairment itself. He outlined three classes: Class 1 – Codec, Class 2 – Multimedia, Class 3 – Moderate error is visible and the measurement system should tell you about this. Warner Johnston agreed, saying picture shift within the overscan ratio is meaningless. John Grigg agreed, so long as the picture shift was small. However, if it extended to a greater degree it will be noticeable and is a quality impairment. David Fibush answered that we will measure it, we will tell you what it is, but it doesn't affect the quality number. Warner Johnston added that if it shifts ten pixels on alternate frames, that's a problem. In terms of gain, that is a different category of problem. Al Morton stated it is a luxury that we can set picture quality apart from gain, spatial shift and temporal shift. He suggested we start categorizing things in simple ways we understand well.

Leon Stanger recommended the idea of a device under test, which could include the transmission system. We can't measure picture quality, we can only measure impairment. What makes this theme unique is we can look at a picture and objectively see these quality impairments. We should read, calibrate and report on things like shift but not degrade the quality reading.

Alan Godber said we need a list of what items we can separate from picture impairment measurements. Arthur Webster said we need to focus on solving a particular problem. Even though things like transmission impairments are important, there is a lot of value in picture quality measurement alone. Warner Johnston added we should be talking about one section of a tree of measures. Some of these have been done since 1940, others we won't know how to do for ten years. Bill Meeker wanted to see a list of items we can separate out from picture quality measurement. David Fibush suggested we contribute this to the Joint Rapporteurs Group. He noted the scope has been limited already, we need to add what other things we are working on.

Alan Godber suggested we have volunteers work on this by e-mail. Bill Meeker asked what should we include? Alan Godber answered "everything". David Fibush recommended listing what we are going to cover – including transmission errors up to some reasonable level.

Leon Stanger said we should list the exceptions. Everything is counted as picture degradation except items like shift, gain, etc. A 10% gain change degrades it. Adjusting some of the items like shift or gain are allowed. David Fibush stated that non-compression system errors should be reported or, alternatively, the measurement canceled.

Leon Stanger asked for a definition of the list. Alan Godber asked how we should do this? Bill Meeker suggested adding it under scope. David Fibush said we were dealing with two issues. One is what things we can actually measure and what things are outside the area we are working on. The other is system issues – HDTV, progressive versus interlaced, etc.

Bill Meeker suggested we have the scope distributed before the next meeting, perhaps by email. Alan Godber agreed, saying something must be done before the next meeting. He asked whether it should be the scope or the list. Bill Meeker answered "the whole thing." Alan Godber asked whether the list should be added under the scope writing group. Bill Meeker said yes, but circulate the revised scope document when the changes are made. Alan Godber commented that we must make sure we have the people making the list covered adequately by the scope group. The rewriting has to be done quickly to get data into the ITU, in a few weeks. David Fibush remind the group that we need to look at the scope and suggested that we look at generating the list under that item. Alan Godber agreed and offered that the list could be generated by a separate group.

Chairman Alan Godber changed the topic of discussion to classes of distribution. David Fibush remarked he does not like the bit rate distinction. He suggested video used in complete post production, video used for simple modification/editing and video for delivery to the home. If we want, we could call them contribution, primary, secondary, but these terms may already be defined elsewhere. Warner Johnston offered contribution – uncompressed or loss-less video, where 100 or more edits or modifications could be made to the material. The next level would allow some loss, limited to two or three levels of modifications. This would allow some affiliate work, such as lower third keys. The final level would be for delivery to the home and modification would consist of things like your third keys.

David Fibush asked how he would measure quality in a loss-less system if input and output pictures are identical. He questioned why we should even consider loss-less systems when doing picture quality measurements. Warner Johnston answered because things break. David Fibush replied that there is test equipment that can check this now, so why should we extend into this area. Leon Stanger suggested an intra studio level for testing, including formats such as JPEG, MPEG and DCT compression systems that have very little loss. David Fibush said the loss-less signal is what is used as a reference. Warner Johnston recommended it be in the hierarchy. Alan Godber and Warner Johnston both agreed we want to be able to see if systems have failed.

Arthur Webster said the bit rate used for nominal Class 0 would be 1.2 Gbs. Warner Johnston agreed that 270 Mbs would be too low. John Grigg would like to be able to guarantee a particular quality point for a service. Define services in terms of picture quality. Everything should be defined in terms of picture quality. David Fibush asked if bandwidth limitation would be measured by picture quality. John Grigg replied that different methods may be used to deliver a particular picture quality. If a carrier is be able to deliver the specified quality at a lower bit rate, they should have that option.

Leon Stanger listed one class as intra-studio as applying to tape machines and video servers. Inter-site would apply between studios. Another class would be distribution. Distribution would apply to distribution to the consumer. Warner Johnston would rather rate it in terms of generations, like video tape. David Fibush said that the high end of the quality measurement should have no limit. He urged we throw out bit rate. Alan Godber was concerned that there be no ambiguity. Warner Johnston commented that you could use bit rates, but they would have to be circa a particular year, like 1997.

Break between 10:15 AM and 10:37 AM

When the subcommittee returned, Alan Godber moved the discussion to other issues, starting with in and out-of-service measurements.

David Fibush began by stating we must show it works using test scenes. The measurement system should be applicable for in service measurements, but correlation is more difficult. Arthur Webster added that we can have an out-of-service measurement that takes a long time to complete. He commented that in-service measurements are important and suggested we work in parallel on that direction as well. These could be coarser measurements that may not be as susceptible to breaking down with things like picture shifts and gain changes. As an example, he referred to the QUOVADIS system, which uses feature extraction and transmits the information with the picture. It is accurate enough for some purposes.

Steve Wolf agreed, saying the gain corrections and such can be done in-service. David Fibush added "if you have enough compute power." To specify equipment you need to use test scenes. For in-service measurements, you might consider a cheaper, less capable measurement. Steve Wolf noted that with in-service measurements you don't know the source. Warner Johnston mentioned soap operas as an example - the video peaks at 50 IRE units for long periods. He suggested a footnote should include a provision that the full gamut of video levels are used.

David Fibush said the data about the input video could be sent down a separate path. Warner Johnston was concerned that would take another circuit. Arthur Webster replied that DVB has already allocated a certain number of bits in their stream so these measurements can be made, under the QUOVADIS plan. David Fibush explained that DVB is a consortium of companies organized to define digital video transmission characteristics for satellite, terrestrial and cable. It is fully MPEG2 compatible. The Echostar direct to home satellite format is DVB.

Bill Meeker asked how in-service versus out-of-service measurements relate to the scope document. David Fibush said the requirement is the measurement method should not preclude in-service measurements. Warner Johnston wants to see a more positive statement, such as "in-service measurements have to be devised." Al Morton commented that it sounds like in-service measurements are highly desirable. He asked if they have to be available in real time? The discussion seems to imply non-real time availability.

David Fibush said that a requirement for in-service / real time measurements would throw out most methods now. As a result, we would end up with the simplest method, not the best. Alan Godber asked "What is real time?" Warner Johnston replied "How long can I hold my breath?" He elaborated that in his operation they get a call and have to determine the problem in about 45 seconds. Al Morton asked that with enough compute power, is this amount of non-real-time delay sufficient? Warner Johnston answered "Yes."

Arthur Webster noted that in-service and out-of-service measurements may not use the same method. We may need the out-of-service measurement first to evaluate the in-service measurements. David Fibush agreed, saying that if the input picture has to be available for a test, it may not be possible to use the same test for in-service work. Steve Wolf said that quality depends on the scene. An extreme position would be that all broadcasters care about is what is being transmitted.

David Fibush pointed out that in-service measurements are more important now because of stat-mux systems. Too many high complexity programs at the same time will reduce their quality. Today's in-service measurements with VITS don't always match the picture. Al Morton said that if we have both highly accurate out-of-service measurements and less accurate in-service measurements they will work well together. Steve Wolf stated the committee should view these as complimentary measurements. John Grigg added that the in-service measurements should be referenced to the out-of-service measurements.

Alan Godber asked for a discussion of the formats issue.

David Fibush started by saying we should do everything in 525I (525 line, interlaced). When we talk about high quality measurements, it should hold for different formats. Leon Stanger added that subjective viewing tests will be a problem if we have 25 different formats. Alan Godber noted that while these have to be taken into account, they don't have to be tested at this time. Bill Meeker suggested dividing the process into Phase A, Phase B, etc.

Chairman Godber asked if there was anything more on maintenance.

David Fibush replied there is a list, operational monitoring is what we are talking about.

Alan Godber then asked if there were any other issues.

Arthur Webster made a plea for subjectively rated test materials.

Alan Godber emphasized the importance of this document, saying it guides the committee's work. There was a short discussion of how to review it. It was agreed that it would be reviewed line by line.

Alan Godber replied it looked good to him. No one else offered any comments.

David Fibush suggested adding a fourth item - "To the extent possible this will be done in conjunction with the ITU Joint Rapporteurs Groups." Alan Godber commented that the IEEE has not agreed to a joint effort, but agreed that liaison is required. David Fibush, commenting on the third item -- "To correlate the results of quantitative picture measurements against those obtained by subjective viewing" -- said we won't do all the work. The four groups should each be doing a piece of this. Alan Godber questioned this. David Fibush replied that we will not do all this work ourselves unless we have to. He added that the ITU focus is broader because it includes a wider range of bit rates and it would be a shame if we did not take advantage of the work ITU does if it meets our standards. Alan Godber cautioned that we have to reserve the right to change methods if we turn things over to ITU. David Fibush answered that we are turning it over to ourselves. Arthur Webster suggested adding words in the liaison section mentioning coordination specifically with the ITU and the Joint Rapporteurs Group in particular. He hoped this group would participate. Steve Wolf said one goal could be to feed input to the ITU. Alan Godber agreed that we want to contribute, but we also want two way communication. IEEE may decide we want a different standard. David Fibush emphasized we should correlate measurements, noting that we can't get the same quality if we don't coordinate with ITU, which has much greater facilities available to them. Alan Godber recommended we suggest liaison with ANSI T1 in the scope.

Leon Stanger suggested emphasizing ANSI T1 and ITU in the liaison section. Alan Godber noted that we haven't gotten any data from ATSC and he didn't expect much from within IEEE. David Fibush said we need we need something stronger than liaison, suggesting "Close cooperation with the Joint Rapporteurs Group of ITU Study Group 12 and Study Group 9 and ANSI T1A1.5." Warner Johnston noted that the liaison should be ATIS-T1, not ANSI-TI. The other bodies mentioned are also under ANSI. Eric Hauch suggested using Committee T1. Warner Johnston said we need to include SCTE – the Society of Cable Telecommunications Engineers. They are ANSI accredited. He also mentioned the need to include the computer industry and suggested contact with Intel.

Leon Stanger said he'd drop ANSI and use Committee T1. Warner Johnston asked about the computer industry. Alan Godber said to add a marker and investigate it.

David Fibush recommended adding under goals "Close cooperation is planned with the Joint Rapporteurs Group ITU-T of Study Group 12 and 9 and ANSI T1A1.5" and "The subcommittee will monitor the effectiveness of this cooperation towards meeting our stated goals."

Leon Stanger said he'd add it as section 2A under liaison instead of as a fourth goal, leaving the goals to stand as presented.

There were no additional comments on the "Background Information" or "Liaison" sections.

"Television System Standards":

John Grigg noted that implicit in this definition is that frames will arrive in some order. He asked if we want to say 30 frames per second. David Fibush answered that it can be measured by other means and should be. Here we are talking about picture quality. Dan Baker commented that this falls under Purpose, which doesn't include jitter. Rick Redford suggested saying System-M. Leon Stanger replied that this also implies a transmission system, which is irrelevant. Rick Redford explained that to specify interlaced video and a frame rate we could use the ITU definition of System M. Leon Stanger said the recommendation is to focus on interlaced video. Alan Godber added "because that is what we can do." He noted that there are still people who don't know what standard definition is. Rick Redford agreed, saying we need to define it more. Alan Godber asked if there was anything wrong with saying frames? David Fibush recommended referencing ITU-RB.624.

Warner Johnston suggested adding the adjective "initial" to the word "efforts" in the first sentence of the Television System Standards section and adding the word "digital" to "component" in the next section. Alan Godber and Leon Stanger agreed to this.

Alan Godber questioned the words "To the extent possible" in the second sentence. David Fibush commented that now it isn't possible. Warner Johnston said adding "initial", as noted before, should take care of that. Leon Stanger suggested replacing "To the extent possible" with "when practical." David Fibush remarked that the original wording was better. Alan Godber suggested "As soon as practical". David Fibush replied that it won't happen soon.

"Interface Signal Formats":

Warner Johnston said he would like to see the phrase composite analog formats dropped. David Fibush commented the problem with that is there is some equipment where the only ports available are analog in or outs, such as set-top boxes. Warner Johnston said this may change within the time frame we are looking at, mentioned technologies such as fire-wire. Bill Meeker added that the interface brings up the problem of picture shift going between analog and digital. Leon Stanger replied that we can't exclude systems that have analog segments in them. The standard has to be broad enough to allow this. For measurement purposes, we have to get back to component digital. Bill Meeker noted that there is a difference between integer picture shifts and partial shifts.

David Fibush pointed out there are two issues we need to deal with. One is the hardware interface to the equipment. The second is whether or not the algorithm should take into account what happens when you go into and out of analog (composite) domains. Warner Johnston asked who was building analog equipment now? David Fibush answered maybe in El Paso, but emphasized we can't ignore it. People are building gear with composite analog interfaces. Leon Stanger added that delivery to home is still composite analog out. David Fibush commented that the measurement must be made in component. He questioned whether the section has the wrong title. We must know what will happen to the signal if it goes through a composite system.

Alan Godber asked if we need an analog measurement system. David Fibush and Leon Stanger said not as separate system, but it should be included.

Leon Stanger suggested deleting section 4 - Interface Signal Formats. Under section 5, state that "for measurement purposes the measurement will be done with a digital component signal." Warner Johnston suggested listing specification names for the standards used. Leon Stanger answered that we don't want to exclude systems. Rick Redford suggested adding CCIR-601. Warner Johnston then recommended excluding references to interfaces for the scope. Leon Stanger said we don't want to get into the composite / component argument. Warner Johnston observed the CCIR-601 has the problem it only covers standard definition.

Leon Stanger said he would mention CCIR-601 for SDTV.

There was no discussion on the sections on "Types of Compression Systems and Impairments" or "Intended Users".

"Subjective Results":

Alan Godber commented that we must have it. David Fibush suggested the wording "The committee will define the type of subjective tests to be used and the required degree and type of correlation."

Rick Redford asked what "transmission" referred to in section 2. David Fibush answered that this is what we were talking about. Alan Godber reminded the committee that "transmission" was changed to "distribution" in the title of G-2.1.4 to avoid confusion with RF. Al Morton suggested using the word "transport". There was general agreement on this.

Pat Tweedy commented that when compressed channels are multiplexed, in some cases strange colors appear in the video. Leon Stanger said this type of impairment would come into this. Warner Johnston suggested changing the sentence so you don't eliminate transmission. Leon Stanger suggested changing the first part of section 2 to "Analysis of the effects of transport and storage defects to the extent they relate to visible impairments will not be excluded, except…"

With regards to section 3, excluding audio measurement or quality assessment methods, David Fibush noted that part of the liaison with the ITU might include audio, but not as part of our group. Alan Godber explained this was referred back to the audio subcommittee for discussions with AES. He said they hadn't gotten any further with it.

James O'Neal asked if section 3 covered audio-video synchronization. Leon Stanger said that was listed as a red herring a few meetings ago. The committee decided to exclude audio. Alan Godber warned that we should always be clear. He suggested specifically excluding audio/video synchronization. Al Morton commented that the T1A1.5 standard covers this. He said this should be made available to Randy Hoffner (chairman of the IEEE subcommittee G-2.1.2 working with digital audio measurement standards), if in fact they are not already aware of it. While the T1 audio synchronization work concentrates on speech signals and telephone applications, the group should look at this.

Leon Stanger commented that nothing says we can't do this. Alan Godber, however, asked who has the expertise? Is this more audio than video? David Fibush said he don't know why measuring delay has to be in this group. Alan Godber agreed, saying delay is what we are talking about. David Fibush said compression is the only area where these audio delay problems can occur. Leon Stanger recommended putting a "TBD" (to be determined) next to audio if it is not being done by another committee.

Rick Redford suggested work on audio-video synchronization should be done in the distribution subcommittee (G-2.1.4). We could have multiple standards. David Fibush said that while we might develop a different way to measure it if we didn't have compression, it has to done through the compression system.

Arthur Webster referred to a document submitted from Australia that shows delay specs. Leon Stanger said impairment is not a TBD, but audio / video relative delay is. Al Morton emphasized the difference between relative and absolute delay. Dan Baker suggested adding lip sync parenthetically.

Leon Stanger asked for comments on section 4, excluding the effects of concatenation summation for multiple compression/decompression cycles.

James O'Neal questioned the wording "decompression/compression cycles". David Fibush suggested emphasizing the exclusion of summation – we can't take separate measurements and sum them. Warner Johnston commented that we shouldn't exclude it, just say we are not going to do it now. Rick Redford agreed that we should "stake our claim" on this type of measurement. Leon Stanger noted that summation results will be different for different systems. David Fibush said that is we consider all as black box systems there may be some way to do this. Leon Stanger suggested moving summation up, saying "The effects of summation of concatenation…"

Warner Johnston suggested adding that all measurements will be out-of-service. David Fibush replied that we should not exclude in-service testing. Leon Stanger suggested adding "An in-service measurement method is desired."

Dan Baker commented that the Purpose section of the Scope does not include all our goals. He suggested the wording be changed to:

"…to investigate and recommend methods of directly quantizing image-sequence impairments or differences from the original image-sequences, resulting from compression and decompression cycles, that are well correlated with subjective comparisons to the original image-sequence…"

Leon Stanger said he will make the changes and submit them back to the study group. David Fibush said they should go to the whole group.

David Fibush asked about the list of items we are not going to test and those we will look at long term. Alan Godber said this should be a separate document. Bill Meeker said that if the list was small, it could be folded into the scope. David Fibush commented that there will be a lot of discussion on this. It would be something we could contribute to the Joint Rapporteurs Group.

Alan Godber suggested sending the scope to the Joint Rapporteurs Group and asking them to consider it. David Fibush agrees. He said that the Joint Rapporteurs Group is doing a ten page work plan listing all things that have to be done. We will want to comment on about five of those pages. He asked if these will be brought into our scope?

Alan Godber said that if we can agree on a method to finalize it, we can get the scope to the Joint Rapporteurs Group quickly. The list, however, may take longer and might wait until the next meeting.

Leon Stanger said he felt the list should be a separate document. Alan Godber agreed, saying it probably goes later. David Fibush also agreed. Alan Godber commented that we don't want to slow down the scope work while waiting for the list. He suggested adding words to the scope that this work will be done.

Alan Godber asked if there is any reason others shouldn't see the scope. Leon Stanger recommended working it up quickly, then getting agreement from those present here before posting it to all.

David Fibush suggested starting the discussion on the e-mail reflector and if no one comments, whatever he suggests goes. He volunteered to coordinate the discussion on the list of things we want to consider in this methodology and the things present in other methods we might consider later.

Leon Stanger was concerned that this might describe a piece of equipment. David Fibush answered that was not the case, the equipment Tektronix was working on includes items which are excluded.

Alan Godber stated Bill Meeker should chair the group, so there is no appearance of it being skewed to one manufacturer. He was concerned about the appearance of a conflict of interest. David Fibush noted that SMPTE isn't concerned about this.

David Fibush explained that he did not say he was heading the group, but was willing to post the record on the reflector. After discussion, it was agreed that Bill Meeker will head the group, David Fibush will start the discussion and Al Morton and Leon Stanger will participate.

Alan Godber wanted to be sure there are contributors to the group. David Fibush asked what the committee is allowed to sent to the ITU Joint Rapporteurs Group. Alan Godber answered "Whatever the committee decides." He added that we need to work out a mechanism for due process.

David Fibush stated we need to set time limits and that if people don't reply, we go ahead without them. Alan Godber asked for a time frame. David Fibush suggested well before the next meeting, say by the end of June.

Warner Johnston commented about e-mail reflector virus problems. Doug Lung said he would have to work through IEEE on this, since they maintain control over the e-mail reflector.

Alan Godber reported that Bill Zou's list is incomplete, but promised an update for the next meeting. David Fibush wondered if we are beyond a fundamental reading list, considering all we've discussed. He suggested we ask what would be appropriate based on what we have discussed at this meeting.

There was no further discussion.

David Fibush reported he and Bill Zou have been talking to ATTC. He found that getting the material and seeing it isn't a problem. The problem is no one knows if we can use it. The idea was that we would down-convert the HDTV material, but we will have to find someone to do this.

David Fibush observed that a starting point would be to look at the CCIR test scenes. In the ITU document it says they are in the public domain, but it adds "for experimental purposes." He proposed starting with the ITU test scenes – 525 and 625 line versions are available, then determining what we must find somewhere else. We need to be careful in selecting materials. If compression killer sequences kill everything, they aren't interesting. He asked if anyone has materials that might be useful for this? As far as actual test materials, the ITU said CBC had the test materials. He stated that we presently have and are duplicating these tapes.

There was a discussion on how to know if you've made a perfect copy. Michel Poulin observed that no one knows where the originals are. Leon Stanger responded it doesn't matter, provided they are perfect. David Fibush said Tektronix will look through the sequences to check for errors and added that perhaps SMPTE will take over making copies of them. That way they would be available to everyone. He said that the CCIR material is close to public domain, but they may not be sufficient.

Alan Godber asked David Fibush if he had gotten confirmation that ATTC owned the materials. The answer was no – no one would commit to anything. Alan Godber said he will follow up on this, adding that ATSC has a responsibility to help us. The ATSC document defining implementation standards stated IEEE would do the measurement standards and definitions. David Fibush said he will continue to pursue CCIR and see if SMPTE will take it over. We still need adequate materials.

Alan Godber commented this might take a separate meeting. Al Morton said the most detail he had on the next T1A1 meeting in Kansas City was that it is in Kansas City. David Fibush remarked that if someone could supply a D-1 recorder and serial digital monitor we could go through the sequences. He will bring the tapes. Alan Godber cautioned that we will need a fair amount of time to do this, noting it took four sessions of four to five hours each for the ATTC testing.

Michel Poulin observed that we need to look for the attributes of each test sequence and added it would be good to have this ITU document. It would be good to have a list of attributes, even for new tapes. David Fibush said the tapes are nominally available and we can view them when we want. Alan Godber asked if we needed to arrange a separate meeting. David Fibush replied we need to have the equipment available. Alan Godber said he will try to make it happen.

This item was folded into the earlier discussion under Item 8.

Alan Godber asked if there was any additional work to be done. David Fibush answered that with the e-mail reflector and pressure from the ITU we may start coming up with a list of topics for the next meeting.

There was no other business.

Item 12 - Date(s) of Future Meeting(s)

This was discussed in Item 6. The next meeting will be in Kansas City, August 4^th.

The committee thanked NTIA for providing space for the meeting. A motion to adjourn was presented, seconded and approved at 12:51 PM.

Submitted by:
H. Douglas Lung
Secretary

APPENDIX "A"

Draft Agenda - IEEE Compression and Processing Subcommittee G-2.1.6 Fourth Meeting Monday May 5th, 1997 , Alan Godber, Chairman, 5 May1997.

Draft Meeting Record, G-2.1.6, Compression and Processing Subcommittee, Meeting #3, January 20, 1997, Sheraton Orlando North Hotel, Maitland, Florida, Doc. G-2.1.6/25, 20 January 1997.

Report of the ITU-R and ITU-T Meetings Held April 8 - 17 Relating to Picture Quality Assessment, David Fibush, Tektronix, April 23, 1997, Doc. G-2.1.6/26, 28 April 1997.

Overview of Picture Quality Measurement Methods, David Fibush, Tektronix, May 6, 1997, Doc. G-2.1.6/27, 28 April 1997. [Covers methods of subjective and objective testing including system considerations.]

Methods for Evaluation of Digital Television Picture Quality, William Y. Zou, Public Broadcasting Service, and Philip J. Corriveau, Communications Research Centre, Doc. G-2.1.6/28, 29 April 1997.

Transmission of Time Differences Between the Sound and Vision Components of a Television Signal, (ITU-T SG9 D.010), Geoffrey Smith, Scientific-Atlanta, Australia, April 1997, Doc G-2.1.6/29, 4 May 1997.

Progress Report on Development of Digital Compressed Picture Quality Assessment System in Japan, (ITU-T SG9 D.015), Dr. S. Matsumoto, KDD, Japan, Doc. G-2.1.6/30, 4 May 1997.

Comments on COM 12-7-E, Objective Measures of Video Quality, (ITU-T SG12 D.020),
A. Webster, NTIA, March 1997. Doc. G-2.1.6/31, 4 May 1997. [NTIA response to the KPN put-down of the ANSI toolbox.]

QUOVADIS first results, (ITU-T SG9 D.022), E.B.U., April 1997, Doc. G-2.1.6/32, 4 May 1997.

Objective Measurement of Video Quality, (ITU-T COM 12-7-E), J. Beerends, KPN Research, The Netherlands, February 1997, Doc. G-2.1.6/33, 4 May 1997. [A modification of some ANSI tools, emphasis on matrix methods.]

Why is Objective Evaluation Needed for Compressed Digital Video Systems, C. Dalton, ITA, UK, Made-to-Measure Seminar, Montreux, Nov 1996, Doc. G-2.1.6/34, 4 May 1997. [Shows that concatenation of many compression systems will be used.]

Video Codec Evaluation System Based on Characteristics of Human Visual Perception, T. Hamada, KDD Research, Japan, Konnichiwa, Spring 1997, Doc. G-2.1.6/35, 4 May 1997. [Measures weighted SNR for various objects in the picture.]

Objective Measurement of Picture Quality for Digital Television Broadcasting, I. Yuyama, NHK, Made-to-Measure Seminar, Montreux, November 1996, Doc. G-2.1.6/36, 4 May 1997. [NHK work on criticality.]

Video Compression: The Need for Testing, G. Beakley, C. Cressy, and J. Van Pelt, Stellacom, SMPTE Journal, November 1995, Doc. G-2.1.6/37, 4 May 1997. [An early, very good, survey article that includes the idea of IQR.]

Correlation Between Objective and Subjective Measurements for Video Compressed Systems, M. Ardito and M. Visca, RAI, SMPTE Journal, December 1996, Doc. G-2.1.6/38, 4 May 1997. [Early criticality work by RAI]

Measurement Issues Relating to Program Types, Bit Rate and Quality, M. Ardito, L. Boch, A. Messina, M. Stroppiana, and M. Visca, RAI, Made-to-Measure Seminar, Montreux, November 1996, Doc. G-2.1.6/39, 4 May 1997. [Later criticality work by RAI including quant_scale values.]

Report on Studies for the Definition of the Criticality of the Test Materials Used for Subjective Evaluation of Digitally Coded Pictures, (ITU-R 11E/43), 8 April 1997, Doc. G-2.1.6/40, 4 May 1997. [A WP11E rapporteurs report, includes a list of some objective methods.]

TIA1.5/97-108, Committee T1 Performance Standards Contribution: ITU-T Activity on Video Quality Assessment, Arthur Webster, May 5,1997, Doc.G-2.1.6/41, 4 May1997.

An Objective Video Quality Assessment System Based on Human Perception, (ANSI T1A1.5/93-32) A. Webster, C. Jones, M. Pinson, S. Voran, and S. Wolf, NTIA, 1993, Doc. G-2.1.6/42, 4 May 1997. [Technical details of the m1, m2, m3 used in IQR.]

Report and Summary of First Meeting, May 4^th, 1997, Doug Lung, IEEE Compression and Processing Subcommittee G-2.1.6 Task Force on Compression Measurement Methodologies, Doc. G-2.1.6/43, 5 May 1997.

DRAFT 5, Detailed Scope of Activities, Leon Stanger, IEEE Compression and Processing Subcommittee G-2.1.6 Task Force on Preparation of Scope for Subcommittee Work, 28 January 1997, Doc. G-2.1.6/44, 5 May 1997.