| Thread Links | Date Links | ||||
|---|---|---|---|---|---|
| Thread Prev | Thread Next | Thread Index | Date Prev | Date Next | Date Index |
|
Subject: [802-20-GENERAL:] Comments on IEEE 802.20 Technology Selection Process Document
Dan, In reviewing the latest version of your Technology Selecton contribution (C802.20-04-72r3) I have a few questions and comments. 1. Is there a relationship between the selection rounds described in 72r3 and the phases described in the evaluation criteria document (chapter 6). For example, do you anticipate that the first down select round after phase 1 or after phase 2 evaluations? [Dan]: My understanding is that phase 1 and phase 2 of the evaluation should be completed in Stage-3 of the TSP. The down-selection rounds start in Stage-4. 2. It's not clear what proposal compliance to the evaluation criteria document means. [Dan]: Compliance with the Evaluation Criteria document is to be understood as meeting all the simulation, simulation results and supplemental data disclosure requirements outlined in the appropriate sections of the (EC) document. 3. The material in "Table 1: Technical Specifications Summary and SRD Compliance" is completely contained in "Table 2: Compliance with the 802.20 Requirements and Evaluation Criteria documents" . Do we really need two tables? Is it correct to interpret Table 2 as stating compliance to the SRD requirement and that the ECD column simply shows what are sections of the evaluation criteria document are associated with that SRD requirement? Everything to the left of the double line should be provided in the Annex of the Technology Selection document and those submitting proposals need add only their compliance and notes. [Dan]: Table 2 does not contain the actual specifications of a given TP. It merely provides a detailed account of a TP's compliance statement. We need both tables. 4a. Section 2.3.5 states, "An ad-hoc evaluation team, appointed by the 802.20 working group, shall review all the evaluation technical reports and prepare a comparison report that should rank the performance of the individual proposals in several key categories (TBD)." I don't believe this is a valuable exercise and I have concerns about an subset of the working group performing an official ranking of the different proposals and how that will be interpreted by the rest of the working. Of course, working group members are always free and likely will provide their own commentary on various proposals and so formalizing an ad hoc group seems unneeded. I recommend that this section be removed from the technology selection process. [Dan]: I tend to agree that we need not require an Ad-hoc evaluation team a priori. We may want, though, to define some evaluation results reporting format requirements in the Evaluation Criteria document to facilitate easier TP comparison. 4b. Section 2.3.6 shows "shoulds" as a requirement. "Shoulds" are recommendations and are not requirements for the purposes of evaluating proposals. [Dan]: I believe I heard other opinions. My view is that while the "should" requirements are important for the evaluation of proposals, they should not be considered for establishing "full compliance". 5. In section 2.3.9 it states that "(a) Down-selection rounds shall require a working group quorum and may take place in either interim or plenary sessions. The final selection round shall be conducted in a plenary session." Don't the IEEE 802 P&P address this issue adequately? Down selection should be considered technical votes. A quorum is required for votes taken at an any Interim meeting and a quorum is assumed to exist [in] Plenary meetings. The working group can always vote to waive the quorum requirement at interim meetings or can reaffirm at the next Plenary meeting a[n] vote taken at an Interim meeting. So, I don't think this clause is needed and it unnecessarily limits the working groups operational flexibility. [Dan]: I would propose to make the following change: "A quorum is required for all votes taken in the course of the Technology Selection Process". 6. In terms of actual voting mechanics for down selection, there are several approaches that are credible. Here are a couple of options, assuming that there are N proposals being considered: a) Each WG member votes for 1 proposal. Any proposal receiving > 75% support is selected (game over!). If none receives > 75%, then, it doesn't seem practical to set a specific bar for getting to the next round. For instance, if there were 4 proposals it is highly possible that none would receive >50% and that the amount of support that each receives could be relatively close to each other (e.g. 20%, 22%, 28% and 30%). There are at least two ways forward ii) the top m proposals totaling > 75% support makes it to the next round. This approach drops the proposal that received 20% support iii) any proposal receiving > 25% support makes it to the next round. This approach drops the two proposals that received 20% and 22%. [Dan]: What if there are three TPs and the results of the votes are: 36%, 30% and 34%? - the down-selection process is stuck! Thus, it is better to devise a flexible elimination-criteria. In addition, I feel that the suggested method would divide the WG into TP camps from the get go and would not allow non-aligned members to favor (and vote for) two TP that seem equally good to them. b) Each WG member assigns proposals a ranking of 1 to N, 1 being the best and N being the worst. The N-1 proposals with the fewest ranking points progress to the next round. (Of course, this works just as well with N being the best, 1 being the worst and forwarding to the next round the proposals the N-1 proposals with the most ranking points.) [Dan]: This method would be similar to an elimination process whereby each member votes (Yes/No) for each proposal and the proposal that gets the least support is eliminated. To complete the suggested process, I would add the following:
Those are a few thoughts that come to mind immediately. I may send more comments as I further study the document. Best regards, Joanne
|