In a message dated 2/15/2005 2:36:45 PM Eastern Standard Time,
sklein@CPCUG.ORG writes:
>
> Herb -
>
> Here is a possible approach
to think about:
>
> 1. The number of cast vote records
unrecoverably lost or altered as a
> result of machine malfunction
shall be less than 1 in N ballots cast,
> where N is something like a
million.
>
> (Note this recognizes that inaccuracy is not the
same as malfunction,
> which is defined as in the current spec.
....)
>
I'm not sure what the note in parentheses really means; however,
here are my thoughts on the matter:
1. Loss or alteration of votes in any
non-random manner is a MUCH more serious matter than random
loss or alteration of votes.
2. Loss of 300 votes in one machine is much more serious than loss
of one vote in each of 300 machines.
3. ANY detection of errors in "normal operation" (unrelated to
component failures) should be clear grounds for rejection of a system.
(But is drift in touch-screen adjustment considered a component failure?
There seems to be a tolerance for that type of error.)
4. In my experience, there is a significant amount of arbitrariness
or subjectivity in estimating the probability of very unlikely events.
Does anyone really believe that a practical procedure exists for
distinguishing between an expected loss of one vote in 500,000 and one vote in
2,000,000? What level of confidence is required?
5. Calculating an expected number of lost or altered votes "as
a result of machine malfunction" requires estimating the probability of each
of many types of malfunctions as a function of usage (what about aging
effects?) along with the expected number of votes lost or altered
for each type of malfunction. Does this seem practical? I know
that NASA did (and maybe still does) this kind of analysis for complex
space systems, but it's very expensive -- and yields only a general idea of
the probability of any failure. As far as I know, NONE of the dramatic
failures of space equipment was identified as high-risk by such studies.
6. A-priori calculations of MTBF are, in my opinion, very
unreliable. Statistical evaluations of MTBF can be very accurate,
if enough failures actually are observed. If large MTBF values are
required, that would imply very long (and expensive) tests.
Most of the above is negative, yet some type of practical
reliability/accuracy standard is required. Let me suggest something
along the following lines:
1. Each of (12?) significantly different ballots is programmed for
the voting system. For each ballot, an independent automatic system
simulates the actions of (1000?) voters completing their ballots in (10?)
different ways, in random sequence. Automatic testing is done on each
ballot (8?) times, simulating a total of (96,000?) voters.
2. If all the vote totals from the (96?) voting system runs match
what the independent automatic testing system generated, the voting system
passes this accuracy/reliability criterion.
3. If there is any type of failure during the above testing,
then:
a. if the failure caused no alteration of
recorded votes, and no loss of votes other than the last
one cast before the failure, and
b. if a repaired or replaced system finishes
(2?) more complete tests (steps 1 & 2) successfully, it passes;
c. otherwise, the system must be modified to
reduce failures before retesting from the start.
I'm strongly tempted to DOUBLE the testing requirements each time a
system is forced into modification via step 3c.
The above obviously is much less stringent than the one-in-a-million
requirement originally proposed, but I argue that "proving" one-in-a-million
is either impractically expensive or largely subjective. This
suggested criterion attempts to address software errors, hardware failures and
any problem that affects many votes at once.
I don't know how to enforce it, or who would collect the
data, but rare problems should be documented wherever and whenever the
systems are used. There should be a clear mechanism to de-certify a
system.