[Fwd: RE: MTBF for voting machines] part 1
As discussed on today's teleconference, I'm forwarding a discussion
between Herb Deutsch and myself regarding reliability.
This email contains most of the thread. There is another that I am also
forwarding.
Stan Klein
-----Forwarded Message-----
> From: "Deutsch, Herb" <hdeutsch@essvote.com>
> To: 'Stanley A. Klein' <sklein@cpcug.org>
> Subject: RE: MTBF for voting machines
> Date: 03 Feb 2005 11:33:05 -0600
>
> Stan,
>
> You make a lot of good points. Especially the stop working. There could be
> an intermittant problem as you described. There is a lot to think about. I
> agree that the pure DRE may need different considerations from all other
> systems (DRE with paper trail and optical scan). This distinction is
> already in the current draft regarding COTS and source analysis.
>
> Herb
>
> -----Original Message-----
> From: Stanley A. Klein [mailto:sklein@cpcug.org]
> Sent: Thursday, February 03, 2005 11:33 AM
> To: Deutsch, Herb
> Subject: RE: MTBF for voting machines
>
>
> Herb -
>
> I've read the appendix (I downloaded a copy from the FEC site just after
> it was released). I just don't see in there what you are seeing about
> the derivation. I do know the number flowed down, because I have a
> paper copy of the 1990 standard.
>
> It seems to me that you are mentally defining a failure differently from
> what the specs from 2002 forward have defined (and I agree with their
> definition). A failure is any loss of one or more functions or any
> degradation in a function lasting more than 10 seconds. This does not
> translate to MTBF meaning that something works and then stops. It can
> work, stop (for more than 10 seconde), then get cleared and work again.
> (Something like a voltage or thermal problem could cause that kind of
> behavior.) Or it can work during part of setup, stop (just one or a few
> functions) for the remainder of setup, then work again when taken to the
> polling place and restarted. That's how I think the one-and-a-fraction
> percent of improper ballots or candidates missing from ballots I
> calculated from the problems reported by TrueVote MD may have happened.
> I don't see it as a data issue. The rate of occurrence is too
> consistent with being a significant fraction of the allowable failure
> rate during setup, once you adjust the operating time in the scenario.
>
> I'm not sure that spec-ing MTTR makes sense for election day failures
> that require opening the machine. Besides, MTTR really includes
> dispatch time, travel time, diagnostic time, repair parts logistics
> time, repair parts installation time, and repair verification time.
> With adjustments, you only replace the logistics and installation times
> with the duration of the adjustment procedure, assuming the technician
> has immediately available all needed tools for the adjustment (and
> doesn't have to order some tool sent in, involving a logistics time for
> that).
>
> On the relationship between accuracy and reliability - What you say
> makes very good sense for optical scan. The unit works but there are
> tolerances, and when they occasionally add up wrong, you get a
> mistabulated ballot.
>
> However, what you say doesn't make sense for DREs. A mis-recorded
> ballot on a DRE will either be due to a software bug, a hardware
> failure, or malicious tampering. And because of the close interaction
> between hardware and software, software bugs and hardware failures are
> often indistinguishable from each other.
>
> BTW, in thinking further about your ballot jam example, I assume that is
> an optical scan situation. I don't think I would be as worried about a
> technician replacing parts on a precinct-based optical scan device used
> primarily for overvote checking as I would about a technician opening up
> a DRE. There are things that can be done to ensure integrity of the
> optical scan (use a central count optical scan for the actual ballot
> tabulation, rerun all previous ballots at the polling place, etc.).
> However, with the DRE there is no backup means of ensuring integrity.
>
> We may wind up effectively needing different specs for optical scan, DRE
> without AVVPAT, DRE with AVVPAT, EBM, EBP, and EAMBM. I think the point
> of commonality is really accuracy, and I suspect that to develop
> comparable specs we need some assumption about how many voters use a DRE
> during an election. For example, the worst case is it was never storing
> the votes and you lose all of them (only the storage function was lost
> and you don't find out until poll closing). How does that look in terms
> of accuracy? (I'm also not sure what that does to your statement that
> early voting is likely to work out OK.)
>
> We probably have to make some different assumptions (about election day
> and early voting) and run the math to get to a spec.
>
>
> Stan
>
>
> On Wed, 2005-02-02 at 23:08, Deutsch, Herb wrote:
> > Stan,
> >
> > The number did flow from the FEC 2002 which flowed I believe from the
> > 1990 standards. Here's the appendix that details its derivation.
> > Check out C4. 6.2.2.1 was extract from this appendix.
> >
> > I disagree with pre-election setup being a factor in MTBF. MTBF
> > implies it was working and then stopped. With a setup problem, it
> > never worked properly, and an improper ballot is a data issue and not
> > a hardware issue.
> >
> > There should be a spec on MTTR (Mean Time to Repair). I believe we
> > did have to supply this info as part of Certification testing.
> >
> > The MTBA concept had the touch screen alignment in mind. The
> > adjustment would have to be done without taking the unit apart or
> > replacing parts. A ballot jam that a technician had to correct could
> > be another example.
> >
> > Accuracy is a separate requirement from MTBF. It assumes the unit is
> > operating but imperfectly. The imperfection may be a consequence of
> > unit design and not failure.
> >
> > Herb
> >
> >
> >
> >
> > -----Original Message-----
> > From: Stanley A. Klein [mailto:sklein@cpcug.org]
> > Sent: Wednesday, February 02, 2005 9:12 PM
> > To: Deutsch, Herb
> > Subject: Re: MTBF for voting machines
> >
> >
> > Herb -
> >
> > It seems to me that what you are getting to is the idea of looking at
> > failure modes and consequences, and I agree that is the right area to
> > be looking at.
> >
> > I don't think we can just "come up with a number," although my earlier
> > comments tried to do that. My more recent comments began to tie the
> > issue to things like accuracy and failing-safe.
> >
> > I agree that "adjustment" failures may be correctable on election day
> > but others require the unit to go out of service. Of course that
> > depends on a proper definition of "adjustment" -- I think it could be
> > something that can be fixed without opening the machine, although it
> > would need to be something that could be witnessed and understood by
> > polling place officials and observers. However, even "adjustment"
> > failures can have consequences like apparent vote switching (i.e., due
> > to touchscreen misalignment).
> >
> > I think recoverabilty is important, and probably needs to be factored
> > in somehow. A higher rate could be acceptable for failures that can
> > be detected and recovered than for failures that either nobody knows
> > about and change or fail to record votes, or that once detected are
> > impossible to recover from except, perhaps, by rerunning the election.
> >
> > Another factor is pre-election failures, some of which cause
> > improperly installed ballots (e.g., candidates or contests missing).
> > The TrueVote MD report had some of those last November. BTW, the
> > scenario in the MTBF spec seems to use the worst case setup time for
> > L&A testing and does not show the more typical setup time, that I
> > estimate to be about 4 hours operation.
> >
> > BTW, I had looked at the appendix and didn't get the clear
> > understanding you seem to have of where the number came from. At the
> > beginning it talks about concern for testing costs. A lot of the rest
> > looks to me like straightforward statistics tutorial. I don't know
> > where it flowed in from, and I doubt it was written originally for the
> > P1583 draft, but the draft (in 6.2 ?) says something about doing the
> > MTBF test concurrently with the other testing. I think that is the
> > origin. They backed into the number on that basis. I think the
> > number itself got there in 2002 and the initial P1583 because people
> > just copied the provision and nobody stopped to do the math. I also
> > seem to recall another comment on that spec among the thousand for
> > version 5.0 by somebody else who said something like "That number
> > looks wrong and needs to be checked".
> >
> > Bottom line, I think we could have different allowable failure rates
> > for different circumstances, but that they should include
> > considerations of accuracy and recoverability, and should take account
> > of pre-election setup failures.
> >
> >
> > Stan Klein
> >
> >
> >
> > On Tue, 2005-02-01 at 22:02, Deutsch, Herb wrote:
> > > Stan,
> > >
> > > I totally agree with you that an MTBF of 163 hrs is not acceptable.
> > > This is a carry forward from the original FEC standard which may be
> > > due to the manner of testing that was anticipated. (Volume II
> > Appendix
> > > C of the 2002 VSS describes where the number came from).
> > >
> > > To develop an MTBF, we need to identify the % of units that failure
> > > will be tolerated during election day. I recognize that some units
> > > will be used for early voting and thus be used over a much longer
> > > period of time but I think that will fall out OK.
> > >
> > > I also think we should have 2 tiers of failure; one where an
> > > adjustment can restore operation and the other were equipment will
> > > need to be replace (repairs on election day should not be permitted
> > > for service restoration). The former would be more stringent than
> > the
> > > latter.
> > >
> > > (This obviously is a subject that could be up for discussion.)
> > >
> > > If you would consider a jurisdiction with 1000 units, what number of
> > > units of failure can be tolerated? I would propose 5 (.5%). If we
> > > would consider a 15 hour day. This would indicated a total hours of
> > > 15,000 and an MTBF of 3000. (Note that the 163 hour would produce
> > 92
> > > failures (9.2%)
> > >
> > > For the adjustment spec, maybe we use 1% or 10 in this case. That
> > > would be a 1500 hour MTBA.
> > >
> > >
> > > What do you think?
> > >
> > >
> > > Herb
> > --
> >
> >
> --
> Stanley A. Klein <sklein@cpcug.org>
--