Evaluating Election Systems
Victoria Lovegren, Ph.D.[1]
Draft
Evaluation should be based upon multiple criteria.
Each criterion should be given some sort of weight or priority, indicating its relative importance in the decision-making process.
Some criteria will be more difficult to measure than others—some are very subjective; some will require significant research; some will require the assistance of “experts” to help identify and measure them.
Below are criteria that have been used by other counties in their decision-making process. Note that these are not mutually exclusive—there is some overlap among various criteria. I have ranked them based upon the research that I have been conducting over the past 1.5 years, and my subjective association of their relative importance in protecting democracy.
And all of these must be considered relative to regulatory constraints, like HAVA, Ohio Law (e.g House Bill 262, House Bill 3, Reform Ohio Now constitutional amendments), Blackwell directives, FEC certification requirements, etc. These are not subject to debate—they are legal requirements—and are discussed in the Hard Requirements / Certification Issues section below.
The decision of election equipment should consider the public’s confidence in the county boards of elections and the secretary of state in carrying out fair and honest elections. Voter confidence is extremely low after the last two Presidential elections.
Requirements (standards) are not well defined, nor are they finished, or even mandatory. They are, in no way, adequate to define the standards to be met by voting equipment. They are dynamic. Examples are, HAVA “standards”, other legislation, certification requirements, public’s demand for transparency.
Election officials, board members and the public are largely ignorant of the risks of electronic voting machines.
Decisions should be made based upon the most current and reliable information – e.g. research and case studies from counties who have been using/rigorously testing e-voting machines.
Where does accountability lie for security breaches? Invalid elections? Costly recounts? Public distrust of the system? Use of un-certified equipment? -- The county? Blackwell?
|
Criterion |
Examples |
|
Public oversees all counting of votes; counts are posted at precinct level; system is not vulnerable to tampering |
|
Confidence that they will be able to cast their vote in private. Confidence that the “election system” (procedures, operation, technology, etc.) will capture and count their vote accurately. |
|
Robust system of checks and balances; random sampling in real time; accurate and timely recounts; ballots can be securely and easily stored, retrieved and recounted. |
|
Minimal number of breakdowns; minimal down-time; minimal number of delays for voters; efficient throughput of voters; no bottlenecks |
|
Voter intent is reflected accurately; overvotes and undervotes (residuals) are minimized; vote cast is the vote counted. |
|
Start-up costs to purchase e-voting systems that meet the above criteria; operating costs to maintain purchased equipment; |
|
Viability of vendor as a business; reputation for product quality and service; ethics demonstrated by vendor; quality of product |
|
Voter’s ability to vote using e-voting system; acceptable to all—elderly, ESL, handicapped (blind, disabled, quadriplegic, etc.). Minimal training requirements for poll workers and election officials. Privacy of vote. |
Considerations relative to each of the listed criteria
Transparency/ Security
Electronic voting machines are NOT transparent. The public cannot witness the counting of the votes. The counting of the votes is done “inside a black box”, and the public must trust that the counting is done accurately. Advocates of electronic voting machines claim that the systems have been tested and tested and tested by independent authorities and, therefore, the public should trust these black boxes.
Despite the “testing” that has been done on SOME machines, electronic voting machines, in general, can never achieve the level of security that is necessary in our elections. Ask the experts! The ACM (Association for Computing Machinery) website conducted a survey of software engineers and computer experts, 95% of whom believe that electronic voting machines are vulnerable to failure/breaches in security and must be accompanied by robust and transparent audit procedures. See www.acm.org/announcements/acm_evoting_recommendation.9-27-2004.html for their statement.
On Sept 16th, 2005, Dr. David Dill, Stanford Computer Science Professor, leading expert on election security, and opponent of touch-screen voting systems, stated at a lecture at the University of Utah Engineering School, that election officials who support DRE voting machines do so for one of 4 reasons: 1) they do not understand computer technology, 2) they have relationships with vendors who have wined and dined them for 20 years, telling them the technology works, 3) they associate with other election officials who tell each other that the technology is okay, and 4) they are constrained by bad legislation that politicians won’t admit is bad. Professor Dill went on to say that when he speaks to crowds of computer scientists, everyone in the crowd accepts what he says without question. But when he speaks to groups of election officials, they say “How can you say that these computers aren’t okay?”
Dr. Dill goes on to say that we cannot verify that the desired software is running on our voting machines, even with stringent software design/review. And voting systems, because of the fact that we discard vital information, are much more difficult to secure than banking systems, and, in banking systems, fraud still occurs.
In Leon County, Florida, Diebold memory cards used in the Optiscan machines were found to contain executable code, and were easily hacked by several election activists. These hacks were accomplished without the need for a system password. Notably this type of hack was not detected (or reported) by independent testing groups. See www.blackboxvoting.org/BBVreport.pdf. The author of the report, Harri Hursti told me that, despite the fact that the hack occurring on Optical Scan machines, the DRE machines are even more vulnerable to such tampering because they are more software-driven (vs. Optical Scans which are more hardware-driven.)
Homeland security website lists Diebold GEMS moderate security risk BEFORE the 2004 election. See http://www.us-cert.gov/cas/bulletins/SB04-252.html#diebold.
Is the board concerned with security or is it not?
Voter Confidence / Acceptance by public
Voter confidence after the 2004 presidential election is extremely low, even lower than it was after the 2000 election debacle. Some 35% of the American public now believes the 2004 election may have been tampered with. Those who have any technical knowledge of the way electronic voting machines work are worried about the security of their vote. Polls indicate that the most informed people, computer scientists, do not trust these machines without an audit trail.
The more informed the public, the more concerned they are about electronic voting machines. Though many may feel that the machines are slick and fun to use, if they were aware of their security vulnerabilities and taxpayer money necessary to implement these “fortresses”, they would not accept these new “ATM-like” devises. Vendors and election officials who claim that “voter surveys” and “exit polls” demonstrate acceptability by the public are counting on the ignorance of the public, just as our president relied upon citizen ignorance to build his case for an unnecessary war. Unfortunately our mainstream media has forsaken its responsibility to inform the public of the truth, resulting in our being manipulated like lemmings to sacrifice our sons and daughters and, now, our democracy.
Auditability / Recountability
Computer scientists who feel that audits are necessary for electronic voting machines, would not consider “Ohio Recounts” sufficient mechanisms for robust audits--i.e. on the level of financial audits.
“Ohio Recounts” are a sham, as thousands of volunteers in
the “Ohio Recount” will attest. In “Ohio Recounts”, we only hand-count approximately 3% of the
ballots (in a “sample” of 3% of the precincts). The “precincts” making up the non-random
“sample” were chosen by the county boards of elections (with the exception of
one county, I believe). Recounts were
not conducted in a timely manner—delays by Blackwell ensured election employees
would do anything to streamline the recount process, including making sure no
“surprises” would occur.
Future recounts will be more difficult to initiate in the future, as the cost to conduct the recount will be $50/precinct (vs. $10/precinct in 2004). Future recounts in counties with touch-screen voting devices will be significantly more difficult from an operational standpoint—the VVPAT (voter-verified paper audit trail), on flimsy cash-register-like paper will have to be unrolled, cut into pieces, sorted into “valid” and “rejected” votes, examined, stacked into piles by candidate, stored and likely re-handled multiple times. Furthermore, if there is a discrepancy, the paper record will not be binding. It can only be an input into a laborious and costly lawsuit whose result, in the best-case scenario, will be to raise questions about the election’s legitimacy. The net result: the non-transparent “black box” counts will determine the election winner. The public will have to trust that a black box created by extremely partisan vendors will have counted their vote properly.
Vendors and election officials have tried to fool the public, saying the voter can “verify” that his/her vote is accurately captured and counted because he can see his vote reflected on a small piece of paper beneath a plastic window. This paper report, in no way guarantees that the voter’s vote is counted as he/she intended. Those who say the paper report proves the voter’s vote is counted accurately are engaged in “false advertising” at best. They are attempting to give voters a false sense of security.
Other types of audits, e.g. surprise random recounts, will be more difficult to carry out with the fragile paper rolls. These types of recounts are discussed in Ohio House Bill 3.
Reliability
How reliable are the machines on election
day? How many machines will be
needed to prevent bottlenecks and long lines?
Long lines translate into denial of access to the polls. Selective distribution of voting machines
disenfranchises certain populations more than others, as evidenced in
Election systems are unique in the fact that they are used infrequently, and when they are used, they are “mission critical.” The training needed to troubleshoot and repair systems that malfunction on election day must accommodate fact that people forget things when they don’t use the information often, and there must be many trained people “on call”, just in case. Are these “rovers” non-partisan individuals or employees of e-voting corporations with highly partisan allegiances? www.ohiovigilance.org/Cuyahoga/OlsonReport.pdf.
Secretary Blackwell states that 1:175 is the correct ratio. Director of Cuyahoga County BOE states that the county will purchase additional machines to result in 1:150 ratio.
These machines have already demonstrated numerous failures and and security breaches (see www.votersunite.org/MB2.pdf and www.blackboxvoting.org.)
Any new technology is particularly vulnerable to failure, and “Best Practices” suggest implementation strategies that introduce new technology in parallel with old technology systems, in piloted usage, or in phases. Our election officials would rather “plunge” into the adoption of new technology which places democracy at risk!
Voter-verified paper audit trails were intended to provide
the “auditability” that is needed to complement
“black-box voting,” but, unfortunately, the required mechanical technology
(printers) adds a whole new level of complexity to the equipment, making it
more likely to break down than those without the VVPAT technology. See article entitled High-Tech Voting Accessory: Paper at http://www.verifiedvoting.org/article.php?id=5814.
Measures that must mitigate the risk of breakdown/failures include additional training for poll workers, additional reliance on vendor consulting and troubleshooting, “backup resources” (e.g. battery generators, back-up paper ballots, etc.), all of which are costly. Additionally, the reliance on election-equipment vendors is troublesome.
The most reliable instrument for capturing votes is paper ballots.
Accuracy
“Accuracy” is a broad term, used in several different ways pertaining to electronic voting systems. The first, is interpreted in terms of how the computer helps the voter to vote the way he/she intends. This type of “accuracy”, computers can minimize the errors due to voter mistakes—accidentally voting for the wrong candidate/issue, accidentally voting for two or more candidates for the same contest, and accidentally failing to vote for a contest/issue. The first situation can be avoided if the voter carefully examines the printout of his vote and “catches his mistake” before casting his/her vote. The latter two can reduce “residuals” errors by 1) preventing the voter from voting for more than one candidate and 2) warning the voter that he/she did not vote in a particular race.
Vendors of touch-screen electronic-voting devices boast that
their technology results in the lowest “residual error rates.” And this “residual” statistic is disputable
by numerous sources who indicate that Optical Scan technology results in lower
“residual rates.” See www.cs.duke.edu/~justin/voting/totals.html,
www.votersunite.org/info/newmexicophantomvotes.asp,
www.verifiedvotingfoundation.org/article.php?list=type&type=26
, and http://www.ohiovigilance.org/Evoting/Papers/MiamiInitialReportfromSoE.pdf
(page 6
of 35).
[How important are “residual errors” compared to the threat of wholesale vote tampering, a threat endemic to electronic voting machines? I believe voters would rather be sure that their vote counts than to require a system that helps them not make careless mistakes.]
Accuracy also encompasses correctly processing, reporting and counting our votes. This cannot be easily measured. Some say “they’ve been used successfully in the past.” I say “How would you know?”
Another type of accuracy is discussed in the FEC voting systems standards (in section 3.2.1) which were in effect at the time of HAVA enactment (2002). See http://www.epic.org/privacy/voting/eac_foia These accuracy requirements will be discussed in the Hard Requirements / Certification Issues section below.
Cost – Initial and Ongoing
Though it may appear that, because of the “paper” costs involved with optiscan machines, touch-screen equipment (DRE’s) are cheaper to operate. HAVA supposedly does not cover the “paper operating costs.”
HAVA does not specify the requirements sufficiently—“standards” are in a state of flux, and will continue to evolve. As new legislation (e.g. House Bill 3) is adopted, new requirements will dictate replacement and upgrades to voting equipment.
Making large capital expenditures on equipment whose requirements are dynamic is financially irresponsible. Windows-based software products, in particular, are particularly in need of upgrades.
Maintaining thousands of machines which will need to be upgraded and reconfigured will cost millions of dollars.
Obsolescence costs will be high! Diebold talks about its systems being good for 20 years. Who are they kidding? Even if the systems which they are selling were perfect, they’d be obsolescent in a few years.
Counties who, wisely, did not rush to implement electronic
voting machines right after the 2000 election can benefit from the experiences
of other counties who did.
Cost categories that should be considered when evaluating the cost criterion can be found at:
http://www.ohiovigilance.org/Evoting/Cuyahoga/OV-VotingSystemCostAnalysisCuyahogaPunchDieboldDRE.pdf .
Those cells with numbers are taken from Cuyahoga and
See Election Chief Pares Massive Budget Request at www.verifiedvotingfoundation.org/article.php?id=5792
Vendor / Product reputation – business viability
Electronic-voting equipment vendors are all known to be
partisan, primarily Republican. They are
all involved in lobbying government officials.
Diebold, in particular, has been shown to have
funneled money to several
Mark Radke admitted in a board meeting that Diebold has established an Ethics department.
“Dieb
Throat” articles on http://www.bradblog.com or http://www.bradblog.com/archives/00001838.htm
Diebold Talking Points on http://www.bbvforums.org/forums/messages/2197/8580.html.
Digital Quagmires on http://www.voicesofcleveland.com.
Diebold Rep’s Tactics Criticized http://www.verifiedvotingfoundation.org/article.php?id=6279
Diebold is reportedly in a
weakened financial condition and therefore a threat to support voting machines
for their lifetime. “Diebold's
Elections Division is in serious trouble. In this week' s delayed second
quarter earnings report Diebold admits that negative perceptions
of Diebold's AccuVote TS
and TSx voting systems have produced 'lower revenue'
and lower' margin and earnings per share'. In its second quarter earnings
report, Diebold had high hopes its Elections Division
would prop up earnings in its troubled ATM division by 5% to 37%.” See http://www.scoop.co.nz/stories/WO0508/S00199.htm Diebold's Election Division Jolted by Reliability and Security Concerns, Voting
Unit Proves Drag on Earnings and Corporate Image.
Diebold has hired a new PR person, Joe Andrew, a former Democratic National Committee chairman, as ambassadorship for electronic voting. See http://www.bbvforums.org/forums/messages/8/9735.html?1124580871
Bob Bennett admitted a 15-year friendship with the president of Diebold, Wally O’Dell.
Voter Ease of Use / Accessibility / Privacy
Electronic voting machines, for many citizens, seem to be easy to use and even fun. Americans are enamored with high-tech devices.
For
some handicapped voters (blind), touch-screen devices with audio capability can
provide the privacy of voting not had previously available with punch-card or
paper ballots. But are their machines
certified for blind accessibility? What
about other types of handicaps? Do the screens work if the finger
is a prosthetic and not wet-flesh? See Touch
Screen Not Best Choice for Disabled Voters at http://www.votersunite.org/article.asp?id=5588
or http://www.votersunite.org/info/accessibility.asp.
How
accessible are these machines to the elderly?
The elderly make up a much larger segment of the population than the
blind. Why would we want to
disenfranchise the elderly to enfranchise a smaller one?
Also,
the new Automark devices, which can be used with the
Optical Scan machines for blind accessibility, are endorsed by the National
Federation of the Blind, and are now certified in the state of
We should not talk about accessibility for the handicapped or the elderly without talking about the inaccessibility of large groups of individuals to the voting machine due to long lines, intimidation by poll workers and other parties, being turned away because of inaccurate voter-registration data, poorly trained poll workers, or erroneous or missing voter-education information. Many voters fear coming to the polls, who must leave the polls without voting, vote incorrectly, vote in the wrong precinct, etc. These voter-suppression accessibility barriers impact millions of voters! How do DRE’s address these types of inaccessibility? DRE’s will, no doubt, cause longer lines, disenfranchising even more people than before.
There is also a privacy issue relative to DRE’s. With the verified voter paper audit trail, the paper roll holding the election results records the votes in the same sequence as the voters sign in the poll books. Is that not infringing on privacy rights? With the poll book (signature pages) aligned to the VVPAT roll, it would be relatively easy to connect a voter to a vote.
Hard Requirements / Certification Issues
“HAVA made me do it”
Title III, Section 301.a.5 of HAVA states “Error rates—The error rate of the voting systems in counting ballots…shall comply with the error rate standards established under section 3.2.1 of the voting systems standards issued by the Federal Election Commission which are in effect on the date of the enactment of this Act.”
HAVA was enacted in 2002.
Therefore election systems must have proof (i.e., certification from a recognized engineering laboratory) that they meet the 2002 FEC 3.2.1 error-rate standard for error rates.
According to NASED Qualified Voting Systems document, (see pages relative to Diebold systems here: http://www.ohiovigilance.org/Evoting/NASEDDieboldCertification.pdf.) all certified Diebold systems are certified to 1990 voting system standards, NOT the 2002 standards required by HAVA Section 301.a.5. For more information, see http://www.ohiovigilance.org/Evoting/JanowReportDieboldCertification.pdf.
Note: When I brought this certification issue before the Cuyahoga Board of Elections, Chairman Bob Bennett dismissed this issue, saying that certification was not the board’s concern—the board is relying on Ohio Secretary of State Kenneth Blackwell’s due diligence to ensure certification standards are being met. (Bennett, on numerous occasions, has hidden behind Blackwell or behind HAVA.) The level of trust in Blackwell’s “doing the right thing” is marginal. Who will be accountable when election results are challenged because they were derived from non-HAVA-compliant voting machines? Shouldn’t the Boards of Elections assume some responsibility if they know about these violations and still go ahead with their implementation?
It would seem that the board is concerned with security issues. (The board brought in Diebold’s Mark Radke to respond to a number of accusations made by Bev Harris. Board member Ed Coaxum’s questioning of Radke at the July 11th board meeting (www.voicesofcleveland.com) would imply the board is concerned with security issues that Blackwell’s “net” (or any others) had missed. Why shouldn’t the board, likewise, be concerned about certification issues that Blackwell’s office might not have “taken care of?”
HAVA’s Dynamic “Requirements”
Blackwell’s Directives
Risk
To Democracy
The ultimate criteria, is the risk of wholesale, nationwide
fraud that even honest election officials can’t stop. Those who have knowledge of electronic voting
machines, and care about restoring democracy to this country, see these
machines as the ultimate threat to our nation.
For those with a more narrow view, there is great risk to the
reputation of
Cuyahoga County’s Decision Rationale
Director Vu admitted that Optical Scan systems were not even considered, except for absentee-ballot purposes.
One of the criteria used by Cuyahoga to “justify” its Diebold DRE decision was a subjective, questionably measured and weighted, and erroneously tabulated survey by an internal evaluation committee based on information obtained, primarily, from vendor marketing materials and demonstrations. There were 11 criteria which each member of the evaluation team “scored.” For each vendor and for each of the 11 criteria, an average score was obtained by averaging the scores of the respondents. Then, for each vendor, an average score was obtained by adding up the score of the 11 criteria and dividing by 11. The implication is that each criterion was equally weighted—equally prioritized. The 11 criteria are listed on the next page:
Note that “Experience in Conducting Elections” was no more important than “Vendor’s Product Line options—Voter Registration Systems.” (Is this why the Diebold DIMS system was “rush implemented” two months before the Nov 2004 election?) “Ease of Use (BOE Staff)” was weighted equally with “Service Support; Warranty; Maintenance.” “Ballot on Demand”, an Optical Scan attribute, was as important as “Experience in Conducting Election.” I wonder how a committee of non-technical personnel could even measure the last four criteria.
The total (“average”) scores for the four vendors showed Diebold to be the clear preference. But, when I checked the arithmetic, I noticed that ES&S’s score was mis-tabulated. The correct tabulation would have resulted in a “tie” between Diebold and ES&S, but the erroneous tabulated “score” caused Diebold to “appear” to be the undisputed winner (See www.ohiovigilance.org/Cuyahoga/DieboldDecisionAnalysis.pdf)
A second criterion was used to “justify” the Diebold DRE decision—the 2003 Compuware report. The 2003 report showed that Diebold had 5 high-risk problems, as compared to 1 for ES&S. The more recent Compuware report shows that Diebold has “mitigated” the risks, but, at the time of the “final” decision, the 2003 Compuware report was all that was available.
The board also considered some voter surveys, where voters in some shopping malls were asked to “use” the voting machine and describe how they liked the voting systems. The survey was conducted BEFORE the voter-verified paper audit trail was operational, so they did not have to read or understand how to interpret the VVPAT. These results were highly subjective, and Sequoia was the preferred vendor.
Finally, a very rudimentary cost analysis was used by the board to argue that, for “large counties”, optical scan machines are more expensive than touch-screens. The problem with their spreadsheet is that only a small fraction of the costs involved in implementing electronic voting were represented. They virtually ignored all operating costs except for maintenance fees or OS paper. They basically used the numbers which would result in Diebold DRE decision. The original spreadsheet and more comprehensive spreadsheet are found on the website: http://www.ohiovigilance.org/Evoting/Cuyahoga/Index.html.
We’ve asked for other, more comprehensive and more current, cost-analysis or decision-analysis documents, but have not received them.
According to Michael Vu, the board made its “final” decision
back in Feb 2004. I am assuming, since
I’ve received nothing else in the way of decision justification, that the Diebold DRE decision was based upon the above criteria. He acknowledged to Ms. Eisner and me that it
was HE who suggested Diebold DRE’s. Yet Bob Bennett stated in the
Diebold’s Diekman stated at Sept 12, 2005 board meeting what I stated at the Sept 1st meeting—that the board should make its decisions based on “the most recent and up-to-date information.” He cited the more recent Compuware report showing Diebold’s mitigation of the security risks identified in the earlier Compuware report. The board asked Mr. Diekman if they could get a copy of the report, and he responded that he could provide them with one.
It is alarming to me that the board does not already know about that report—especially because they have so steadfastly supported their Diebold decision based on the 2003 Compuware report showing Diebold’s 5 high-risk vulnerabilities, and have virtually ignored the most recent reports and studies casting doubt on their outdated decision.
Diebold seems to think that one of the decision criteria is: the number of other counties who have decided to “go with” Diebold DRE’s. Both Mr. Diekman and Mr. Radke have stated these numbers, as if # of installations were a key criterion. Sometimes this is an important factor, but, in this case, it is not necessarily so. In fact, the large number of installations will require more highly trained troubleshooters on election day—a huge spike in personnel requirement!
Mr. Diekman stated that Diebold had met with the board numerous times to discuss electronic-voting machines. How much time has the public been given to discuss its concerns? Will there be a meeting where the public can air its concerns?
There is an Alternative
We recommend the “National
Ballot Integrity Project’s” solution:
Optical Scans, with hand-counted Federal Elections…and challenge you
to discuss this alternative with other activists who really care about our
democracy and who see the big picture.
This is a hybrid solution that satisfies HAVA, Ohio Revised Code, Blackwell’s Directives, a lucky vendor, concerned voters, your staff (much simpler operation), the taxpayers (is much less expensive), and you, the board (for your wisdom and courage.)
· Use a single technology – for “in person” voters as well as absentee voters
· OptiScans can be made accessible for the blind, and are already better for other types of handicapped voters
· They are as good or better in terms of catching under/over votes
· They supports routine random hand-recounts – during the election, when it is most important
· Much more feasible to do recounts with these ballots
· Much less expensive than DRE’s operationally
[1]
Dr. Lovegren is a senior decision-science analyst, systems and
data analyst, business owner and Mathematics Lecturer at