June 20, 2007 | Jason Kitcat

ORG Election Report highlights problems with voting technology used

Today ORG releases its report into the May 2007 elections in Scotland and England. The result of a huge team effort and planning which began late last year, the report provides a comprehensive look at elections that used e-counting or e-voting technologies.

As a result of the report's findings ORG cannot express confidence in the results for the areas we observed. This is not a declaration we take lightly but, despite having had accredited observers on location, having interviewed local authorities and having filed Freedom of Information requests, ORG is still not able to verify if votes were counted accurately and as voters intended.

The report identifies problems with the procurement, planning, management and implementation of the systems concerned. But more fundamentally, given that problems were so widespread, the evidence supports the view the e-voting and e-counting technologies are not suitable for conducting statutory elections.

The report can now be downloaded from our e-voting pages.

Comments (20)

  1. The Open Rights Group : Blog Archive » ORG verdict on London Elections: “Insufficient evidence” to declare confidence in results:
    Jul 02, 2008 at 12:02 AM

    [...] Elects will pay Indra – the company who supplied both Bedford and Breckland during last year’s chaotic trials of e-counting technology in local elections – upwards of £4.5 million for delivering the London e-count. Today’s report recommends a [...]

  2. Links » I Can Haz Votez?:
    Jun 20, 2007 at 10:48 AM

    [...] The Open Rights Group released its report on e-counting and e-voting in the recent elections. Executive summary: it didn’t work very well. [...]

  3. Timothy Birt:
    Jun 26, 2007 at 10:26 AM

    Thank goodness for the Open Rights Group!

    As the contributor who filed submissions on behalf of the Breckland Green Party I thank the ORG for confirming and voicing the concerns of many people who were present. It appears to me that Breckland were hunkering down and waiting for it all to be 'forgotten' but reports such as the one by the ORG force the issues into the open. The catalogue of errors and procedural mistakes need to be addressed and corrected but there still appears to be a significant "spin" being applied; Mary Palmer at Breckland was quoted (EDP 2-6-2007, p.9) as saying: "We carried out manual and electronic counts in Humbletoft and it has no affect on the overall result". I don't agree ... the numerical RESULT changed dramatically with more than twice the number of votes included when counted manually, it did confirm that the correct person was elected but importantly by a different percentage margin. This may, or may not, be the case elsewhere in other wards, whatever the situation we need to be confident that the correct number of votes were counted.

    I have stated elsewhere that I don't intrinsically have a problem with mistakes or errors, it is human nature that these will happen. It is what happens after they are known which is more important. I am minded that Breckland have overstepped the normal allowance for mistakes and errors as the the Returning Officer repeatedly chose to ignore my comments on the day.

    I understand that the Norfolk Police have been asked to investigate the matter, but whether there is sufficient evidence of any illegality remains to be seen. It is certainly my opinion that Breckland acted unethically and continues to do so by not openly addressing the issues.

  4. Jason Kitcat:
    Jun 24, 2007 at 12:01 PM

    Steve, there are already some EU funded e-voting projects as well as a number of nationally or regionally funded e-voting projects in countries such as Estonia, France and I think in an autonomous region of Italy.

    However there are fundamental computer science challenges to building a secure and anonymous electronic voting system, it is a 'hard' problem due to the requirements of voter anonyminity, security, vote authenticity and auditability. ORG's e-voting briefing pack provides more details on the challenges in building such systems which meet the requirements for free and fair democratic elections.

    The technology and processes for secure e-voting are not yet available in practical forms, so ORG argues against introducing e-voting.

  5. Jason Kitcat:
    Jun 23, 2007 at 04:05 PM

    Thanks for your comment James, I too responded to it elsewhere, but aggregate my replies here for reader convenience.

    There is no doubt that we had access to less information overall when examining the Scottish elections when compared with the English pilots. This was in part because we started later in Scotland, because we'd been advised that we wouldn't be able to observe in Scotland, and also because the Scotland Office and Scottish Executive were not willing to share even a smidgeon of information with us, which DCA did in England (though they could have done much more).

    In drafting the report ORG tried to avoid pulling in too much deep background, instead focussing on what we observed in and around the elections themselves. The Arbuthnott Commission's recommendations, whilst obviously of some importance to the process, should not have created a situation where the logical alternative of presenting the columns the other way around was not tested at all. This is poor usability practice. Douglas Alexander MP, Secretary of State for Scotland, specifically referred to the ballot paper design research as involving 'focus groups in Glasgow, Edinburgh, Inverness and Dundee'. (House of Commons 8th May 07)

    It is another failing of the actual report produced by the firm commissioned, Cragg Dawson Ross, that it does not actually name the methodology used, but from the brief details supplied it is clear qualitative marketing methods were used, not usability testing methods.

    This matter has been followed up with the Electoral Commission who only responded to this query the day before publication, stating that the method was 'hall testing' which is again a marketing method - not a usability one. So while what actually was done remains unclear, I think it would be fair to surmise that a market research approach was taken making the study unsuitable for the task of judging the ballot designs.

    I do not think it is a fair characterisation to say that ORG's report is incorrect on this matter, it's certainly not correct to say that we didn't check up on this fact. It wasn't easily verifiable and still isn't, despite talking to the Electoral Commission in Scotland and London as well as the Scotland Office plus checking Hansard.

    I think you have misunderstood our point regarding media coverage. We do not claim that the ballot sheets were mis-represented, but that general media reporting always discussed the constituency before the regional. I have screenshots to support this. This year and previously print and broadcast media, the Scottish Parliament and local authorities all reported constituency results before regional and laid them out on the screen with the constituency either above or to the left of regional results. It is ORG's view that this played an important part in what voter's expected from the ballot paper. As the Arbuthnott Commission argued, putting the regional column on the left would have been to highlight it's important -- which in ORG's view was the opposite of how voter's perceived the elections, they thought of constituencies first. We do not question combining the ballot papers to a single paper, we question the design and the process used to support the decisions made in making that design. A proper process may have argued that separate ballot papers were better, or a workable combined paper could be produced. We don't know as those tests weren't done in the correct way.

    While I'm sure the report isn't perfect, I'm yet to receive details of any factual errors in the report. We may agree to differ on why so many ballots were rejected in Scotland, but that will come down to opinion because the papers themselves are not available for inspection.

    One point that James raises is over the distinction between rejected and spoilt ballots. The ORG report does interchange those terms occasionally, we should have been clearer on that. However it is worth noting that across Scotland and England the term 'rejected' and 'spoilt' are used in woolly and inconsistent ways by Returning Officers and others in the election world. So perhaps getting better at our terminology - along the US lines of overvotes, undervotes, intentionally spoilt etc would be helpful for the discussions we are sure to be continuing as the Electoral Commission starts to issue its reports.

    All the best,

  6. engineeering:
    Jun 22, 2007 at 08:24 PM

    its really ver important for all......the goverment should give more empahsis of this matter ........In light of the problems it uncovered, the ORG said a halt should be called to e-voting trials to ensure that their shortcomings are addressed before they are more widely used. ..........

  7. Jason Kitcat:
    Jun 22, 2007 at 08:53 PM

    Whilst certainly the Open Voting Consortium solution is better than the commercial touchscreen solutions used primarily in the USA, I still am not keen on its design. It reads barcodes for counting votes, but barcodes are not a human-readable representation of votes and so are not easy to scrutinise.

    Making available the source code of e-voting or e-counting systems is no doubt better than not having it available, there are significant problems in ensuring that the source code you inspect is actually the basis of the software used in systems on election day. This is a real problem: suppliers have been caught using different versions of software to what they have claimed. My paper Source availability and e-voting: an advocate recants goes into more detail about why opening the source doesn't fix enough problems to make e-voting worth the risk.

    Essentially there are a large number of risks involved in using e-counting and e-voting systems. As the vast majority of voters and candidates will be unable to understand source code or associated technical details of systems, then whether open source or not, the technology renders elections difficult or impossible to verify and scrutinise. This is not an acceptable state of affairs for statutory elections, hence ORG's opposition to introducing e-voting and e-counting in the UK.

  8. Steve Richards:
    Jun 24, 2007 at 09:10 AM

    The Open Voting Consortium solution is one way forward for the UK, but I would have thought that within the UK, we have enough talent to develop an open system which would satisfy all of our security and verification concerns.

    Within the UK we have many thousands of engineers who build safe secure systems, comfortable with the need to perform research into man machine interactions.

    We could setup a project to specify some solutions to e-voting, funding would not be an issue until hardware is required.

    I am sure that a number of companies would build prototype hardware free of charge to the groups specification (hoping to get the production orders)

    Even EU or UK funding could be explored for the hardware development.

    The development of a safe, secure and trusted e-voting solution would be problematic but not unsolvable if the will is there and the talent of UK engineers is used to the full.

  9. Jonathan Hogg:
    Jun 23, 2007 at 10:48 AM

    Well, the story has made it on to the BBC.

  10. The Opinions of a Loud Mouth Man :: Being open about being open.:
    Aug 01, 2007 at 08:45 AM

    [...] I was very disappointed then to have joined the Mailing list for the ORG just as the discussion about “should the archives be open ?” was kicking off. Now colour me confused but if I join a group devoted to encouraging transparent decision making processes and open formats then I might have a reasonable expectation that the maillist and conversations that I have will also be available to the public without a barrier or “subscription”. [...]

  11. Elliot Long:
    Jun 22, 2007 at 01:01 PM

    One must be wary of claiming that e-voting is inherently unreliable. Suggestions exist for accountable and transparent solutions that would improve the process while retaining accountability. I'd love to see UK-wide support for the Open Voting Consortium - the ideal solution. http://www.openvotingconsortium.org

  12. James Gilmour:
    Jun 23, 2007 at 03:30 PM

    I tried to post this comment soon after I had read the report on 20 June. The box cleared after I hit the "Submit Comment" button, but nothing appeared and it seems nothing has arrived. So I'll try again. This is the unaltered, original comment that has been the subject of discussion elsewhere. JG


    The ORG report on the recent elections is extremely valuable, but so far as the Scottish Parliament elections are concerned, it contains some small but significant mistakes and omits a great deal of background information that is extremely relevant to the main point of failure and the main point of criticism. I have no connection with any of the organisations responsible for these elections, but I was an Accredited Observer (independent, private individual) and attended the counts at the Edinburgh Counting Centre.

    The recommendation to present the two ballot papers for the Scottish Parliament elections on a combined "ballot sheet" came from the Arbuthnott Commission because research had shown that voters did not understand the function of the two AMS votes and did not appreciate the importance of the regional vote in determining the outcome. The Arbuthnott Commission also recommended that the regional ballot paper should be placed first of the sheet, i.e. on the left, to emphasise its importance. This is the design used for AMS elections in New Zealand, where it was adopted for the same reasons. Because this was the priority consideration, it should be no surprise that only designs of this kind (regional ballot paper on the left) were compared with two separate ballot papers in the qualitative research.

    Repeated mention is made of the use of "focus groups" and criticism made of the use of such groups for evaluating alternative designs. But so far as I can see, NO focus groups were involved. The Cragg Ross Dawson report describes the selection of the "test voters" in these terms: "2. Methodology • 100 short qualitative interviews were conducted in four locations in Scotland. • interviews were semi-structured and lasted about 20 minutes. • respondents were recruited on the street and interviewed immediately, with no prior warning. • they were asked to use each of the five proposed ballot papers in polling booths as if taking part in a real election, and then discussed them with the interviewer". This indicates to me that the 100 voters participated individually, not in any kind of focus group.

    It should be noted that the most significant finding to emerge from that qualitative research was the overwhelming preference for a combined ballot sheet (83/100) rather than two separate ballot papers (17/100). No other finding showed such a clear margin of difference.

    The report states: "ORG notes that, in general, media coverage tended to place constituencies before regions when reporting on the Parliamentary elections." I saw no examples of this. All the media illustrations and descriptions of the AMS ballot sheet I saw had the regional ballot paper of the left. If ORG has any examples of incorrect media coverage of the kind they suggest, it is essential that that evidence is given to the Independent Inquiry now being conducted by Ron Gould.

    The ORG report mostly refers (correctly) to "rejected ballot papers", but occasionally refers (incorrectly) to "spoilt ballot papers". These two terms have very clear and very different meanings in electoral legislation. The problems in the Scottish Parliament elections were with "rejected ballot papers". The word "spoilt" also conveys the implication that the voters whose ballot papers were rejected had, necessarily, made mistakes. In some cases this was certainly true, most commonly by placing two Xs in the regional vote column and consequently making no mark in the constituency vote column.

    Constituency ballot papers that were blank were correctly "rejected", but it would be quite wrong to assume that all such rejected papers had been left blank by mistake. The differences in the numbers of rejected regional ballot papers and rejected constituency ballot papers show that, in many constituencies, the voters had cast a valid regional vote (one X) but did not vote in the constituency. I know that this was done deliberately in some cases because I have been consulted about it by voters since the election and have read in newspapers and blocs about voters who also did exactly that and did it intentionally. The interpretation of the numbers and proportions of "rejected ballot papers" is thus more complex than suggested in the ORG report. I have done a great deal of analysis of the data on rejected ballot papers and there are clearly very significant differences between electoral regions and among constituencies within most of the electoral regions.

    There were obvious design mistakes in the AMS ballot papers, in particular the omission of the "directional" arrows above the voting columns on the ballot sheets used in the Glasgow and Lothians electoral regions. However, the real puzzle is why so many Scottish voters did make real mistakes in completing the combined ballot sheets. Such combined ballot sheets are used for AMS elections in both New Zealand and in Germany. In both countries the proportions of rejected ballot papers are very much lower than we saw in Scotland on 3-4 May.

    James Gilmour

  13. Geoffrey A. Landis:
    Jun 21, 2007 at 11:45 PM

    This is very frightening-- it's not just that the election is unreliable, but they can't confirm whether the result is trustworthy or not.

    My thinking is that it should be illegal to use any voting software unless the source code is available for inspection by anyone. The electronic-voting companies all seem to be saying that the source code is proprietary, nobody is allowed to look at it and see it. That's the equivalent of saying, I'll count the votes, and you should just trust me. The actual voting in a democracy needs to be secret, but every aspect of counting the votes needs to be completely, absolutely, without exception open.

  14. Matt Harwood » Blog Archive » Open Rights Group E-Voting Report:
    Jun 20, 2007 at 12:43 PM

    [...] ORG have released a report alerting to the flaws in the technology used within the UK’s E-voting system. Further coverage, in The Register and the Guardian. [...]

  15. Toblog : TCSOTD 2007-06-20:
    Jun 20, 2007 at 07:01 AM

    [...] ORG’s damning report on Electronic Voting has been released [...]

  16. Light Blue Touchpaper » “No confidence” in eVoting pilots:
    Jun 20, 2007 at 04:40 PM

    [...] Today ORG launches its collated report into all of the various eVoting and eCounting experiments that took place in May — documenting the fiascos that occurred not only in Bedford but also in every other place that ORG observed. Their headline conclusion is “The Open Rights Group cannot express confidence in the results for areas observed” — which is pretty damning. [...]

  17. W:
    Jun 20, 2007 at 06:08 AM

    Do we know what the trials cost, and is there a "cost per vote" figure?

    Something else I dont get - we have elections every two years or so. Does this kit just get used once then hang around unused for two years? What does the depreciation look like? This stuff goes obsolete fast, I'd have thought.

    Imagine having to sit down and do a critical online purchase using your PC untuched from two years ago.

    Imagine a recording studio that only gets used once every two years. What chance it works right out of the box? You'd have to clean up the cobwebs and conections...

    GOOD LUCK TONIGHT ALL. I'll be there :-)

  18. Sicurezza, ICT ed altro » Blog Archive » Sempre problemi con il voto elettronico:
    Jul 06, 2007 at 02:09 PM

    [...] Un ennesimo report, del quale parla anche Ross Anderson, che descrive nuovamente problemi con i sistemi di voto elettronico. Si può anche continuare ad insistere con l’idea di trovare sistemi di voto più o meno sofisticati, ma è un dato di fatto che quello che viene prodotto non offre le garanzie necessarie per un’attività così delicata. Purtroppo, molti dei rapporti entusiasti di alcune sperimentazioni sostengono che sono state un successo perché non sono stati riportati problemi; il problema è se questi problemi sono stati cercati, o se era possibile trovarli. Ci sono un paio di passaggi dell’articolo di Anderson che vorrei riportare: [...]

  19. No confidence in e-voting « UK Liberty:
    Jun 21, 2007 at 06:27 PM

    [...] No confidence in e-voting June 21st, 2007 See the Open Rights Group report, with commentary from volunteer Richard Clayton, Ideal Government, and SpyBlog. [...]

  20. Radio Clash Mash Up Podcast - a weekly podcast of mixes, mash-ups, and more since November 2004! - one of England's longest running podcasts! » Blog Archive » Our own Florida?:
    Jul 02, 2008 at 02:03 PM

    [...] cannot easily know as all the votes are counted inside a machine, and the company behind it Indra was involved with those trials last year which were far from perfect, total chaos in fact. And now the just released Open Righs Group report [...]