KeyError when attempting to access Parser

[cjwinchester]

(moved from #13)

Looking at precinct-level results from the 2014 and 2016 primaries in Sarpy County, Nebraska -- in both cases, I'm getting a vote type KeyError when I try to access the Parser instance. I'm running Python 3.4.

The XML file for the 2014 primary is here.

Here's the code I'm stepping through for the 2014 results, up to the point where it breaks:

from __future__ import print_function
from zipfile import ZipFile

try:
    from StringIO import StringIO as ioDuder
except ImportError:
    from io import BytesIO as ioDuder

import unicodecsv
import requests
import clarify


county = 'Sarpy'

url = 'http://results.enr.clarityelections.com/NE/Sarpy/51545/184335/en/summary.html'

election_type = 'primary'

def clarify_sarpy():
    '''
    1. Fetch zipped XML file.
    2. Unzip in memory.
    3. Load into Clarify.
    4. Loop over results, write to file.
    '''

    # discover path to zipfile and fetch
    s = clarify.Jurisdiction(url=url, level='county')
    r = requests.get(s.report_url('xml'), stream=True)
    z = ZipFile(ioDuder(r.content))

    # hand off to clarify
    p = clarify.Parser()
    p.parse(z.open('detail.xml'))

    for result in p.results:
        print(result)

Here's the traceback:

Traceback (most recent call last):
  File "parse_2014_primary_sarpy_precinct.py", line 132, in <module>
    clarify_sarpy()
  File "parse_2014_primary_sarpy_precinct.py", line 35, in clarify_sarpy
    p.parse(z.open('detail.xml'))
  File "/home/cjwinchester/.virtualenvs/clarify/local/lib/python3.4/site-packages/clarify/parser.py", line 48, in parse
    self._contests = self._parse_contests(tree, self._result_jurisdiction_lookup)
  File "/home/cjwinchester/.virtualenvs/clarify/local/lib/python3.4/site-packages/clarify/parser.py", line 256, in _parse_contests
    return [self._parse_contest(el, result_jurisdiction_lookup) for el in contest_els]
  File "/home/cjwinchester/.virtualenvs/clarify/local/lib/python3.4/site-packages/clarify/parser.py", line 256, in <listcomp>
    return [self._parse_contest(el, result_jurisdiction_lookup) for el in contest_els]
  File "/home/cjwinchester/.virtualenvs/clarify/local/lib/python3.4/site-packages/clarify/parser.py", line 283, in _parse_contest
    for r in self._parse_no_choice_results(contest_el, result_jurisdiction_lookup, contest):
  File "/home/cjwinchester/.virtualenvs/clarify/local/lib/python3.4/site-packages/clarify/parser.py", line 322, in _parse_no_choice_results
    subjurisdiction = result_jurisdiction_lookup[subjurisdiction_el.attrib['name']]
KeyError: 'ABSENTEE'

Thanks for taking a look!

[bsmithgall]

Hello! I was poking around some of the open elections code looking for ways to contribute this weekend and found this. I am entirely new to the world of open elections so this might be off-base, but I did some research about Sarpy county in particular and found the following:

1. There are three things that are considered to be <Precinct> elements in a particular <VoteType> underneath a <Contest> in the given election above (2014 General Election) that do not appear in the list of Precincts underneath VoterTurnout: ABSENTEE, Nonpartisan-Partisan, and Nonpartisan-Partisan 2. ABSENTEE has no votes in the entire county, but the other ones do have some:

   grep '"ABSENTEE" votes=' test.xml | grep -v '"0"' | wc -l
          0
   
   grep '"Nonpartisan-Partisan"' test.xml | grep -v '"0"' | wc -l
         63
   
   grep '"Nonpartisan-Partisan"' test.xml | grep -v '"0"' | head -n5
           <Precinct name="Nonpartisan-Partisan" votes="1"/>
               <Precinct name="Nonpartisan-Partisan" votes="5"/>
               <Precinct name="Nonpartisan-Partisan" votes="30"/>
               <Precinct name="Nonpartisan-Partisan" votes="7"/>
               <Precinct name="Nonpartisan-Partisan" votes="43"/>

   Since the parser uses the XPATH at /ElectionResult/VoterTurnout/Precincts/Precinct to figure out what to consider to store in _result_jurisdictions, the whole thing would blow up when it finds any of the above. This same thing happens in the 2016 primary.

2. Oddly enough, summing ballotsCast attributes of the precincts found under the /ElectionResult/VoterTurnout/Precincts/Precinct XPATH match the ballotsCast attribute of the VoterTurnout (which is probably calculated from summing the individual precincts. This leads the broader question of where the ballots considered under the Nonpartisan-Partisan name are counted.

3. The most recent election to appear on the list of election results[0] (the 2016 general election[1]) have Precincts that are not prefaced by "Precinct," but those do appear in the list of Precincts that ultimately ends up in the parser's _result_jurisdictions. This combination of things leads me to believe that it was probably a bug in whatever report generation engine that created the XML file that has (maybe?) since been fixed.

This leaves the question of what to do. I notice that when a new ResultJurisdiction is created (such as on parser#146[2]), it's initialized with all of the aggregated data coming directly out of
/ElectionResult/VoterTurnout/Precincts/Precinct. It seems like there is an edge case where we don't have that pre-built aggregation, so maybe the solution would be to add some methods to the ResultJurisdiction object that would let us build that manually? This would fix the issue but perhaps create another one where we would have invalid ballot counts (or at least counts that seem to contradict the final results?) so I would love some extra input on where to go.

[0] http://www.sarpy.com/election/election_results.html
[1] http://results.enr.clarityelections.com/NE/Sarpy/64652/184054/en/summary.html
[2] https://github.com/openelections/clarify/blob/master/clarify/parser.py#L146