Support ES&S Multi-Contest CVRs

[artoonie]

Closes #981

This PR deprecates "Treat blank as UWI" and "Undervote label." For ES&S, the undervote label is hardcoded. For UWI, we now explicitly search the XLSX for write-in images. Blanks are invalid unless the entire row is blank. When that happens, we report it as part of an irrelevant contest in the audit logs.

This PR discovered that the 2015 Portland Mayor CVRs had images, but "treat blank as UWI" was false. That meant we mistakenly treated blanks as undervotes, not UWI. This PR fixes that in two ways:

1. 2015 Portland Mayor test: update the output files, now properly handling UWI
2. 2015 Portland Mayor Codes test: update the input CVR, removing the images and replacing with "undervote," and demonstrating the output files do not change

Given two fewer config options on the right-hand column, the GUI columns became imbalanced. I moved "Contest ID" from the left column to the right to balance it.

[artoonie]

This PR found a bug:

While Apple Numbers only shows non-empty rows, the underlying XLSX XML data can store empty rows. That led to some CVRs having hidden empty rows which we erroneously marked as undervotes. We now correctly detect these rows as rows to be completely ignored and thrown out.

[artoonie]

TODO: Audit log should say "read X ballots, of which Y were ignored" or similar, rather than X=only non-empty ballots

[artoonie]

TODO: Audit log should say "read X ballots, of which Y were ignored" or similar, rather than X=only non-empty ballots

Rather than modifying that row, I opted to add a warning so (A) it's more visible, and (B) we don't need to propagate what is likely source-specific values up from the ES&S reader through the generic reader and into the tabulator session.

It says "Ignored %d rows with no votes for any candidates"

[yezr]

Testing I found that setting Treat Blanks As Undeclared Write-In to true has some unexpected behavior now. When it sees a fully blank record it calls handleEmptyCells at the beginning of endCvr(). It writes into each of those blank ranks an undeclared write-in ranking.

However, the hasSeenAnyNonBlankCandidateCells boolean is only updated in the cell data call back, which never gets fired because all cells are blank. The ranking of all Undeclared Write-In ranks is then filtered out.

Maybe we need to set the hasSeenAnyNonBlankCandidateCells to true when we set a UWI?

[artoonie]

It seems that Treat Blanks As Undeclared Write-In is fundamentally incompatible with this approach. Either blanks mean UWI or it means Unrelated Contest. I wonder if we should go back to the drawing board and consider looking at the other columns to disambiguate. Or, better yet -- find out what's actually possible in the ES&S data format so we can support it without guessing.

If we want to keep this approach, I suggest changing the Treat Blanks As Undeclared Write-In from a checkbox to a dropdown: Blanks are... > [Ignored, Treated as Undeclared Write-Ins]

[yezr]

If we want to keep this approach, I suggest changing the Treat Blanks As Undeclared Write-In from a checkbox to a dropdown: Blanks are... > [Ignored, Treated as Undeclared Write-Ins]

I think you are right that it needs to be mutually exclusive. For our options

• default option of Any Blank Invalid - Causes Tabulation to Fail
• for any single blank being treated as a UWI, the current behavior when the Treat Blanks As Undeclared Write-In is checked, it now says Any Blank Valid - Individual Blanks Treated As UWI
• I think for the Ignored option we might change the description to make it explicit that it is only ignored if it is a full ranking of blanks. Maybe Valid Only As An All Blank Ranking Ballot - Not included in tabulation (For Multi-Contest Exports)

We can work on this vocab...

[artoonie]

Makes sense.

Suggested vocab:

1. Invalid (fail tabulation)
2. Undeclared Write-In (valid if only one blank per voter)
3. Irrelevant Contest (skip row if all rankings are blank)

With 3, the CSV field listing the number of voters who didn't vote at all (Undervotes-No Ranking) would always be zero, and that's invalid. Does it make sense to write "unknown" instead of zero for that field?

[artoonie]

This change is getting larger than it seemed as first, so we'll definitely need to add tests to handle each of these options. The code isn't ready yet, but the functionality is more or less there if you want to give it an initial test @yezr to verify this is the right direction.

[yezr]

As we got into this ticket, we started to see clearly that getting some of our assumptions confirmed about the structure of the ES&S CVR export format would help. I was able to speak to ES&S reps about the CVR export and confirmed some assumptions.

• Most importantly, I confirmed that it is always the text "undervote" when a ballot has an option to rank/vote in a contest and no ranking/vote is made.
• When a ballot is included in an CVR export and a contest is included in that export that doesn't appear on that ballot, the rankings/votes for that contest will all be blank.
• For write-ins, there is an option in the export to either 1) insert a picture of the write-in portion of the ballot in the cell of that ranking. That picture is interpreted in our parser as a blank OR 2) the text "Write-in". I'm going to find a CVR export that has the pictures in it to confirm that. If we could at least differentiate between a picture and a blank that might help us even more.

That should at least remove the need to handle blanks as skipped ranks.

[yezr]

Finally got an example ES&S CVR with write-in snippets.
ess_cvr_with_images.xlsx

It's a plurality contest, but it should have the same structure as an RCV CVR.

[artoonie]

This gets fun-- images aren't associated with cells, they're simply placed relative to cells but are global items.

Step 1: Go through all images, find which cells they seem to be associated with, and mark those cells as UWI
Step 2: When parsing, check the list of UWI images we found

That means blanks can either only be "invalid" or "irrelevant contest".

Code needs clean-up, but app is ready to test @yezr

[yezr]

What we know now about the ES&S CVR structure

• We know that there is only one possible reason that we would ever see blanks in an ES&S CVR. If a contest included in the export is not a contest on a ballot that is included in the export, then ALL rankings for that contest will be blank. Any other blanks, like a single blank in a contest that has at least one ranking, would be a halting error.
• We know that if a ballot has an opportunity to rank a contest, but choosing not to use one or more rankings in that contest, the text undervote will be in the ranking cell.

Also, with the changes we've made we are now able to "see" images in the CVR. Before, these were being parsed as empty cells. Now, we are able to differentiate between an empty cell (an unused ranking), and a cell that has an image (a ranking for a write-in candidate).

I think then we can get rid of some config options in RCTab

• We can get rid of the Undervote Label configuration textbox altogether. We know that this will always be the text undervote.
• We can get rid of any Treat Blank As _____ configuration options. The only time a blank is not a halting exception that stops tabulation is if ALL rankings for a contest are blank. In that case, the contest was not on the ballot and that ballot should not be included in tabulation. Any other blank is a malformed CVR and should halt tabulation.
• I think we can also get rid of the Undeclared Write-In Label and Treat Blank as Undeclared Write-in? Now we know that we can flag something as UWI if the ranking cell is an image or the text Write-in.

[artoonie]

TODO -- think about how to handle this.

Currently the logic is based on whether UWIs are configured. For ES&S, there is no configuration needed: the presence of an image is sufficient. So we'd need to refactor this logic to occur after we read the CVRs.