Description (e.g. "Related to ...", etc.)

We do not currently, and have never needed to pass a public key pem to the piv context manager.
The code that is invoked if this pem is passed does not work, but the general idea is also not robust enough (see comments in #515).
Since we are not using this, I think that we should just remove this code. We'll implement it if a need for it arises.
Also found and fixed a minor setup_test_yubikey issue

Closes #515

Code review checklist (for code reviewer to complete)

• Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR)
• Title summarizes what is changing
• Commit messages are meaningful (see this for details)
• Tests have been included and/or updated, as appropriate
• Docstrings have been included and/or updated, as appropriate
• Changelog has been updated, as needed (see CHANGELOG.md)