Don't let Cubic payment URLs get indexed by Google during test phase

[i0sea]

Summary of changes

Asana Ticket: Don't let Cubic payment URLs get indexed by Google during test phase

---

update robots.txt to stop urls from being indexed

General checks

• Are the changes organized into self-contained commits with descriptive and well-formatted commit messages? This is a good practice that can facilitate easier reviews.
• Testing. Do the changes include relevant passing updates to tests? This includes updating screenshots. Preferably tests are run locally to verify that there are no test failures created by these changes, before opening a PR.
• Tech debt. Have you checked for tech debt you can address in the area you're working in? This can be a good time to address small issues, or create Asana tickets for larger issues.

New UI, or substantial UI changes

• Cross-browser compatibility is less of an issue now that we're no longer supporting IE, but changes still need to work as expected in Safari, Chrome, and Firefox.
• Are interactive elements accessible? This includes at minimum having relevant keyboard interactions and visible focus, but can also include verification with screen reader testing.
• Other accessibility checks such as sufficient color constrast, or whether the layout holds up at 200% zoom level.

New endpoints, or non-trivial changes to current endpoints

• Have we load-tested any new pages or internal API endpoints that will receive significant traffic? See load testing docs
• If this change involves routes, does it work correctly with pertinent "unusual" routes such as the combined Green Line, Silver Line, Foxboro commuter rail, and single-direction bus routes like the 170?

[kotva006]

Slice function logic needs reworking. Reach out if you need more clarity or to pair

[kotva006]

Both of these slice calls need to be changed. The function as written will return String.slice("/charlie/", 0..4) -> /char and String.slice("/policies/terms-use-charlie/", 0..4) -> /poli. This will fail to match on the desired string

[kotva006]

The urls we wish to not index with this pattern looks like there is no final slash. I believe it should be /policies/terms-use-charlie*
This might need to be double checked with the robots.txt documentation.
Also this needs to be updated in the app.html.eex file to not include the final slash

[kotva006]

Since the slash got removed in the /policies/terms-use-charlie the count in slice needs to go down as well