- Author: Marc. C. Ubaldino, MITRE Corporation
- Date: 2014-June; updated 2017-August
- Copyright MITRE Corporation, 2012-2017
XCoord is a geographic coordinate extractor. It finds the most common coordinate patterns in free text. That is, if you want to geocode documents, chat messages, bulletins, etc that contain degrees/minute/seconds, decimal degrees or military grids (MGRS) you will want to use something like XCoord.
Try the XCoord demo in Examples source folder or in the distribution, as below.
The XCoord function is integrated fully with Xponents SDK and REST API by default.
Input files that contain various coordinate patterns to see how the extraction behaves.
./script/demo.sh xcoord -h
TestXCoord -f -- system tests
TestXCoord -i TEXT -- user test
TestXCoord -t FILE -- user test with file; one test per line
TestXCoord -t FILE -- user test with file
TestXCoord -a -- adhoc tests, e.g., recompiling code and testing
TestXCoord -h -- help.
The main program and class XCoord is at XCoord while the accompanying patterns and rules file is geocoord_patterns.cfg .
For reference, review the XCoord DEFINES as you review RULES. There are subtle variations in field definitions.
For brevity sake, only true positive tests are included. "FAIL" tests or true negatives are omitted.
One test case per RULE is provided to illustrate each pattern. Sources of patterns are derived from
federal research projects performed by the MITRE Corporation.
These five families of patterns are supported:
Conventions in pattern IDs. Each pattern is enumerated with the its family; Additional nomenclature includes:
- a = trailing hemisphere
- b = prefix hemisphere
- v = variable field length
- dot = use of period separator
- fs = fractional second variant
- deg = has explicit use of degree symbol, and others
Table 1. Sample Listing of XCoord Patterns and Example Targets for Extraction
| Family | Pattern ID | Example |
|---|---|---|
| MGRS pattern | ||
| MGRS | MGRS-01 | 38SMB4611036560 |
| UTM pattern | ||
| UTM | UTM-01 | 17N 699990 3333335 // Zone/Latitude band + northing + easting; Optionally with units "m" // for meters and or N/E marker |
| Degree-Minute-Second patterns | ||
| DMS | DMS-01fs-a, DMS-01fs-b | 01°44'55.5"N 101°22'33.0"EN01°44'55.5" E101°22'33.0"// fractional second resolution, w/hash marks, with hemisphere |
| DMS | DMS-01fs-deg | 01°44'55.5" 101°22'33.0"// fractional second resolution, w/hash marks, NO hemisphere |
| DMS | DMS-01dot-a, DMS-01dot-b | 01.44.55N 055.44.33EN01.44.55 E055.44.33// explicit dot separator |
| DMS | DMS-02 | N42 18' 00" W102 24' 00" // variable length fields with separators and hemisphere |
| DMS | DMS-01a, DMS-02a | 421800N 1022400WN421800 W1022400// no field separators, D/M/S |
| DMS | DMS-03a, DMS-03b | 4218001234N 10224001234WN4218001234 W10224001234// no field separators; D/M/S.ss assummed |
| Degree-Minute patterns | ||
| DM | DM-00 | 4218N-009 10224W-003// obscure fractional minute notation |
| DM | DM-01a, DM-01a-dash, DM-01a-dot; DM-01b, DM-01b-dash, DM-01b-dot |
42 18-009N 102 24-003W42-18-009N; 102-24-003W42.18.009N 102.24.003W// Ambiguous fractional minute separator // is handled with distinct patterns N4218.009W10224.003N42 18-005 x W102 24-008N42.18.005 x W102.24.008 |
| DM | DM-02a, DM-02b, DM-02b-dash | 4218.009N 10224.003WN4218.0 W10224.0N4218-0018 W10224-0444// 02a/b allows for fixed-width D/M without separators. |
| DM | DM-03a, DM-03b | 4218009N10224003WN4218009W10224003// Fixed-width patten for D/M.mmm |
| DM | DM-03-av, DM-03-av-deg, DM-03-av-decdm | N42 18' W102 24'42° 18' 102° 24'42° 18.44' 102° 24.11'// D/M pattern with explicit hashmarks and separators // 03-av-decdm is pattern with NO hemisphere |
| DM | DM-03-bv | 42° 18'N 102° 24'W// trailing hemisphere, minute resolution |
| DM | DM-04a, DM-04b | N4218 W102244218N 10224W// trivial DMH or HDM pattern. |
| DM | DM-05 | /4218N4/10224W5/// Rare military format with checksum value. |
| DM | DM-06 | OBE |
| DM | DM-07 | 42 DEG 18.0N 102 DEG 24.0W// 'DEG' spelled out. fractional minute resolution |
| DM | DM-08 | +42 18.0 x -102 24.0 |
| Decimal Degree patterns | ||
| DD | DD-01 | N42.3, W102.4 |
| DD | DD-02 | 42.3N; 102.4W |
| DD | DD-03 | +42.3°;-102.4° // explicit degree notation required, otherwise it is just a pair // of floating point numbers. |
| DD | DD-04 | Latitude: N42.3° x Longitude: W102.3° // Lat/Lon fields in text, decimal degree resolution |
| DD | DD-05 | N42°, W102° |
| DD | DD-06 | 42° N, 102° W |
| DD | DD-07 | N42, W102 |
| END |
- Numerous javadoc API updates
- MGRS filters for well known dates/months, lower case (default is to filter out lowercase), and Line endings in Latband/GZD
- MGRS exception reporting calmed down
- Version number for XCoord carrries with the rest of Xponents
- Xponents now fully maven built, set apart from the "opensextant" super project.
- Improved concept of precision/resolution
- Added more MGRS filters and pattern refinements
- Parsing unbalanced pattern matches better now, while being more flexible with allowed punctuation. Managing false positives from looser patterns is important. E.g., pattern with +/-DEG DEG MIN SEC would be an imbalanced lat/lon pair where longitude is specified to seconds, but latitude is degrees only. Such a case is very much a false positive.
- MGRS date filtering for various common dates in recent time; for example 03 MAR 12
- Allowing for "dashes" as separators between lat & lon, where in some cases it may appear as a hemisphere sign.
- Introduction of Maven
Major improvements
- Support for asymmetric coordinates; valid lat/lon formats that have varying resolution, e.g., 34:45N 44:55:10W. valid MGRS grids with typos, e.g., 483QR 443 55 ( easting has 3 digits, northing should be "550"); Grids with line breaks or interrupting whitespace, 483QR 44\n3 55; Grids with no whitespace, but invalid easting/northing, eg. 483QR44355 -- both 443 / 550 and 440 / 355 are emitted as potentially correct interpretations
- Allow for multiple interpretations of a coordinate pattern, e.g., MGRS case above where easting/northing is ambiguous due to typos.
- Testing data included patterns from wikipedia geotagging conventions for wiki pages.
- Expanded test cases
- Improved date filtering for DMS patterns with odd punctuation, e.g. 2012-12-11 10:45:00 is a time not a coordinate. Two-digit year date/time patterns matched often.
- Consolidated patterns library primarily due to consideration of asymmetry in resolution. More smarts put into place in parsing/normalizing.
Minor improvements:
- added more DM/DMS patterns to ensure coordinates with DMS symbols only and no indication of hemisphere (-,+, ENSW) are detected and parsed
- added such tests to Truth data; see GeocoderEval project
Major improvements:
- Precision
- Test & Evaluation
- Rule order -- it is preserved and is currently MGRS UTM DMS DM DD
- Fixed many rules, added others
Details:
- Concepts added/adjusted (org.mitre.xcoord pkg):
- GeocoordPrecision -- tracks number of digits mainly in DD, DM, DMS patterns, and the associated precision.
- Hemisphere -- tracks hemisphere if it is +/-, Alpha or null.
- PrecisionScales -- much improved assessment of inherent precision in any patter *Testing added
- TestCase, TestScript added to capture input tests
- TestUtility improved to report truth data if given, and the carry both truth and test results along with output for evaluation later.
- UTMParser hemisphere detection fixed. "S" was allowed to be a Northern lat band, when it usually means South.
- DMSOrdinate now handles all logic for DD, DM, DMS parsing and calculation. Hemisphere parsing moved here (out of PatternManager)
- PatternManager --- ensure Rules order-of-appearance is preserved from configuration file
- GeocoordMatch -- formatting was a major concern, as with Java, printing a Float or Double allows near infinite decimal
places (I mean a lot). Where you want 35.01, Java might print "35.00999989082911237", but you want "35.01" the string.
- Use GeocoordMatch.formatLatitude() or formatLongitude() to get a printable string version of the calculated lat or lon