proposal: v2 Search endpoint and generalized entity model

[adamdecaf]

With the recent additions of entity lists from around the world we've started to notice there are many overlapping fields (often named slightly differently) and the current /search endpoint has grown well beyond its initial scope. The current endpoint returns many key/array-of-objects results which are hard to parse though.

This proposal is two fold.

1. Create a generalized model for entities
2. Create a /v2/search endpoint which accepts generalized field queries to search across all lists

Generalized model for enties

The idea of a generalized model would be to offer a shared high-level search across all supported lists and shared fields for every entity supported by Watchman.

Initially the modeling could look like the following:

type Entity[T any] struct { // TODO: list all of the shared fields
    Name string `json:"name"`
    Type EntityType `json:"entityType"`
    Source SourceList `json:"sourceList"`

    Addresses []Address `json:"addresses"` // TODO: define 

    SourceData T `json:"sourceData"` // Contains all original list data with source list naming
}

type EntityType string
var (
    EntityPerson EntityType = "person"
    EntityBusiness EntityType = "business"
    EntityAircraft EntityType = "aircraft"
    EntityVessel EntityType = "vessel"
)

type SourceList string
var (
    SourceOFAC SourceList = "ofac"
    SourceUSCSL SourceList = "us_csl"
    SourceUKCSL SourceList = "uk_csl"
)

Q: Should we adopt the OpenSanctions modeling?

v2 Search Endpoint

Create a new /v2/search endpoint which accepts all of the generalized entity fields as parameters and searches across all supported lists. Results are returned in their generalized form with a sub-object of their source list data. (In the original naming of the source list.)

GET /v2/search?name=...&country=....&type=individual

{
  "entities": [
    {
      "name": "John Doe",
      "entityType": "person",
      "sourceList": "us_csl",
      "addresses": [
        {
          "address1": "..",
          "country": ".."
        }
      ],
      "sourceData": {
        "name": "John Doe",
        "sdnType": "individual",
        "addresses": [
          {
            "address1": "..",
            "country": ".."
          }
        ],
      },
      "match": 0.754213444
    }
  ]
}

Pros: One stop source for search with a basic interface across all lists.

Cons: Dynamic types can be annoying to deal with in some languages.

Related Issues

#525, #441, #443, #444, #445, #353, #352