Merge pull request #3 from Mbanq/develop

Release new version

.github/workflows/docker-push-develop.yml

@@ -0,0 +1,22 @@
+name: Build and Push to GHCR.IO
+
+on:
+  push:
+    branches:
+    - develop
+
+jobs:
+  build:
+    name: Build and Push
+    runs-on: ubuntu-latest
+    steps:
+      - name: Build:checkout
+        uses: actions/checkout@v2
+      - name: Build:dockerimage
+        uses: docker/build-push-action@v1
+        with:
+          registry: ghcr.io
+          username: "sophanith2020"
+          password: ${{ secrets.GHCIO }}
+          repository: mbanq/watchman
+          tags: dev-latest

.gitignore

@@ -17,6 +17,7 @@ coverage.txt
 
 /bin/
 /lint-project.sh
+/gitleaks.tar.gz
 misspell*
 staticcheck*
 
@@ -28,4 +29,6 @@ openapi-generator*jar
 *.db
 
 webui/build/
-webui/node_modules/
+webui/node_modules/
+
+makefile.dev

.gitleaksignore

@@ -0,0 +1 @@
+client/api/openapi.yaml:generic-api-key:2923

AUTHORS

@@ -1,10 +1,18 @@
 # This file lists all individuals having contributed content to the repository.
 # For how it is generated, see `make AUTHORS`.
 
+Adam Pugh <[email protected]>
 Adam Shannon <[email protected]>
 Brooke Kline <[email protected]>
-Daniel Tonks <[email protected]>
+Chris Kelly <[email protected]>
+ckelly-digicert <[email protected]>
+Daniel Tonks <[email protected]>
 Franklin Henderson <[email protected]>
+Gregory Oschwald <[email protected]>
+Hamp Goodwin <[email protected]>
+Logan Hendricks <[email protected]>
+Nathan Lakritz <[email protected]>
 Ray Johnson <[email protected]>
-Renovate Bot <[email protected]>
+rayjlinden <[email protected]>
+Vincent Xiao <[email protected]>
 Wade Arnold <[email protected]>

CHANGELOG.md

@@ -1,3 +1,141 @@
+## v0.27.0 (Released 2023-12-14)
+
+This release of Watchman includes additional improvements to the search match scores to [reduce false positives and increase true positive matches](https://github.com/moov-io/watchman/pull/524#issue-2031927107). A few of the specific improvements are:
+
+1. Compare tokens in the search to the index tokens
+   - i.e. "find matches for every search token" rather than "find match for every indexed token"
+   - Improves scores of searches that don't include "middle" names
+   - Prevents sanctioned names that are 1 word (HADI, EMMA, KAMILA) matching long searches
+   - Has a side-effect that short search terms will have more false positives. I think this is a good trade off as the sanction lists will always contain the full name, but the search might not
+2. Once a token has matched something, it can't match a different token
+   - This prevents names with repeated words having artificially high scores
+   - e.g. prevents any search containing "Vladimir" matching "VLADIMIROV, Vladimir Vladimirovich"
+3. Weights each word-score by the length of the word, relative to the search and indexed name
+   - This corrects for error that is introduced by splitting names into tokens and doing piecewise Jaro-Winkler scoring
+   - Combing word-scores using a simple average gives short words (like Li, Al) equal weight to much longer words
+   - The length-weighted scores are comparable to what you get by doing whole-name to whole-name Jaro-Winkler comparison
+4. Punishes word-scores when the matching tokens have significantly different length
+5. Punishes word-scores when the matching tokens start with different letters
+
+## v0.26.1 (Released 2023-11-20)
+
+This release of Watchman has removed Company/Customer models and Watches. They've been deprecated for a while and do not perform as users expect. Stay tuned for a future Moov OSS project integrating with Watchman for sanctions screening.
+
+IMPROVEMENTS
+
+- feat: return matchedName in non-OFAC results
+- search: apply more edge case logic to decrease bad scoring
+- search: return matchedName for OFAC SDNs, Alts, and DPL records
+- test: remove duplicate (and skipped) UK/EU CSL tests
+
+## v0.25.0 (Released 2023-11-15)
+
+This release of Watchman lowers most match percentages by comparing names better.
+
+IMPROVEMENTS
+
+- fix: close xml encoder
+- fix: panic cleanup from newer linter rules
+- cmd/server: only check adjacent terms for local jaro max score
+- cmd/server: read ADJACENT_SIMILARITY_POSITIONS env var
+- cmd/server: weight term score by length similarity
+
+BUILD
+
+- build: bump numerous javascript dependencies
+- build: update to Go 1.21
+- build: update to node 20
+- build: update Debian, Fedora, node, and Go base images
+
+## v0.24.2 (Released 2023-04-03)
+
+IMPROVEMENTS
+
+- fix: keep numbers during stopwords step
+- fix: stop setting level=error for info logs
+
+BUILD
+
+- bump golang.org/x/crypto to v0.6.0
+- bump golang.org/x/net from 0.6.0 to 0.7.0
+- build(deps): bump activesupport from 6.1.7.2 to 7.0.4.3 in /docs
+- build(deps): bump webpack from 5.75.0 to 5.76.1 in /webui
+
+## v0.24.1 (Released 2023-02-16)
+
+IMPROVEMENTS
+
+- fix: filter SDNs in async search rather than truncate
+
+BUILD
+
+- build: update Go dependencies
+- docs: bundle update
+- webui: npm audit fix
+
+## v0.24.0 (Released 2023-02-02)
+
+ADDITIONS
+
+- search: Add the EU Consolidated Screening List
+- search: Add the UK Consolidated Screening List
+
+IMPROVEMENTS
+
+- feat: log status after download
+- fix: guard around race condition in pkg/download
+- fix: cap match percentage
+
+BUILD
+
+- build: upgrade golang to 1.20
+- build: try using hashicorp-forge/go-test-split-action to speedup tests
+- build(deps): bump activesupport from 6.0.3.4 to 6.0.6.1 in /docs
+- build(deps): bump json5 from 1.0.1 to 1.0.2 in /webui
+- build(deps): bump loader-utils from 2.0.0 to 2.0.4 in /webui
+- build(deps): bump nokogiri from 1.13.6 to 1.13.9 in /docs
+
+## v0.23.1 (Released 2022-10-17)
+
+IMPROVEMENTS
+
+- api,client: add 'match' to CSL results
+- webui: update country list
+
+BUILD
+
+- build: require go1.19.1 in CI/CD
+- build: upgrade golang.org/x/text to v0.3.8
+
+## v0.23.0 (Released 2022-09-08)
+
+Watchman v0.23.0 adds the US Consolidated Screening List results to search queries. A new `/search/us-csl` endpoint has also been added to only search the CSL. See the [API docs for full details](https://moov-io.github.io/watchman/api/#get-/search/us-csl).
+
+Lists added in this release:
+- Capta List (CAP)
+- Foreign Sanctions Evaders (FSE)
+- ITAR Debarred (DTC)
+- Military End User (MEU) List
+- Non-SDN Chinese Military-Industrial Complex Companies List (CMIC)
+- Non-SDN Menu-Based Sanctions List (NS-MBS List)
+- Nonproliferation Sanctions (ISN)
+- Palestinian Legislative Council List (PLC)
+- Unverified List (UVL)
+
+IMPROVEMENTS
+
+- download: only return files which are found or downloaded, not entire directory
+- feat: add GET /search/us-csl endpoint
+- fix: refer to `LOG_LEVEL` env var for when to log about migrations
+
+BUILD
+
+- build: upgrade to Go 1.19
+- build: replace deprecated ioutil functions
+- build(deps): bump terser from 5.13.1 to 5.14.2 in /webui
+- chore(deps): update dependency tzinfo to v1.2.10 [security]
+- fix(deps): update module github.com/moov-io/base to v0.34.1
+
 ## v0.22.0 (Released 2022-07-21)
 
 Watchman v0.22.0 has started to use Go 1.18 and its support for generic programming has helped us greatly clean up our code. We've also refreshed our [documentation website](https://moov-io.github.io/watchman/).

CONTRIBUTING.md

@@ -24,7 +24,7 @@ We use GitHub to manage reviews of pull requests.
 
 Before starting please make sure you have Go setup and can build our projects from source.
 
-This project uses [Go Modules](https://github.com/golang/go/wiki/Modules) and uses Go 1.14 or higher. See [Golang's install instructions](https://golang.org/doc/install) for help setting up Go. You can download the source code and we offer [tagged and released versions](https://github.com/moov-io/watchman/releases/latest) as well. We highly recommend you use a tagged release for production.
+This project uses [Go Modules](https://go.dev/blog/using-go-modules) and Go v1.18 or newer. See [Golang's install instructions](https://golang.org/doc/install) for help setting up Go. You can download the source code and we offer [tagged and released versions](https://github.com/moov-io/watchman/releases/latest) as well. We highly recommend you use a tagged release for production.
 
 ```
 # Just pull down into the Go Module's cache, not the source code.

Dockerfile

@@ -1,18 +1,18 @@
-FROM golang:1.18-buster as backend
+FROM golang:1.21-bookworm as backend
 WORKDIR /go/src/github.com/moov-io/watchman
 RUN apt-get update && apt-get upgrade -y && apt-get install make gcc g++
 COPY . .
 RUN go mod download
 RUN make build-server
 
-FROM node:18-buster as frontend
+FROM node:21-bookworm as frontend
 COPY webui/ /watchman/
 WORKDIR /watchman/
 RUN npm install --legacy-peer-deps
 RUN npm run build
 
 FROM debian:stable-slim
-LABEL maintainer="Moov <support@moov.io>"
+LABEL maintainer="Moov <oss@moov.io>"
 
 RUN apt-get update && apt-get upgrade -y && apt-get install -y ca-certificates
 COPY --from=backend /go/src/github.com/moov-io/watchman/bin/server /bin/server
@@ -22,6 +22,6 @@ ENV WEB_ROOT=/watchman/
 
 # USER moov # TODO(adam): non-root users
 
-EXPOSE 8080
-EXPOSE 9090
+EXPOSE 8084
+EXPOSE 9094
 ENTRYPOINT ["/bin/server"]

Dockerfile-openshift

@@ -1,20 +1,20 @@
-FROM quay.io/fedora/fedora:37-x86_64 as builder
+FROM quay.io/fedora/fedora:40-x86_64 as builder
 RUN yum install -y git golang make npm wget glibc
 WORKDIR /opt/app-root/src/
 COPY . .
 RUN make build
 
-FROM node:18-buster as frontend
+FROM node:21-bookworm as frontend
 COPY webui/ /watchman/
 WORKDIR /watchman/
 RUN npm install --legacy-peer-deps
 RUN npm run build
 
-FROM quay.io/fedora/fedora:37-x86_64
+FROM quay.io/fedora/fedora:40-x86_64
 RUN yum install -y glibc
 
 ARG VERSION=unknown
-LABEL maintainer="Moov <support@moov.io>"
+LABEL maintainer="Moov <oss@moov.io>"
 LABEL name="watchman"
 LABEL version=$VERSION