Skip to content

Commit e191e0e

Browse files
rverdilejlsherrill
andauthored
Add rpm name search and integration testing (#1)
* Add rpm name search and integration testing * update config.yaml.example fix query add limit add close method add new test cases add mock * limit whole query --------- Co-authored-by: Justin Sherrill <[email protected]>
1 parent e809983 commit e191e0e

27 files changed

+1498
-5
lines changed

.github/workflows/tang-actions.yaml

+50-4
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,59 @@ on:
99
paths-ignore:
1010
- '**.md'
1111
jobs:
12-
govet:
13-
name: Vet
12+
golangci:
13+
name: Lint
1414
runs-on: ubuntu-latest
1515
steps:
1616
- uses: actions/checkout@v2
1717
- uses: actions/setup-go@v2
1818
with:
1919
go-version: "1.20"
20-
- run: |
21-
go vet ./...
20+
- name: golangci-lint
21+
uses: golangci/golangci-lint-action@v2
22+
with:
23+
version: v1.54.2
24+
skip-go-installation: true
25+
args: --timeout=5m
26+
gotest:
27+
name: Test
28+
runs-on: ubuntu-latest
29+
services:
30+
postgres:
31+
image: postgres:15
32+
env:
33+
POSTGRES_PASSWORD: postgres
34+
options: >-
35+
--health-cmd pg_isready
36+
--health-interval 10s
37+
--health-timeout 5s
38+
--health-retries 5
39+
ports:
40+
- 5432:5432
41+
steps:
42+
- uses: actions/checkout@v2
43+
- uses: actions/setup-go@v2
44+
with:
45+
go-version: "1.20"
46+
- name: start pulp
47+
uses: isbang/[email protected]
48+
with:
49+
compose-file: ./compose_files/pulp/docker-compose.yml
50+
down-flags: --volumes
51+
- name: Wait for pulp
52+
run: |
53+
docker run --network=host --rm -v ${PWD}:/local curlimages/curl \
54+
curl --retry-all-errors --fail --retry-delay 10 --retry 32 --retry-max-time 240 http://localhost:8087/pulp/default/api/v3/repositories/rpm/rpm/ -u admin:password
55+
sleep 30
56+
- name: integration tests
57+
run: |
58+
make test-integration
59+
env:
60+
DATABASE_HOST: localhost
61+
DATABASE_PORT: 5434
62+
DATABASE_USER: pulp
63+
DATABASE_NAME: pulp
64+
DATABASE_PASSWORD: password
65+
SERVER_URL: http://localhost:8087
66+
SERVER_USERNAME: admin
67+
SERVER_PASSWORD: password

.gitignore

+4-1
Original file line numberDiff line numberDiff line change
@@ -29,4 +29,7 @@ release/
2929
Containerfile
3030

3131
# Pulp Oci Images
32-
compose_files/pulp/pulp-oci-images
32+
compose_files/pulp/pulp-oci-images
33+
34+
#config
35+
configs/config.yaml

.golangci.yaml

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Configuration for golangci-lint. See https://golangci-lint.run/usage/configuration/.
2+
linters:
3+
disable-all: false # use default linters
4+
enable:
5+
- gofmt
6+
- whitespace
7+
- govet
8+
- misspell
9+
- forcetypeassert
10+
- gci
11+
- bodyclose
12+
issues:
13+
exclude:
14+
- composite

Makefile

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
##
2+
# Entrypoint for the Makefile
3+
#
4+
# It is composed at mk/includes.mk by including
5+
# small make files which provides all the necessary
6+
# rules.
7+
#
8+
# Some considerations:
9+
#
10+
# - Variables customization can be
11+
# stored at 'config.env', 'mk/private.mk' files.
12+
# - By default the 'help' rule is executed.
13+
# - No parallel jobs are executed from the main Makefile,
14+
# so that multiple rules from the command line will be
15+
# executed in serial.
16+
##
17+
18+
include mk/includes.mk
19+
20+
.NOT_PARALLEL:
21+
22+
# Set the default rule
23+
.DEFAULT_GOAL := help

README.md

+62
Original file line numberDiff line numberDiff line change
@@ -1 +1,63 @@
11
# tang
2+
3+
The tangy package provides methods to read from a [pulp](https://pulpproject.org/) database.
4+
5+
## Installation
6+
`go get github.com/content-services/tang`
7+
8+
## Usage
9+
The tangy package is meant to be imported into an existing project that is using pulp. It can be used like this:
10+
```go
11+
// Pulp database configuration information
12+
dbConfig := tangy.Database{
13+
Name: "pulp",
14+
Host: "localhost",
15+
Port: 5434,
16+
User: "pulp",
17+
Password: "password",
18+
CACertPath: "",
19+
PoolLimit: 20,
20+
}
21+
22+
// Create new Tangy instance using database config
23+
t, err := tangy.New(dbConfig, tangy.Logger{Enabled: false})
24+
if err != nil {
25+
return err
26+
}
27+
defer t.Close()
28+
29+
// Use Tangy to search for RPMs, by name, that are associated to a specific repository version, returning up to the first 100 results
30+
versionHref := "/pulp/e1c6bee3/api/v3/repositories/rpm/rpm/018c1c95-4281-76eb-b277-842cbad524f4/versions/1/"
31+
rows, err := t.RpmRepositoryVersionPackageSearch(context.Background(), []string{versionHref}, "ninja", 100)
32+
if err != nil {
33+
return err
34+
}
35+
```
36+
See example.go for a complete example.
37+
38+
## Developing
39+
To develop for tangy, there are a few more things to know.
40+
41+
### Create your configuration
42+
`$ cp ./configs/config.yaml.example ./configs/config.yaml`
43+
44+
### Connecting to pulp
45+
46+
#### Connect to an existing pulp server
47+
To connect to an existing pulp server, put the corresponding connection information in `configs/config.yaml`.
48+
49+
#### Create a new pulp server
50+
To create a new pulp server, you can use the provided make commands. You will need to have podman & podman-compose (or docker) installed.
51+
The default values provided in config.yaml.example will work with this server.
52+
53+
##### Start containers
54+
`make compose-up`
55+
56+
#### Stop containers
57+
`make compose-down`
58+
59+
#### Clean container volumes
60+
`make compose-clean`
61+
62+
### Mocking
63+
Tangy also exports a mock interface you can regenerate using the [mockery](https://github.com/vektra/mockery) tool.
+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#!/bin/bash
2+
# This logic enables us to have multiple servers, and check to see
3+
# if they are scaled every 10 seconds.
4+
# https://serverfault.com/a/821625/189494
5+
# https://www.nginx.com/blog/dns-service-discovery-nginx-plus#domain-name-variable
6+
7+
set -e
8+
9+
if [ "$container" != "podman" ]; then
10+
# the nameserver list under podman is unreliable.
11+
# It will look like "10.89.1.1 192.168.1.1 192.168.1.1", but only the 1st IP works.
12+
# This doesn't mess up `nslookup`, but it messes up `getent hosts` and nginx.
13+
export NAMESERVER=`cat /etc/resolv.conf | grep "nameserver" | awk '{print $2}' | head -n1`
14+
else
15+
export NAMESERVER=`cat /etc/resolv.conf | grep "nameserver" | awk '{print $2}' | tr '\n' ' '`
16+
fi
17+
18+
echo "Nameserver is: $NAMESERVER"
19+
20+
echo "Generating nginx config"
21+
envsubst '$NAMESERVER' < /etc/opt/rh/rh-nginx116/nginx/nginx.conf.template > /etc/opt/rh/rh-nginx116/nginx/nginx.conf
22+
23+
# We cannot use upstream server groups with a DNS resolver without nginx plus
24+
# So we modifying the files to use the variables rather than the upstream server groups
25+
for file in /opt/app-root/etc/nginx.default.d/*.conf ; do
26+
echo "Modifying $file"
27+
sed -i 's/pulp-api/$pulp_api:24817/' $file
28+
sed -i 's/pulp-content/$pulp_content:24816/' $file
29+
done
30+
31+
echo "Starting nginx"
32+
exec nginx -g "daemon off;"
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
DNmNdwgyZugTax9S64J0FITTr9IHPxbuoF1F1CGPr68=
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
error_log /dev/stdout info;
2+
worker_processes 1;
3+
events {
4+
worker_connections 1024; # increase if you have lots of clients
5+
accept_mutex off; # set to 'on' if nginx worker_processes > 1
6+
}
7+
8+
http {
9+
access_log /dev/stdout;
10+
include mime.types;
11+
# fallback in case we can't determine a type
12+
default_type application/octet-stream;
13+
sendfile on;
14+
15+
# If left at the default of 1024, nginx emits a warning about being unable
16+
# to build optimal hash types.
17+
types_hash_max_size 4096;
18+
19+
server {
20+
# This logic enables us to have multiple servers, and check to see
21+
# if they are scaled every 10 seconds.
22+
# https://www.nginx.com/blog/dns-service-discovery-nginx-plus#domain-name-variable
23+
# https://serverfault.com/a/821625/189494
24+
resolver $NAMESERVER valid=10s;
25+
set $pulp_api pulp_api;
26+
set $pulp_content pulp_content;
27+
28+
# Gunicorn docs suggest the use of the "deferred" directive on Linux.
29+
listen 8080 default_server deferred;
30+
listen [::]:8080 default_server deferred;
31+
32+
# If you have a domain name, this is where to add it
33+
server_name $hostname;
34+
35+
# The default client_max_body_size is 1m. Clients uploading
36+
# files larger than this will need to chunk said files.
37+
client_max_body_size 10m;
38+
39+
# Gunicorn docs suggest this value.
40+
keepalive_timeout 5;
41+
42+
# static files that can change dynamically, or are needed for TLS
43+
# purposes are served through the webserver.
44+
root /opt/app-root/src;
45+
46+
location /pulp/content/ {
47+
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
48+
proxy_set_header X-Forwarded-Proto $scheme;
49+
proxy_set_header Host $http_host;
50+
# we don't want nginx trying to do something clever with
51+
# redirects, we set the Host: header above already.
52+
proxy_redirect off;
53+
proxy_pass http://$pulp_content:24816;
54+
}
55+
56+
location /pulp/api/v3/ {
57+
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
58+
proxy_set_header X-Forwarded-Proto $scheme;
59+
proxy_set_header Host $http_host;
60+
# we don't want nginx trying to do something clever with
61+
# redirects, we set the Host: header above already.
62+
proxy_redirect off;
63+
proxy_pass http://$pulp_api:24817;
64+
}
65+
66+
location /auth/login/ {
67+
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
68+
proxy_set_header X-Forwarded-Proto $scheme;
69+
proxy_set_header Host $http_host;
70+
# we don't want nginx trying to do something clever with
71+
# redirects, we set the Host: header above already.
72+
proxy_redirect off;
73+
proxy_pass http://$pulp_api:24817;
74+
}
75+
76+
include /opt/app-root/etc/nginx.default.d/*.conf;
77+
78+
location / {
79+
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
80+
proxy_set_header X-Forwarded-Proto $scheme;
81+
proxy_set_header Host $http_host;
82+
# we don't want nginx trying to do something clever with
83+
# redirects, we set the Host: header above already.
84+
proxy_redirect off;
85+
proxy_pass http://$pulp_api:24817;
86+
# static files are served through whitenoise - http://whitenoise.evans.io/en/stable/
87+
}
88+
}
89+
}
+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
root:x:0:0:root:/root:/bin/bash
2+
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
3+
bin:x:2:2:bin:/bin:/usr/sbin/nologin
4+
sys:x:3:3:sys:/dev:/usr/sbin/nologin
5+
sync:x:4:65534:sync:/bin:/bin/sync
6+
games:x:5:60:games:/usr/games:/usr/sbin/nologin
7+
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
8+
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
9+
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
10+
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
11+
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
12+
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
13+
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
14+
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
15+
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
16+
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
17+
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
18+
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
19+
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
20+
postgres:x:26:26::/var/lib/postgresql:/bin/bash

compose_files/pulp/assets/settings.py

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
SECRET_KEY = "aabbcc"
2+
CONTENT_ORIGIN = "http://pulp_content:24816"
3+
DATABASES = {"default": {"HOST": "postgres", "ENGINE": "django.db.backends.postgresql", "NAME": "pulp", "USER": "pulp", "PASSWORD": "password", "PORT": "5432", "CONN_MAX_AGE": 0, "OPTIONS": {"sslmode": "prefer"}}}
4+
CACHE_ENABLED = True
5+
REDIS_HOST = "redis"
6+
REDIS_PORT = 6379
7+
REDIS_PASSWORD = ""
8+
ANSIBLE_API_HOSTNAME = "http://pulp_api:24817"
9+
ANSIBLE_CONTENT_HOSTNAME = "http://pulp_content:24816/pulp/content"
10+
ALLOWED_IMPORT_PATHS = ["/tmp"]
11+
ALLOWED_EXPORT_PATHS = ["/tmp"]
12+
TOKEN_SERVER = "http://pulp_api:24817/token/"
13+
TOKEN_AUTH_DISABLED = False
14+
TOKEN_SIGNATURE_ALGORITHM = "ES256"
15+
PUBLIC_KEY_PATH = "/etc/pulp/keys/container_auth_public_key.pem"
16+
PRIVATE_KEY_PATH = "/etc/pulp/keys/container_auth_private_key.pem"
17+
TELEMETRY = False
18+
STATIC_ROOT = "/var/lib/operator/static/"

0 commit comments

Comments
 (0)