Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transition to the newest version of TUF #561

Open
wants to merge 93 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
60b8fa9
Update tuf, securesystemslib and cryptography deps
lukpueh Aug 23, 2024
f62907e
Add alternative TUF metadata repo implementation
lukpueh Aug 23, 2024
ad2b58f
Change create and add_keys API to take signers
lukpueh Aug 26, 2024
b46645b
Make sure targets is signed on add key
lukpueh Aug 26, 2024
33750eb
Assert keytype rsa in taf.tuf.keys helper
lukpueh Aug 27, 2024
42fbfac
Add signer implementation for Yubikeys
lukpueh Aug 28, 2024
25371d7
Comment out legacy imports (WIP)
lukpueh Aug 28, 2024
ec294a8
Merge branch 'feature/tuf-repositoty' into tuf-upgrade
renatav Oct 29, 2024
70927b7
Merge pull request #512 from lukpueh/tuf-upgrade
renatav Oct 29, 2024
9bf3fb9
feat: moved get threshold to the new repository class, implement dele…
renatav Oct 29, 2024
c7be3cb
refact: move get expiration date, get all targets and check expiratio…
renatav Oct 30, 2024
7601b9b
refact: move get role paths and all target files to the new repositor…
renatav Oct 30, 2024
b8d247e
refact: work on making the create repository method more flexible
renatav Oct 31, 2024
ade9af1
refact: extend creation of repositories using the new TUF, added supp…
renatav Oct 31, 2024
f1c1b6c
test: add create repository with delegations test
renatav Nov 1, 2024
ea05769
test: use repository created using create in tests
renatav Nov 1, 2024
c9857d0
test, refact: reorganize tests, implement keyid-roles mapping using t…
renatav Nov 1, 2024
b7eb34a
test: re-enabled add target test
renatav Nov 2, 2024
ccb94b8
refact: moved modify targets to the new repository class
renatav Nov 4, 2024
43af588
test: create test repos with target files and custom data
renatav Nov 5, 2024
feb7b53
refac, test: move get_all_target_files_state to the new repository cl…
renatav Nov 5, 2024
3202900
refact: move generate_roles_description to the new repository class
renatav Nov 5, 2024
af59138
test, refact: minor tests refactoring, reimplement is_valid_metadata_key
renatav Nov 6, 2024
51d63c9
refact: test: reimplement add metadata keys, enable keys tests
renatav Nov 6, 2024
e57b8e3
feat, test: implement revoke key
renatav Nov 7, 2024
abeaa2c
test, fix: minor add and revoke key improvements
renatav Nov 7, 2024
f3bc1f7
refact, test: initial work on reworking signing, add set expiration d…
renatav Nov 8, 2024
2681d4a
refact: remeve outdated imports
renatav Nov 8, 2024
0d67dd5
refact: check and set expiration date reimplemented
renatav Nov 8, 2024
ba7d3eb
refact: update key generation
renatav Nov 8, 2024
ea93127
refact: reimplement repository_at_revision
renatav Nov 9, 2024
4122100
refact: update updater and the creation of a new repository
renatav Nov 9, 2024
e21edfd
fix: bare repositories fix
renatav Nov 9, 2024
d9e5cc0
refact: reimplement addition of verification keys when creating a new…
renatav Nov 11, 2024
67fbc2b
fix: minor create repo fix
renatav Nov 11, 2024
c1cd853
refact: remove do_snapshot and timestamp from add/revoke keys
renatav Nov 11, 2024
7fe4d2f
refact: work on initializing repository and signers in api
renatav Nov 14, 2024
e77210b
fix, feat: fix add keys, add revoke key command
renatav Nov 14, 2024
947f1e4
test: add add delegated paths test
renatav Nov 14, 2024
5c56ede
refact: rework create new role
renatav Nov 15, 2024
c92c39b
refact: support adding multiple new roles
renatav Nov 15, 2024
f63a058
refact, fix: fix create delagations when no previouis delgations, rew…
renatav Nov 15, 2024
66a2742
refact: refactored remove paths
renatav Nov 16, 2024
2fe6d72
fix: minor fixes, update update-expiration-dates
renatav Nov 16, 2024
5ddb0c3
refact, test: remove unused code, remove paths test added
renatav Nov 16, 2024
dfbcde6
refact: rework targets update
renatav Nov 16, 2024
c541df8
test: add update targets roles test
renatav Nov 18, 2024
45c8c02
refact: reimplement add target repo
renatav Nov 19, 2024
4917b71
refact: rework remove target repo
renatav Nov 19, 2024
2f817d3
refact: update add/remove dependencies
renatav Nov 19, 2024
de7f5f8
test, refact: refact repository init tests, remove outdated tests
renatav Nov 19, 2024
96c7df1
test: work on refactoring test_create_repository tests
renatav Nov 20, 2024
7a787f4
test, refact: reorganize conftest
renatav Nov 20, 2024
1b7146b
test: rework test dependencies, metadata and roles api tests
renatav Nov 21, 2024
c5de6e1
chore: remove ramaining test repos
renatav Nov 21, 2024
739c014
test: refact test targets
renatav Nov 21, 2024
4fd29db
test, refact: update repositoriesdb tests, some refactoring, remove u…
renatav Nov 21, 2024
fbc0a8a
fix: fix snapshot info length/hash issue
renatav Nov 22, 2024
68fb96e
feat: initial implementation of git storage backed, which can load me…
renatav Nov 22, 2024
84a52f2
fix: bypass storage singleton
renatav Nov 22, 2024
8224179
fix: update root version number when updating snaphost. Work on updat…
renatav Nov 25, 2024
5afeaf5
test: update test_update_invalid
renatav Nov 26, 2024
6573a6a
fix: fix update expiration dates. Snapshot info was not getting updated
renatav Nov 27, 2024
959ab49
test: update remaining updater tests, minor cleanup
renatav Nov 27, 2024
16abe0c
chore: cleanup, formatting, remove unused code and imports
renatav Nov 27, 2024
4f02ce1
chore: bump yubikey-manager version
renatav Nov 27, 2024
62a8cac
chore: fixing mypy issues
renatav Nov 27, 2024
cb5b45a
chore: import and mypy issues
renatav Nov 27, 2024
2070426
chore: mypy issues
renatav Nov 27, 2024
106836c
chore: remove unused import
renatav Nov 27, 2024
896eb73
chore: comment out yubukey tests
renatav Nov 27, 2024
f262d81
test: remove generation of tests
renatav Nov 27, 2024
324e642
chore: formatting
renatav Nov 27, 2024
ea6dbf2
test: fix a number of failing tests
renatav Nov 28, 2024
32b2930
chore: remove unused import
renatav Nov 28, 2024
b37a631
test: rework keys tests
renatav Nov 28, 2024
d3d7263
test: update number of root keys in updater keys description
renatav Nov 29, 2024
3000095
fix: fix failing tests
renatav Nov 29, 2024
65c128b
refact: update yubikey cli functions
renatav Nov 29, 2024
4805128
test: refact yubikey tests
renatav Nov 30, 2024
789b591
refact: refactor creation of repositories using yubikeys
renatav Nov 30, 2024
ff146df
chore: formatting and mypy fixes
renatav Nov 30, 2024
e402196
test: add pytest-mock to test requirements
renatav Nov 30, 2024
de1bd82
Merge branch 'master' into feature/tuf-repositoty
renatav Nov 30, 2024
da5344d
chore: update changelog
renatav Nov 30, 2024
2f16959
chore: merge master
renatav Nov 30, 2024
f22da80
test: tun tests without having ykman installed
renatav Dec 4, 2024
7db3588
chore: ignore unused import
renatav Dec 4, 2024
6410f52
docs: GitStorageBackend docstring
renatav Dec 4, 2024
4ab8c48
feat: add a command for rotating keys
renatav Dec 9, 2024
dc431c5
test: add test revoke signing key test
renatav Dec 9, 2024
5ea21e7
docs: update docs related to repository classes
renatav Dec 10, 2024
bef7f8e
chore: merge master and resolve conflicts
renatav Dec 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 5 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,18 @@ and this project adheres to [Semantic Versioning][semver].

### Added

- Implement removal of keys [(561)]
- Implement full partial update. Store last validated commit per repo ([559)])

### Changed

- Transition to the newest version of TUF [(561)]

### Fixed


[559]: https://github.com/openlawlibrary/taf/pull/558
[561]: https://github.com/openlawlibrary/taf/pull/561
[559]: https://github.com/openlawlibrary/taf/pull/559


## [0.32.4]
Expand Down
10 changes: 4 additions & 6 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,13 +160,11 @@ To validate commits that could be decades old without being obstructed by expire

This module encapsulates the `GitRepository` class, a high-level abstraction over Git operations, designed to interface directly with Git repositories at the filesystem level. The `GitRepository` class serves as an intermediary, enabling programmatic access to Git actions including: creating branches, working with commits, and working with remotes. It leverages [`pygit2`](https://www.pygit2.org/) for some of the interactions with Git. Other interactions use direct shell command execution via subprocess for operations not covered by `pygit2` or where direct command invocation is preferred for efficiency or functionality reasons.

### `taf/repository_tool.py`
### `taf/tuf/repository`

Contains a `Repository` class, which is a wrapper around TUF's repository, making it simple to execute important updates, like
adding new signing keys, updating and signing metadata files and extracting information about roles, keys,
delegations and targets.

NOTE: Long-term plan is to rework this part of the codebase. This is necessary to transition to the newest version of TUF, since it is relying on parts which no longer exist in newer TUF.
Contains a `MetadataRepository` class, which is an implementation of TUF's `Repository` class for editing metadata.
It simplifies the execution of important updates such as adding new signing keys, updating and signing metadata
files, and extracting information about roles, keys, delegations, and targets.

### `taf/auth_repo.py`

Expand Down
74 changes: 34 additions & 40 deletions docs/developers/repository-classes.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# Repositories

As a tool focused on creation and secure update of Git repositories (authentication repositories and their
targets), TAF contains classes and functions which strive to make integration with Git and TUF as easy as possible.
`GitRepository` can be seen as a wrapper around git calls which make it possible to interact with the actual `Git`
repository located on the file system. E.g. to create a new branch, list commits, push to the remote etc.
On the other hand, `Repository` class contained by the `repository_tool` module can instantiate a TUF repository,
provided that the directory passed to it contains metadata files expected to be found in such a repository. It also
implements important TUF concepts, such as adding a new delegated role, determine which role is responsible for which
target file, add TUF targets etc. An authentication repository can be seen as a Git repository which is also a TUF repository - it
contains TUF's metadata and target files and a `.git` folder. TAF's `auth_repo` module's `AuthenticationRepository`
class follows that logic and is derived from the two previously mentioned base classes. Finally, `repositoriesdb`
is a module inspired by TUF's modules like `keysdb`, which deals with instantiation of repositories and stores the
created classes inside a "database" - a dictionary which maps authentication repositories and their commits
to lists of their target repositories at certain revisions.
As a tool focused on the creation and secure update of Git repositories (authentication repositories and their
targets), TAF contains classes and functions that strive to make integration with Git and TUF as simple as possible.
`GitRepository` acts as a wrapper around Git calls, enabling interaction with the actual `Git` repository on the file
system, e.g., creating a new branch, listing, creating, and pushing commits, etc. Conversely, the `MetadataRepository`
class in `tuf/repository.py` extends TUF's `Repository` class, an abstract class for metadata modifying implementations.
It provides implementations of crucial TUF concepts, such as adding a new delegated role, determining which role is
responsible for which target file, and adding TUF targets etc. An authentication repository can be seen as a Git
repository that is also a TUF repository. It contains TUF's metadata and target files and a `.git` folder. TAF's
`auth_repo` module's `AuthenticationRepository` class follows that logic and is derived from the two previously
mentioned base classes. Finally, `repositoriesdb` is a module inspired by TUF's modules like `keysdb`, which deals with
the instantiation of repositories and stores the created classes inside a "database" - a dictionary which maps
authentication repositories and their commits to lists of their target repositories at certain revisions. Note: the
concept of databases has been removed from TUF and removal of `repositoriesdb` is also planned in case of TAF.

## GitRepository

Expand Down Expand Up @@ -66,44 +66,38 @@ repo.commit_empty('An example message')
repo.push()
```

## Repository tool's `Repository`
## Implementation of TUF's `Repository` class (`tuf/repository/MetadataRepository`)

This class extends TUF's repository interface, providing features for executing metadata updates, such as
adding new signing keys, updating and signing metadata files, and extracting information about roles,
keys, delegations, and targets. It can be used to create a new TUF repository, retrieve information about
a TUF repository, or update its metadata files. TAF's implementation of the repository class follows the
convention of separating metadata and target files into directories named `metadata` and `target`:

This class can be seen as a wrapper around a TUF repository, making it simple to execute important updates, like
adding new signing keys, updating and signing metadata files and extracting information about roles, keys,
delegations and targets. It is instantiated by passing file system path which corresponds to a directory containing
all files and folders that a TUF repository expects. That means that `metadata` and `targets` folders have to exist
and that a valid `root.json` file needs to be found inside `metadata`. So:
```
- repo_root
- metadata
- root.json
- targets
```
Optionally, `name` attribute can also be specified during instantiation. It will be used to set name of the TUF's
repository instance. This value is set to `default` if not provided. If more than one repository is to be used
at the same time, it is important to set distinct names.

TUF repository is instantiated lazily the first time it is needed. This object is not meant to be used directly.
The main purpose of TAF's repository class is to group operations which enable valid update of TUF metadata and acquiring
information like can a key be used to sign a certain metadata file or finding roles that are linked with
the provided public key. To set up a new repository or add a new signing key, it is recommended to use the
`developer_tool` module since it contains full implementations of these complex functionalities. Functionalities
like updating targets and signing metadata or updating a metadata's expiration date are fully covered by repository
class's methods and can be used directly. These include:
- `update_timestamp_keystores`, `update_snapshot_keystores` (`update_rolename_keystores`) and `update_role_keystores` (for delegated roles)
-`update_timestamp_yubikeys`, `update_snapshot_yubikeys` (`update_rolename_yubikeys`) and `update_role_yubikeys` (for delegated roles)

If `added_targets_data` or `removed_targets_data` is passed in when calling these methods (only applicable to
`targets` and delegated target roles), information about target files will be updated and the corresponding metadata
file will be signed. Its expiration date will be updated too. If there is targets data or if the called method
corresponds to a non-targets role, the metadata file's expiration will still be updated and the file will be signed.

It is instantiated by providing the repository's path. Unlike the previous implementation, which was based on an
older version of TUF, this repository does not have, nor does it need, a name. The class can be instantiated
regardless of whether there are `metadata` files located at `path/metadata`. In fact, it is possible to read the
metadata and target files from mediums other than the local file system. TUF enables such flexibility by allowing
custom implementations of the `StorageBackendInterface`. These implementations can redefine how metadata and target
files are read and written. To instantiate a `MetadataRepository` class with a custom storage interface, use the
`storage` keyword argument. If not specified, TUF's default `FilesystemBackend` will be used.

This class is used extensively to implement API functions.


## `AuthenticationRepository`

This class is derived from both `GitRepository` and TAF's `Repository`. Authentication repositories are expected
to contain TUF metadata and target files, but are also Git repositories. It is important to note that only files
inside the `targets` folder are tracked and secured by TUF.
This class is derived from `GitRepository`, and indirectly from `MetadataRepository`. Authentication repositories are
expected to contain TUF metadata and target files, but are also Git repositories. It is important to note that only
files inside the `targets` folder are tracked and secured by TUF.


Instances of the `AuthenticationRepository` are created by passing the same arguments as to `GitRepository` (`library_dir`, `name`, `urls`, `custom`, `default_branch`, `allow_unsafe` and `path` which can replace `library_dir` and `name` combination), as well as some optional additional arguments:
- `conf_directory_root` - path to the directory where the `last_validated_commit` will be stored.
Expand Down
57 changes: 0 additions & 57 deletions docs/testing/testing_notes.md

This file was deleted.

11 changes: 6 additions & 5 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,10 @@
"freezegun==0.3.15",
"jsonschema==3.2.0",
"jinja2==3.1.*",
"pytest-mock==3.14.*",
]

yubikey_require = ["yubikey-manager==5.1.*"]
yubikey_require = ["yubikey-manager==5.5.*"]


kwargs = {
Expand All @@ -54,13 +55,13 @@
"cattrs>=23.1.2",
"click==8.*",
"colorama>=0.3.9",
"oll-tuf==0.20.0.dev2",
"cryptography==38.0.*",
"securesystemslib==0.25.*",
"tuf==5.*",
"cryptography==43.0.*",
"securesystemslib==1.*",
"loguru==0.7.*",
'pygit2==1.9.*; python_version < "3.11"',
'pygit2==1.14.*; python_version >= "3.11"',
"pyOpenSSL==22.1.*",
"pyOpenSSL==24.2.*",
"logdecorator==2.*",
],
"extras_require": {
Expand Down
96 changes: 96 additions & 0 deletions taf/api/api_workflow.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
from contextlib import contextmanager
from pathlib import Path
from typing import Dict, List, Optional, Union

from taf.api.utils._conf import find_keystore
from taf.auth_repo import AuthenticationRepository
from taf.constants import DEFAULT_RSA_SIGNATURE_SCHEME
from taf.exceptions import PushFailedError, TAFError
from taf.keys import load_signers
from taf.log import taf_logger
from taf.messages import git_commit_message
from taf.constants import METADATA_DIRECTORY_NAME


@contextmanager
def transactional_execution(auth_repo):
initial_commit = auth_repo.head_commit_sha()
try:
yield
except PushFailedError:
pass
except Exception:
auth_repo.reset_to_commit(initial_commit, hard=True)
raise


@contextmanager
def manage_repo_and_signers(
auth_repo: AuthenticationRepository,
roles: Optional[List[str]] = None,
keystore: Optional[Union[str, Path]] = None,
scheme: Optional[str] = DEFAULT_RSA_SIGNATURE_SCHEME,
prompt_for_keys: Optional[bool] = False,
paths_to_reset_on_error: Optional[List[Union[str, Path]]] = None,
load_roles: Optional[bool] = True,
load_parents: Optional[bool] = False,
load_snapshot_and_timestamp: Optional[bool] = True,
commit: Optional[bool] = True,
push: Optional[bool] = True,
commit_key: Optional[str] = None,
commit_msg: Optional[str] = None,
no_commit_warning: Optional[bool] = True,
):
try:
roles_to_load = set()
if roles:
unique_roles = set(roles)
if load_roles:
roles_to_load.update(unique_roles)
if load_parents:
roles_to_load.update(auth_repo.find_parents_of_roles(unique_roles))
if load_snapshot_and_timestamp:
roles_to_load.add("snapshot")
roles_to_load.add("timestamp")
if roles_to_load:
if not keystore:
keystore_path = find_keystore(auth_repo.path)
else:
keystore_path = Path(keystore)
loaded_yubikeys: Dict = {}
for role in roles_to_load:
if not auth_repo.check_if_keys_loaded(role):
keystore_signers, yubikey_signers = load_signers(
auth_repo,
role,
loaded_yubikeys=loaded_yubikeys,
keystore=keystore_path,
scheme=scheme,
prompt_for_keys=prompt_for_keys,
)
auth_repo.add_signers_to_cache({role: keystore_signers})
auth_repo.add_signers_to_cache({role: yubikey_signers})
yield
if auth_repo.something_to_commit() and commit:
if not commit_msg and commit_key:
commit_msg = git_commit_message(commit_key)
auth_repo.commit_and_push(commit_msg=commit_msg, push=push)
elif not no_commit_warning:
taf_logger.log("NOTICE", "\nPlease commit manually\n")

except PushFailedError:
raise
except Exception as e:
taf_logger.error(f"An error occurred: {e}")
if not paths_to_reset_on_error:
paths_to_reset_on_error = [METADATA_DIRECTORY_NAME]
elif METADATA_DIRECTORY_NAME not in paths_to_reset_on_error:
paths_to_reset_on_error.append(METADATA_DIRECTORY_NAME)

if auth_repo.is_git_repository and paths_to_reset_on_error:
# restore metadata, leave targets as they might have been modified by the user
# TODO flag for also resetting targets?
# also update the CLI error handling]
auth_repo.restore([str(path) for path in paths_to_reset_on_error])

raise TAFError from e
Loading
Loading