Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Field-level Redaction #1122

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Field-level Redaction #1122

wants to merge 17 commits into from
+1,823 −68

Conversation

ermontross
Copy link
Contributor

@ermontross ermontross commented Oct 14, 2024 •
edited
Loading

Summary

Add the ability to redact field values based on a field name for a specified time range via the manager API. This implemented is heavily based on OpenSearch's field masking.

The admin specifies a time range and a field name to redact. It redacts on read at the Lucene level by overwriting the reader. Currently, replaces the value of the field specified with REDACTED.

Left to-dos for the future

  • hash the value instead of replacing with REDACTED
  • auto-delete redactions whose time range is outside of the data's retention policy
  • possible: redact queries as well as reads

Requirements

@ermontross ermontross force-pushed the emontross/redaction branch 2 times, most recently from df00ed0 to 58e13f7 Compare October 17, 2024 18:24
@ermontross ermontross changed the title [DRAFT] Field-level Redaction Field-level Redaction Oct 17, 2024
bryanlb
bryanlb previously requested changes Oct 21, 2024
View reviewed changes
astra/src/main/java/com/slack/astra/chunk/ReadOnlyChunkImpl.java Outdated Show resolved Hide resolved
astra/src/main/java/com/slack/astra/logstore/search/fieldRedaction/RedactedFieldReader.java Show resolved Hide resolved
...main/java/com/slack/astra/logstore/search/fieldRedaction/RedactionFilterDirectoryReader.java Outdated Show resolved Hide resolved
astra/src/main/java/com/slack/astra/logstore/search/fieldRedaction/RedactionLeafReader.java Show resolved Hide resolved
...rc/main/java/com/slack/astra/logstore/search/fieldRedaction/RedactionStoredFieldVisitor.java Show resolved Hide resolved
.../src/main/java/com/slack/astra/logstore/search/fieldRedaction/RedactionSubReaderWrapper.java Show resolved Hide resolved
astra/src/main/java/com/slack/astra/metadata/fieldredaction/FieldRedactionMetadataStore.java Outdated Show resolved Hide resolved
astra/src/test/java/com/slack/astra/logstore/schema/FieldConflictStrategyTests.java Show resolved Hide resolved
astra/src/test/java/com/slack/astra/util/TestingFieldRedactionMetadataStore.java Outdated Show resolved Hide resolved
kyle-sammons
kyle-sammons previously approved these changes Oct 22, 2024
View reviewed changes
@bryanlb bryanlb dismissed their stale review November 6, 2024 20:23

@kyle-sammons to do final approval

bryanlb and others added 13 commits November 12, 2024 15:59
@bryanlb @ermontross
POC of field-level redaction
2364eb1
@ermontross
Field redaction
c1be098
@ermontross
Cleanup
caade1e 
fix tests

take care of comments

fmt

closes, deletes, logs

fmt

secondary agg logs

remove tostring

remove unused import

more logging yay

add redactions to searchermanagerfromchunkid

remove logs, add test
@ermontross
fmt
bddd5a8
@ermontross
fix tests
a32962e
@ermontross
consolidate redaction method
ec20fc8
@ermontross
fmt
d8579b2
@ermontross
get rid of null check
ce6cee4
@ermontross
redo comment
934f030
@ermontross
fix tests
e5c9e82
@ermontross
address PR comments
2668b88
@ermontross
fix test
1e8a672
@ermontross
address PR comments
9a55a0a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bryanlb bryanlb bryanlb left review comments

@kyle-sammons kyle-sammons kyle-sammons approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ermontross @bryanlb @kyle-sammons