Integration of Clamav for verifying uploaded files

[fpietrosanti]

This ticket is to integrate globaleaks with antivirus scanning capabilities.

The antivirus scanning must work trough a generic filter server, by using the ICAP protocol, to interact with an ICAP server taking care of the antivirus scanning.

By using such a modular filtering, standard based approach, we can integrate with SquidClamav software, that can potentially:

• run on another server
• run other kind of checks

In future we may integrate MAT cleanup, as an ICAP server, using the very same batch/content filtering logic applied for antivirus.

It must be notes that, by employing an ICAP based architecture for file filtering, also GnuPG operations could be moved outside globaleaks main process with a dedicated "encryption server" .

[NSkelsey]

Hey folks, sopratutto @evilaliv3, I am bumping this issue with two different possible approaches to solving the underlying issue which is:

Recipients can be phished with promises of information and tricked into executing malicious files or malware on their workstations by a submission with an evil attachment.

In order to make it harder for the average attacker to take control of a recipient's machine, scanning submitted files for viruses with known signatures makes sense.

There are two points in the submission process where this can be done if GPG encryption of files is performed and one more if it is not.

1. On the client's workstation by a consumer based AV
2. On the globaleaks server before the file is written encrypted on disk via GPG
3. (in the case without file encryption) after the file is written to disk before the receiver can download.

Case 1

The only extra step for this is updating the documentation with the recommendation that receivers install an Anti-Virus from a vendor they trust.

Case 2

The big change would be the inclusion, usage and periodic update of ClamAV. The debian package would have to configured to not send any information to the vendor. It would also require trusting the Cisco Talos team to not have intentionally backdoored the product which is possible.

Further the routines related to GLTemporaryFile would have to change, along with the Submission work flow to pass the raw file maybe via a pipe to a ClamAV subproccess to do the scanning. Its not clear to me if it is better to have one ClamAVScanner running or one created for every file submission, but I think either way would work.

I think this is the best approach if GlobaLeaks is to be used as an all in one appliance.

Case 3

This situation is very similar to Case 2 with relaxed constraints. A background process could very easily be scheduled to scan the unscanned files on disc and mark the dangerous ones in the rtip_file table as viruses and the clean ones as readable.

[evilaliv3]

Hello @NSkelsey ! and than you for the important feedback on this topic.

We generally discarded to move forward this ticket for two reasons, but lets see what we can now do for it:

• exfiltration: the antivirus doesnt have to leak outside the information received on the platform; maybe we can select a technology that at least declare to work offline with respect to the data.
• implementation issues: an antivirus would never fit with the browsercrypto concept we designed; we are now evaluating (considering that the system will always continue to be a trusted party) to implement anyhow this possibility that would require to store the file in plaintext with respect to the antivirus on the server (maybe on a luks partition to preserve encryption with respect to a forensic analysis)

[evilaliv3]

The current implementation of the encryption schema now enables to implement this feature.

It is now possible to run a sandboxed version of an antivirus to check offline for the presence of a malware inside an uploaded file.

[evilaliv3]

This same analysis has been as well proposed by Eduar Martin Borrgon of Poder/Mexicoleaks

[emartinborregon]

We see this option as really necessary to effectively implement the new encryption of the Globaleaks 4. For us, we feel responsible for our users security, to move from decrypt in computer to decrypt in server will mean put aside Tails and make all in the users computers. As México and Latin America journalist and activist face many spy threats, put some barriers to malware and spyware is a must.

[evilaliv3]

Thank you @emartinborregon for your valuable feedback!

Could for the moment users just use Tails directly when downloading the files?

Integration of an antivirus would probably take time more than resource constraits for research aspets:

• an opensource solution should be selected (as far that i know i think only clamav is available)
• we need to be sure exfiltration is not possible (e.g. hashes are checked offline and not sent upstream)
• we need to be sure the antvirus works in ram while checking the file to minimize ricks of traces left in plaintext on the filesystem (current globaleaks never stores a plaintext file on the filesystem)

Have you by any chance already analyzed the points above?

[evilaliv3]

Update:

We are finally collaborating with the Italian National Authority for Anticorruption on the implementation of a first integration with an Antivirus and specifically with the Clamav

Here are current evaluations and ideas.

From an high level point of view we have identified that:

• Given the encryption run on files few are the moments where the file is available in plaintext form and these moments are: 1) during file upload 2) when a recipient access and is in possibility to decrypt the file
• It would be interesting to be able to check the files automatically, report a status of the antivirus check eventually preventing via a privilege user to download files that are not checked or that are marked as containing a virus
• The system should save the date of the check and report it to the user
• The system should enable a user to perform a new check on the file upon request, useful for example to run a check on a file that was received and checked long time before the download need or a file on which the antivirus check was not performed.

From an implementation point of view:

• Currently it has been identified to integrate CLAMD using its STREAMING protocol that enables to interconnect a software with the antivirus with a remote TCP connection or with a local UNIX socket; This will enable to pass the file to a remote antivirus without the need of storing the file on the disk in plaintext before sending it to the antivirus.
• The integration could by done with python3-clamd that is very simple and stable script included in Debian and not updated since 2017 as the clamav protocol is very simple. Even if not maintained the library is composed by a single file of ~800lines where the relevant code for this integration is just a set of ~100 lines that we could probably integrate manually and improve up to our needs if it will become necessary.
• Regarding the CLAMD daemon, it has been noticed while it is able to receive a file via socket in streaming, in order to check it it always need to write the file on the disk on a storage directory in plaintext as it is able to check the file only when it is fully written and accessible via file system calls (clamd/scanner.c#L579). By default it writes files in /tmp that on Debian/Ubuntu is mapped on a persistent disk storage. This path can be changed via a configuration file using the variable TemporaryDirectory and should be mapped on a directory mounted in RAM or to an encrypted filesystem preventing leaking of the plaintext file on the CLAMAV system.

[evilaliv3]

I've remodeled the ticket to precisely track the analysis of the implementation proposed in #4314 and evaluate the change needed to make it conform to our specs and proceed with a possible integration.

Status of the current implementation:

• From the demo of the functionality the functionality seems fully functional

Actions necessary to possibly integrate same functionality:

• Make it possible for the web admin to possible to enable/disable the feature adding an "Antivirus" web interface hosting an enabled (true/false)
• Making it possible for the web admin to configure Clamav endpoint so to be able to use a local antivirus or remote endpoint (it should be enough to configure a string variable to host the endpoint config
• The implementation seems to use a non standard not documented API ('http://localhost:3000/api/v1/scan') different from the one officially specified by ClamD for asyncronous testing (TCP port 3301) referred in Integration of Clamav for verifying uploaded files #711 (comment);
• Implement client or unit tests for the feature to guarantee appropriate code coverage

To guarantee appropriate code coverage probably it should be enough to edit existing browser tests to enable the antivirus and modify the existing whistleblowing module tests loading the EICAR test files are specifically developed by the European Institute for Computer Antivirus Research (EICAR) and Computer Antivirus Research Organization (CARO), to test the response of computer antivirus programs.

[evilaliv3]

I've proceeded with the full revision of the proposed changes to database included in this implementation.

The revisions consider to be just changes in naming necessary for globaleaks for international naming conventions and code consistency that are:

• Add 'antivirus_enabled' variable to make it possible to enable/disable the functionality via UI
• Edit the set of 'antivirus' variables to enable type checking on the configuration of the ClamD endpoint

We have avoided other structural changes understanding this could have major requirements in the refactoring of the code proposing the minimum changes necessary and proposing only the changes already previously indicated.

[evilaliv3]

@joeman65 @alessiofranceschini: We are trying to test this component but i think we need some clarification.

We see used a local API http://localhost:3000/api/v1/scan ; which is the software that offer this is API?