diff --git a/.github/workflows/pythonpublish.yml b/.github/workflows/pythonpublish.yml new file mode 100644 index 0000000..c1918a4 --- /dev/null +++ b/.github/workflows/pythonpublish.yml @@ -0,0 +1,31 @@ +# This workflows will upload a Python Package using Twine when a release is created +# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries + +name: Upload Python Package + +on: + release: + types: [published] + +jobs: + deploy: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + - name: Set up Python + uses: actions/setup-python@v2 + with: + python-version: '3.x' + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install setuptools wheel twine + - name: Build and publish + env: + TWINE_USERNAME: ${{ '__token__' }} + TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} + run: | + python setup.py sdist bdist_wheel + twine upload dist/* diff --git a/.gitignore b/.gitignore index e877f7d..cc70f64 100644 --- a/.gitignore +++ b/.gitignore @@ -125,6 +125,8 @@ venv.bak/ dmypy.json # package related -src/additional_matches.txt -src/duplicate_passwords.txt -src/HIBP_matches.txt +lil_pwny/additional_matches.txt +lil_pwny/duplicate_passwords.txt +lil_pwny/HIBP_matches.txt +*.json +lil_pwny/caching_test*.py diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..aff58b1 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,10 @@ +## 2.0.0 - 2021-01-02 +### Added +- Massive enhancements to make much better use of multiprocessing for the large HIBP password file, as well as more efficient importing and handling of Active Directory user hashes. +- Updated directory structure to play more nicely with more OS versions and flavours, rather than installing in the `src` directory. +- Logging: Removed outdated text file output and implemented JSON formatted logging to either stdout or to .log file +- New option to obfuscate genuine password NTLM hashes in logging output. This is achieved by further hashing the hash with a randomly generated salt. +- Active Directory computer accounts are now not imported with AD user hashes. There is little value in assessing these, so no point importing them. + +## 1.2.0 - 2020-03-22 +Initial Release diff --git a/README.md b/README.md index a4e9da3..e19f0af 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,38 @@ + + # Lil Pwny ![Python 2.7 and 3 compatible](https://img.shields.io/badge/python-2.7%2C%203.x-blue.svg) ![PyPI version](https://img.shields.io/pypi/v/lil-pwny.svg) ![License: MIT](https://img.shields.io/pypi/l/lil-pwny.svg) -A multiprocessing approach to auditing Active Directory passwords using Python. +Fast, offline auditing of Active Directory passwords using Python. ## About Lil Pwny -Lil Pwny is a Python application to perform an offline audit of NTLM hashes of users' passwords, recovered from Active Directory, against known compromised passwords from Have I Been Pwned. The usernames of any accounts matching HIBP will be returned in a .txt file +Lil Pwny is a Python application to perform an offline audit of NTLM hashes of users' passwords, recovered from Active Directory, against known compromised passwords from Have I Been Pwned. Results will be output in JSON format containing the username, matching hash (can be obfuscated), and how many times the matching password has been seen in HIBP There are also additional features: -- Ability to provide a list of your own passwords to check AD users against. This allows you to check user passwords against passwords relevant to your organisation that you suspect people might be using. These are NTLM hashed, and AD hashes are then compared with this as well as the HIBP hashes. +- Ability to provide a list of your own custom passwords to check AD users against. This allows you to check user passwords against passwords relevant to your organisation that you suspect people might be using. These are NTLM hashed, and AD hashes are then compared with this as well as the HIBP hashes. - Return a list of accounts using the same passwords. Useful for finding users using the same password for their administrative and standard accounts. +- Obfuscate hashes in output, for if you don't want to handle or store live user NTLM hashes. + +More information about Lil Pwny can be found [on my blog](https://papermtn.co.uk/category/tools/lil-pwny/) + +## Resources +This application has been developed to make the most of multiprocessing in Python, with the aim of it working as fast as possible on consumer level hardware. + +Because it uses multiprocessing, the more cores you have available, the faster Lil Pwny should run. I have still had very good results with a low number of logical cores: +- Test env of ~8500 AD accounts and HIBP list of 613,584,246 hashes: + - 6 logical cores - 0:05:57.640813 + - 12 logical cores - 0:04:28.579201 -More information about Lil Pwny can be found [on my blog](https://papermtn.co.uk/) +## Output +Lil Pwny will output results as JSON format either to stdout or to file: -## Recommendations -This application was developed to ideally run on high resource infrastructure to make the most of Python multiprocessing. It will run on desktop level hardware, but the more cores you use, the faster the audit will run. +```json +{"localtime": "2021-00-00 00:00:00,000", "level": "NOTIFY", "source": "Lil Pwny", "match_type": "hibp", "detection_data": {"username": "RICKON.STARK", "hash": "0C02C50B2B08F2979DFDE12EDA472FC1", "matches_in_hibp": "24230577", "obfuscated": "True"}} +``` +This JSON formatted logging can be easily ingested in to a SIEM or other log analysis tool, and can be fed to other scripts or platforms for automated resolution actions. ## Installation Install via pip @@ -27,25 +43,33 @@ pip install lil-pwny ## Usage Lil-pwny will be installed as a global command, use as follows: -```bash -usage: lil-pwny [-h] -hibp HIBP [-a A] -ad AD_HASHES [-d] [-m] [-o OUTPUT] +``` +usage: lil-pwny [-h] -hibp HIBP [-c CUSTOM] -ad AD_HASHES [-d] + [-output {file,stdout}] [-o] optional arguments: - -hibp, --hibp-path The HIBP .txt file of NTLM hashes - -a, --a .txt file containing additional passwords to check for - -ad, --ad-hashes The NTLM hashes from of AD users - -d, --find-duplicates Output a list of duplicate password users - -m, --memory Load HIBP hash list into memory (over 24GB RAM - required) - -o, --out-path Set output path. Uses working dir when not set + -h, --help show this help message and exit + -hibp HIBP, --hibp-path HIBP + The HIBP .txt file of NTLM hashes + -c CUSTOM, --custom CUSTOM + .txt file containing additional custom passwords to + check for + -ad AD_HASHES, --ad-hashes AD_HASHES + The NTLM hashes from of AD users + -d, --duplicates Output a list of duplicate password users + -output {file,stdout}, --output {file,stdout} + Where to send results + -o, --obfuscate Obfuscate hashes from discovered matches by hashing + with a random salt + ``` Example: ```bash -lil-pwny -hibp ~/hibp_hashes.txt -ad ~/ad_ntlm_hashes.txt -a ~/additional_passwords.txt -o ~/Desktop/Output -m -d +lil-pwny -hibp ~/hibp_hashes.txt -ad ~/ad_user_hashes.txt -c ~/custom_passwords.txt -output stdout -do ``` -use of the `-m` flag will load the HIBP hashes into memory, which will allow for faster searching. Note this will require at least 24GB of available memory. + ## Getting input files ### Step 1: Get an IFM AD database dump @@ -71,9 +95,9 @@ Get-ADDBAccount -All -DBPath '.\Active Directory\ntds.dit' -BootKey $bootKey | F ``` ### Step 3: Download the latest HIBP hash file -The file can be downloaded from [here](https://downloads.pwnedpasswords.com/passwords/pwned-passwords-ntlm-ordered-by-count-v5.7z) +The file can be downloaded from [here](https://downloads.pwnedpasswords.com/passwords/pwned-passwords-ntlm-ordered-by-count-v7.7z) -The latest version of the hash file contains around 551 million hashes. +The latest version of the hash file contains around 613 million hashes. ## Resources - [ntdsutil & IFM](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/cc732530(v=ws.11)) diff --git a/lil_pwny/__about__.py b/lil_pwny/__about__.py new file mode 100644 index 0000000..79a0d2b --- /dev/null +++ b/lil_pwny/__about__.py @@ -0,0 +1,19 @@ + +__all__ = [ + '__title__', + '__summary__', + '__uri__', + '__version__', + '__author__', + '__email__', + '__license__', +] + +__title__ = 'Lil Pwny' +__summary__ = 'Fast offline auditing of Active Directory passwords using Python and multiprocessing' +__uri__ = 'https://github.com/PaperMtn/lil-pwny' +__version__ = '2.0.0' +__author__ = 'PaperMtn' +__email__ = 'papermtn@protonmail.com' +__license__ = 'GPL-3.0' +__copyright__ = '2021 {}'.format(__author__) diff --git a/lil_pwny/__init__.py b/lil_pwny/__init__.py new file mode 100644 index 0000000..b830d7e --- /dev/null +++ b/lil_pwny/__init__.py @@ -0,0 +1,159 @@ +import os +import time +import builtins +import argparse +import uuid +from datetime import timedelta + +from lil_pwny import hashing +from lil_pwny import password_audit +from lil_pwny import logger +from lil_pwny import __about__ + +OUTPUT_LOGGER = '' + + +def main(): + global OUTPUT_LOGGER + custom_count = 0 + duplicate_count = 0 + + try: + start = time.time() + + parser = argparse.ArgumentParser() + parser.add_argument('-hibp', '--hibp-path', help='The HIBP .txt file of NTLM hashes', + dest='hibp', required=True) + parser.add_argument('--version', action='version', + version='lil-pwny {}'.format(__about__.__version__)) + parser.add_argument('-c', '--custom', help='.txt file containing additional custom passwords to check for', + dest='custom') + parser.add_argument('-ad', '--ad-hashes', help='The NTLM hashes from of AD users', dest='ad_hashes', + required=True) + parser.add_argument('-d', '--duplicates', action='store_true', dest='d', + help='Output a list of duplicate password users') + parser.add_argument('-output', '--output', choices=['file', 'stdout'], dest='logging_type', + help='Where to send results') + parser.add_argument('-o', '--obfuscate', action='store_true', dest='obfuscate', + help='Obfuscate hashes from discovered matches by hashing with a random salt') + + args = parser.parse_args() + hibp_file = args.hibp + custom_passwords = args.custom + ad_hash_file = args.ad_hashes + duplicates = args.d + logging_type = args.logging_type + obfuscate = args.obfuscate + + hasher = hashing.Hashing() + + if logging_type: + if logging_type == 'file': + OUTPUT_LOGGER = logger.FileLogger(log_path=os.getcwd()) + elif logging_type == 'stdout': + OUTPUT_LOGGER = logger.StdoutLogger() + else: + OUTPUT_LOGGER = logger.StdoutLogger() + + if isinstance(OUTPUT_LOGGER, logger.StdoutLogger): + print = OUTPUT_LOGGER.log_info + else: + print = builtins.print + + print('*** Lil Pwny started execution ***') + print('Loading AD user hashes...') + try: + ad_users = password_audit.import_users(ad_hash_file) + ad_lines = 0 + for ls in ad_users.values(): + ad_lines += len(ls) + except FileNotFoundError as not_found: + raise Exception('AD user file not found: {}'.format(not_found.filename)) + except Exception as e: + raise e + + print('Comparing {} AD users against HIBP compromised passwords...'.format(ad_lines)) + try: + hibp_results = password_audit.search(OUTPUT_LOGGER, hibp_file, ad_hash_file) + hibp_count = len(hibp_results) + print(hibp_results) + for hibp_match in hibp_results: + if obfuscate: + hibp_match['hash'] = hasher.obfuscate(hibp_match.get('hash')) + hibp_match['obfuscated'] = 'True' + else: + hibp_match['obfuscated'] = 'False' + OUTPUT_LOGGER.log_notification(hibp_match, 'hibp') + except FileNotFoundError as not_found: + raise Exception('HIBP file not found: {}'.format(not_found.filename)) + except Exception as e: + raise e + + if custom_passwords: + try: + # Import custom strings from file and convert them to NTLM hashes + custom_content = hasher.get_hashes(custom_passwords) + + # Create a tmp file to store the converted hashes and pass to the search function + # Filename is a randomly generated uuid + f = open('{}.tmp'.format(str(uuid.uuid4().hex)), 'w') + for h in custom_content: + # Replicate HIBP format: "hash:occurrence" + f.write('{}:{}'.format(h, 0) + '\n') + f.close() + + print('Comparing {} Active Directory users against {} custom password hashes...' + .format(ad_lines, len(custom_content))) + custom_matches = password_audit.search(OUTPUT_LOGGER, f.name, ad_hash_file) + custom_count = len(custom_matches) + + # Remove the tmp file + os.remove(f.name) + + for custom_match in custom_matches: + if obfuscate: + custom_match['hash'] = hasher.obfuscate(custom_match.get('hash')) + custom_match['obfuscated'] = 'True' + else: + custom_match['obfuscated'] = 'False' + OUTPUT_LOGGER.log_notification(custom_match, 'custom') + except FileNotFoundError as not_found: + raise Exception('Custom password file not found: {}'.format(not_found.filename)) + except Exception as e: + raise e + + if duplicates: + try: + print('Finding users with duplicate passwords...') + duplicate_results = password_audit.find_duplicates(ad_users) + duplicate_count = len(duplicate_results) + for duplicate_match in duplicate_results: + if obfuscate: + duplicate_match['hash'] = hasher.obfuscate(duplicate_match.get('hash')) + duplicate_match['obfuscated'] = 'True' + else: + duplicate_match['obfuscated'] = 'False' + OUTPUT_LOGGER.log_notification(duplicate_match, 'duplicate') + except Exception as e: + raise e + + time_taken = time.time() - start + total_comp_count = custom_count + hibp_count + + print('Audit completed') + print('Total compromised passwords: {}'.format(total_comp_count)) + print('Passwords matching HIBP: {}'.format(hibp_count)) + print('Passwords matching custom password dictionary: {}'.format(custom_count)) + print('Passwords duplicated (being used by multiple user accounts): {}'.format(duplicate_count)) + print('Time taken: {}'.format(str(timedelta(seconds=time_taken)))) + + except Exception as e: + if isinstance(OUTPUT_LOGGER, logger.StdoutLogger): + OUTPUT_LOGGER.log_critical(e) + else: + print = builtins.print + print(e) + + +if __name__ == '__main__': + main() diff --git a/lil_pwny/__main__.py b/lil_pwny/__main__.py new file mode 100644 index 0000000..08919cb --- /dev/null +++ b/lil_pwny/__main__.py @@ -0,0 +1,3 @@ +from lil_pwny import main + +main() \ No newline at end of file diff --git a/lil_pwny/hashing.py b/lil_pwny/hashing.py new file mode 100644 index 0000000..da60f9c --- /dev/null +++ b/lil_pwny/hashing.py @@ -0,0 +1,52 @@ +import binascii +import hashlib +import secrets + + +class Hashing(object): + def __init__(self): + self.salt = secrets.token_hex(8) + + @staticmethod + def _hashify(input_string): + """Converts the input string to a NTLM hash and returns the hash + + Parameters: + input_string: string to be converted to NTLM hash + Returns: + Converted NTLM hash + """ + + output = hashlib.new('md4', input_string.encode('utf-16le')).digest() + + return binascii.hexlify(output).decode('utf-8').upper() + + def get_hashes(self, input_file): + """Reads the input file of passwords, converts them to NTLM hashes + + Parameters: + input_file: file containing strings to convert to NTLM hashes + Returns: + Dict that replicates HIBP format: 'hash:occurrence_count' + """ + + output_dict = {} + with open(input_file, 'r') as f: + for item in f: + if item: + output_dict[self._hashify(item.strip())] = '0' + + return output_dict + + def obfuscate(self, input_hash): + """Further hashes the input NTLM hash with a random salt + + Parameters: + input_hash: hash to be obfuscated + Returns: + String containing obfuscated hash + """ + + output = hashlib.new('md4', (input_hash + self.salt).encode('utf-16le')).digest() + + return binascii.hexlify(output).decode('utf-8').upper() diff --git a/lil_pwny/logger.py b/lil_pwny/logger.py new file mode 100644 index 0000000..42dfd8c --- /dev/null +++ b/lil_pwny/logger.py @@ -0,0 +1,63 @@ +import json +import os +import logging +import sys +import logging.handlers +from logging import Logger + + +class LoggingBase(Logger): + def __init__(self, name='Lil Pwny'): + super().__init__(name) + self.notify_format = logging.Formatter( + '{"localtime": "%(asctime)s", "level": "NOTIFY", "source": "%(name)s", "match_type": "%(type)s", ' + '"detection_data": %(message)s}') + self.info_format = logging.Formatter( + '{"localtime": "%(asctime)s", "level": "%(levelname)s", "source": "%(name)s", "message":' + ' "%(message)s"}') + self.log_path = '' + self.logger = logging.getLogger(self.name) + self.logger.setLevel(logging.DEBUG) + + +class FileLogger(LoggingBase): + def __init__(self, log_path): + LoggingBase.__init__(self) + self.handler = logging.handlers.WatchedFileHandler(os.path.join(log_path, 'lil-pwny.log')) + self.logger.addHandler(self.handler) + + def log_notification(self, log_data, match_type): + self.handler.setFormatter(self.notify_format) + self.logger.warning(json.dumps(log_data), extra={ + 'type': match_type + }) + + def log_info(self, log_data): + self.handler.setFormatter(self.info_format) + self.logger.info(log_data) + + def log_critical(self, log_data): + self.handler.setFormatter(self.info_format) + self.logger.critical(log_data) + + +class StdoutLogger(LoggingBase): + def __init__(self): + LoggingBase.__init__(self) + self.handler = logging.StreamHandler(sys.stdout) + self.logger.addHandler(self.handler) + + def log_notification(self, log_data, match_type): + self.handler.setFormatter(self.notify_format) + self.logger.warning(json.dumps(log_data), extra={ + 'type': match_type + }) + + def log_info(self, log_data): + self.handler.setFormatter(self.info_format) + self.logger.info(log_data) + + def log_critical(self, log_data): + self.handler.setFormatter(self.info_format) + self.logger.critical(log_data) + diff --git a/lil_pwny/password_audit.py b/lil_pwny/password_audit.py new file mode 100644 index 0000000..5c72b3a --- /dev/null +++ b/lil_pwny/password_audit.py @@ -0,0 +1,206 @@ +import gc +import os +import multiprocessing as mp + +from lil_pwny import logger + + +def import_users(filepath): + """Import Active Directory users from text file into a dict + + Parameters: + filepath: Path for the AD user file + Returns: + Dict with the key as the NTLM hash, value is a list containing users matching that hash + """ + + users = {} + with open(filepath) as infile: + for u in _nonblank_lines(infile): + username, hash = u.strip().split(':')[0].upper(), u.strip().split(':')[1].upper() + if not username.endswith('$'): + users.setdefault(hash, []).append(username) + + return users + + +def find_duplicates(ad_hash_dict): + """Returns users using the same hash in the input file. Outputs + a file grouping all users of a hash being used more than once + + Parameters: + ad_hash_dict: imported AD users as a dict + Returns: + List of dicts containing results for users using the same password + """ + + outlist = [] + for u in ad_hash_dict: + if u and len(ad_hash_dict.get(u)) > 1: + output = { + 'hash': u, + 'users': ad_hash_dict.get(u) + } + outlist.append(output) + + return outlist + + +def search(log_handler, hibp_path, ad_user_path): + users = import_users(ad_user_path) + result = mp.Manager().list() + + _multi_pro_search(log_handler, hibp_path, 100, mp.cpu_count(), _worker, [users, result]) + + return result._getvalue() + + +def _nonblank_lines(f): + """Generator to filter out blank lines from the input list + + Parameters: + f: input file + Returns: + Yields line if it isn't blank + """ + + for line in f: + if line.rstrip(): + yield line + + +def _divide_blocks(filepath, size=1024 * 1024 * 1000, skip_lines=-1): + """Divide the large text file into equal sized blocks, aligned to the start of a line + + Parameters: + filepath: Path for the hash file (HIBP or custom) + size: size of 1 block in MB + skip_lines: number of top lines to skip while processing + Returns: + List containing the start points for the input file after dividing into blocks + """ + + blocks = [] + file_end = os.path.getsize(filepath) + with open(filepath, 'rb') as f: + if skip_lines > 0: + for i in range(skip_lines): + f.readline() + + block_end = f.tell() + count = 0 + while True: + block_start = block_end + f.seek(f.tell() + size, os.SEEK_SET) + f.readline() + block_end = f.tell() + blocks.append((block_start, block_end - block_start, filepath)) + count += 1 + + if block_end > file_end: + break + + return blocks + + +def _parallel_process_block(block_data): + """Carry out worker function on each line in a block + + Parameters: + block_data: information on the block to process, start - end etc. + Returns: + List containing results of the worker on the block + """ + + block_start, block_size, filepath, function = block_data[:4] + func_args = block_data[4:] + block_results = [] + with open(filepath, 'rb') as f: + f.seek(block_start) + cont = f.read(block_size).decode(encoding='utf-8') + lines = cont.splitlines() + + for i, line in enumerate(lines): + output = function(line, *func_args) + if output is not None: + block_results.append(output) + + return block_results + + +def _multi_pro_search(log_handler, filepath, block_size, cores, worker_function, worker_function_args, skip_lines=0, outfile=None): + """Breaks the [HIBP|custom passwords] file into blocks and uses multiprocessing to iterate through them and return + any matches against AD users. + + Parameters: + log_handler: logger instance for outputting + input_file_path: path to input file + block_size: size of 1 block in MB + cores: number of processes + skip_lines: number of top lines to skip while processing + worker_function: worker function that will carry out processing + worker_function_args: arguments for the worker function + outfile: output file (optional) + Returns: + List of users matching the given password dictionary file (HIBP or custom) + """ + + jobs = _divide_blocks(filepath, 1024 * 1024 * block_size, skip_lines) + jobs = [list(j) + [worker_function] + worker_function_args for j in jobs] + + if isinstance(log_handler, logger.StdoutLogger): + log_handler.log_info('Split into {} parallel jobs '.format(len(jobs))) + log_handler.log_info('{} cores being utilised'.format(cores)) + else: + print('Split into {} parallel jobs '.format(len(jobs))) + print('{} cores being utilised'.format(cores)) + + pool = mp.Pool(cores - 1, maxtasksperchild=1000) + + outputs = [] + for block_number in range(0, len(jobs), cores - 1): + block_results = pool.map(_parallel_process_block, jobs[block_number: block_number + cores - 1]) + + for i, subl in enumerate(block_results): + for x in subl: + if outfile is not None: + print(x, file=outfile) + else: + outputs.append(x) + del block_results + gc.collect() + + pool.close() + pool.terminate() + + return outputs + + +def _worker(line, userlist, result): + """Worker function that carries out the processing on a line from the HIBP/custom passwords file. Checks to see + whether the hash on that line is in the imported AD users. If a match, a dict containing match data is appended + to the list shared between processes via multiprocessing + + Parameters: + line: line from a block of the hash file + userlist: dict containing imported AD user hashes + result: multiprocessing list shared between all processes to collect results + Returns: + List containing dict data of the matching user + """ + + ntlm_hash, count = line.rstrip().split(':')[0].upper(), line.rstrip().split(':')[1].strip().upper() + if userlist.get(ntlm_hash): + for u in userlist.get(ntlm_hash): + result.append({ + 'username': u, + 'hash': ntlm_hash, + 'matches_in_hibp': count + }) + + return result + + +if __name__ == '__main__': + users = import_users('/Users/andrew/ad_ntlm_hashes.txt') + print(find_duplicates(users)) diff --git a/setup.py b/setup.py index 9e2e0f1..0d4d1cc 100644 --- a/setup.py +++ b/setup.py @@ -1,33 +1,31 @@ import os +import lil_pwny.__about__ as a from setuptools import setup + with open(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'README.md')) as f: README = f.read() setup( name='lil-pwny', - version='1.0.2', - url='https://github.com/PaperMtn/little-pwny', - license='GPL-3.0', + version=a.__version__, classifiers=[ 'Intended Audience :: Information Technology', 'Topic :: Security', 'License :: OSI Approved :: GNU General Public License v3 (GPLv3)', - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', ], - author='PaperMtn', - author_email='papermtn@protonmail.com', + author=a.__author__, + author_email=a.__email__, long_description=README, long_description_content_type='text/markdown', - description='Auditing Active Directory Passwords ', - keywords='audit active-directory have-i-been-pwned hibp lil-pwny little-pwny password password-audit', - packages=['src'], + description=a.__summary__, + packages=['lil_pwny'], entry_points={ - 'console_scripts': ['lil-pwny=src:main'] + 'console_scripts': ['lil-pwny=lil_pwny:main'] } ) diff --git a/src/__init__.py b/src/__init__.py deleted file mode 100644 index c7853e5..0000000 --- a/src/__init__.py +++ /dev/null @@ -1,132 +0,0 @@ -import os -import sys -import time -from datetime import timedelta -import argparse -from src import hashing as h -from src import password_audit as pa - - -def main(): - start = time.time() - - parser = argparse.ArgumentParser() - parser.add_argument('-hibp', '--hibp-path', help='The HIBP .txt file of NTLM hashes', - dest='hibp', required=True) - parser.add_argument('-a', '--a', help='.txt file containing additional passwords to check for', dest='a') - parser.add_argument('-ad', '--ad-hashes', help='The NTLM hashes from of AD users', dest='ad_hashes', - required=True) - parser.add_argument('-d','--find-duplicates', action='store_true', dest='d', - help='Output a list of duplicate password users') - parser.add_argument('-m','--memory', action='store_true', dest='m', - help='Load HIBP hash list into memory (over 24GB RAM required)') - parser.add_argument('-o','--out-path', dest='output', - help='Set output path. Uses working dir if not set') - - args = parser.parse_args() - hibp_file = args.hibp - additional_password_file = args.a - ad_hash_file = args.ad_hashes - duplicates = args.d - memory = args.m - out_path = args.output - - additional_count = 0 - - print(""" - __ _ __ ____ - / / (_) / / __ \_ ______ __ __ - / / / / / / /_/ / | /| / / __ \/ / / / - / /___/ / / / ____/| |/ |/ / / / / /_/ / - /_____/_/_/ /_/ |__/|__/_/ /_/\__, / - /____/ - """) - - print('Loading AD user hashes...') - ad_users = pa.import_ad_hashes(ad_hash_file) - ad_lines = len(ad_users) - - if out_path: - if not os.path.exists(out_path): - out_path = os.getcwd() - print('Not a valid output path, defaulting to current dir: {}'.format(out_path)) - else: - out_path = os.getcwd() - - if memory: - try: - print('Loading HIBP hash dictionary into memory...') - f = open(hibp_file) - content = f.read() - hibp_lines = content.count('\n') - - print('Comparing {} Active Directory users against {} known compromised passwords...'.format(ad_lines, - hibp_lines)) - pa.multi_pro_search(ad_users, content, '{}/HIBP_matches.txt'.format(out_path)) - hibp_count = len(open('{}/HIBP_matches.txt'.format(out_path)).readlines()) - print('HIBP matches output to: {}/HIBP_matches.txt\n' - '-----'.format(out_path)) - except FileNotFoundError as not_found: - print('No such file or directory: {}'.format(not_found.filename)) - sys.exit() - except OSError: - print('Not enough memory available\n' - 'Rerun the application without the -m flag') - sys.exit() - else: - try: - print('Loading HIBP hash dictionary...') - content = pa.import_hibp_hashes(hibp_file) - hibp_lines = content.count(' ') - - print('Comparing {} Active Directory users against {} known compromised passwords...'.format(ad_lines, - hibp_lines)) - pa.multi_pro_search(ad_users, content, '{}/HIBP_matches.txt'.format(out_path)) - hibp_count = len(open('{}/HIBP_matches.txt'.format(out_path)).readlines()) - print('HIBP matches output to: {}/HIBP_matches.txt\n' - '-----'.format(out_path)) - except FileNotFoundError as not_found: - print('No such file or directory: {}'.format(not_found.filename)) - sys.exit() - - if additional_password_file: - try: - print('Loading additional hashes dictionary...') - - additional_content = h.get_hashes(additional_password_file) - additional_lines = additional_content.count(' ') - - print('Comparing {} Active Directory users against {} additional password hashes...'.format(ad_lines, - additional_lines)) - pa.multi_pro_search(ad_users, additional_content, '{}/additional_matches.txt'.format(out_path)) - additional_count = len(open('{}/additional_matches.txt'.format(out_path)).readlines()) - print('Additional matches output to: {}/additional_matches.txt\n' - '-----'.format(out_path)) - except FileNotFoundError as not_found: - print('No such file or directory: {}'.format(not_found.filename)) - sys.exit() - - if duplicates: - try: - print('Finding users with duplicate passwords...') - pa.find_duplicates(ad_users, '{}/duplicate_passwords.txt'.format(out_path)) - print('Duplicate password matches output to: {}/duplicate_passwords.txt\n' - '-----'.format(out_path)) - except FileNotFoundError as not_found: - print('No such file or directory: {}'.format(not_found.filename)) - sys.exit() - - time_taken = time.time() - start - - total_comp_count = additional_count + hibp_count - - print('Audit completed \n' - 'Total compromised passwords: {}\n' - 'Passwords matching HIBP: {}\n' - 'Passwords matching additional dictionary: {}\n' - 'Time taken: {}\n' - '-----'.format(total_comp_count, hibp_count, additional_count, str(timedelta(seconds=time_taken)))) - - -if __name__ == '__main__': - main() diff --git a/src/__main__.py b/src/__main__.py deleted file mode 100644 index 079b83a..0000000 --- a/src/__main__.py +++ /dev/null @@ -1,3 +0,0 @@ -from src import main - -main() \ No newline at end of file diff --git a/src/hashing.py b/src/hashing.py deleted file mode 100644 index 989c552..0000000 --- a/src/hashing.py +++ /dev/null @@ -1,27 +0,0 @@ -import binascii -import hashlib - -"""This module takes an input file of passwords and produces -an output file of the NTLM hashes of those passwords""" - - -def hashify(input_string): - """Converts the input string to a NTLM hash and returns the hash""" - - output = hashlib.new('md4', input_string.encode('utf-16le')).digest() - - return binascii.hexlify(output).decode('utf-8').upper() - - -def get_hashes(input_file): - """Reads the input file of passwords and returns them in a String""" - - pwds = [] - with open(input_file, 'r') as f: - for item in f: - pwds.append(hashify(item.strip())) - - output = ' '.join(pwds) - - return output - diff --git a/src/password_audit.py b/src/password_audit.py deleted file mode 100644 index e7aa42a..0000000 --- a/src/password_audit.py +++ /dev/null @@ -1,114 +0,0 @@ -import multiprocessing as mp -import itertools - - -def import_ad_hashes(ad_hash_path): - """Read contents of the AD input file and return them line - by line in a list""" - - users = {} - with open(ad_hash_path) as u_infile: - for line in nonblank_lines(u_infile): - line.strip() - temp = line.split(':') - uname = temp[0].upper() - phash = temp[1].strip().upper() - if not uname.endswith('$') and phash: - users[uname] = phash - return users - - -def import_hibp_hashes(hibp_hash_path): - """Read contents of the HIBP input file and return them line - by line in a list""" - - hibp = [] - with open(hibp_hash_path) as h_infile: - for line in nonblank_lines(h_infile): - temp = line.split(':') - hibp.append(temp[0]) - - output = " ".join(hibp) - - return output - - -def find_duplicates(ad_hash_dict, output_file_path): - """Returns users using the same hash in the input file. Outputs - a file grouping all users of a hash being used more than once""" - - flipped = {} - - for key, value in ad_hash_dict.items(): - if value not in flipped: - flipped[value] = [key] - else: - flipped[value].append(key) - - with open(output_file_path, 'w+') as f: - for key, value in flipped.items(): - if len(list(value)) > 1: - temp = '{} : {}'.format(str(key), str(value)) - f.write(temp + '\n') - - -def worker(ad_users, hibp, result): - """Worker for multiproccessing, compares one list against another - and adds matches to a list""" - - for k, v in ad_users.items(): - if v in hibp: - result.append(k + ':' + v) - - return result - - -def nonblank_lines(f): - """Filter out blank lines from the input file""" - - for l in f: - line = l.rstrip() - if line: - yield line - - -def multi_pro_search(userlist, hash_dictionary, out_path): - """Uses multiprocessing to split the userlist into (number of cores -1) amount - of dictionaries of equal size, and search these against the HIBP list. - Joins these together and outputs a list of matching users""" - - result = mp.Manager().list() - - chunks = mp.cpu_count() - 1 - print('{} cores being utilised'.format(chunks)) - - # Make sure each chunk is equal, remainder is added to last chunk - chunk_size = round(len(userlist) / chunks) - items = iter(userlist.items()) - - # Creates a list of equal sized dictionaries - list_of_chunks = [dict(itertools.islice(items, chunk_size)) for _ in range(chunks - 1)] - list_of_chunks.append(dict(items)) - - processes = [] - - for dic in list_of_chunks: - p = mp.Process(target=worker, args=(dic, hash_dictionary, result)) - processes.append(p) - p.start() - - for process in processes: - process.join() - - write_output(out_path, result) - - return result - - -def write_output(out_path, out_list): - """Writes the inputted list to a .txt file in the given path""" - - with open(out_path, 'w+') as f: - for item in out_list: - username = item.split(':', 1)[0] - f.write(username + '\n') \ No newline at end of file