1.0.1

What's Changed

• Option for OpenAI special token removal. by @PhilipMay in #164
• Add support for Azure authentication using Azure AD Tokens by @alaeddine-13 in #165
• Update Python Versions by @PhilipMay in #166

New Contributors

• @alaeddine-13 made their first contribution in #165

Full Changelog: 1.0.0...1.0.1

1.0.1rc3

What's Changed

• Option for OpenAI special token removal. by @PhilipMay in #164
• Add support for Azure authentication using Azure AD Tokens by @alaeddine-13 in #165

New Contributors

• @alaeddine-13 made their first contribution in #165

Full Changelog: 1.0.0...1.0.1rc3

1.0.0

What's Changed

• Add FileBasedRestartableBatchDataProcessor. by @PhilipMay in #154
• Add option to pass OpenAI key with os env. by @PhilipMay in #156
• Change data format to pkl / joblib for FileBasedRestartableBatchDataProcessor. by @PhilipMay in #157
• Fix unknown file issue in FileBasedRestartableBatchDataProcessor. by @PhilipMay in #158
• Avoid duplicate results in FileBasedRestartableBatchDataProcessor. by @PhilipMay in #159
• Add openai async function. by @PhilipMay in #160
• Add len to FileBasedRestartableBatchDataProcessor by @PhilipMay in #161
• Handle orphaned lock files for FileBasedRestartableBatchDataProcessor. by @PhilipMay in #162
• Add ignore error in FileBasedRestartableBatchDataProcessor data loading. by @PhilipMay in #163

Full Changelog: 0.12.3...1.0.0

0.12.3

What's Changed

• Add clean function clean_all_invisible_chars_and_strip. by @PhilipMay in #153

Full Changelog: 0.12.2...0.12.3

0.12.2

What's Changed

• Add SoMaJo token extractor. by @PhilipMay in #152

Full Changelog: 0.12.0...0.12.2

0.12.0

What's Changed

• Fix Somajo md link extraction. by @PhilipMay in #150

Full Changelog: 0.11.1...0.12.0

0.11.1

What's Changed

• Add pytest-random-order plugin and configure options. by @PhilipMay in #147
• Add default language for detection. by @PhilipMay in #149

Full Changelog: 0.11.0...0.11.1

0.11.0

What's Changed

• Add ArangoDB upload tool. by @PhilipMay in #141
• Add hypothesis tests for OpenAiTokenCounter class. by @PhilipMay in #143
• Add support for new OpenAI API. by @PhilipMay in #144

Full Changelog: 0.10.0...0.11.0

0.10.1rc1

What's Changed

• Add ArangoDB upload tool. by @PhilipMay in #141

Full Changelog: 0.10.0...0.10.1rc1

0.10.0

What's Changed

• Add fit and distance flags to TextDistance class for call order checks. by @PhilipMay in #122
• Add ArangoDB utility module. by @PhilipMay in #123
• Add ArangoDB collection backup functionality. by @PhilipMay in #124
• Add Beautiful Soup specific tools. by @PhilipMay in #126
• Update ruff config with "ALL" default. by @PhilipMay in #134
• Add pytest-rerunfailures dependency and mark data tests as flaky. by @PhilipMay in #135
• Add hypothesis tests. by @PhilipMay in #136
• Add XML tag detection function. by @PhilipMay in #139

Full Changelog: 0.9.0...0.10.0