improvement: add some SQLite pragma statement settings to improve performance

argilla-server/CHANGELOG.md

@@ -42,6 +42,10 @@ These are the section headers that we use:
 - Added `REINDEX_DATASETS` environment variable to Argilla server Docker image. ([#5268](https://github.com/argilla-io/argilla/pull/5268))
 - Added `argilla-hf-spaces` docker image for running Argilla server in HF spaces. ([#5307](https://github.com/argilla-io/argilla/pull/5307))
 
+### Added
+
+- Added some new performance tuning settings for SQLite database. ([#5150](https://github.com/argilla-io/argilla/pull/5150))
+
 ### Changed
 
 - Change `responses` table to delete rows on cascade when a user is deleted. ([#5126](https://github.com/argilla-io/argilla/pull/5126))
@@ -75,6 +79,7 @@ These are the section headers that we use:
 
 - Fixed error when updating records in bulk with wrong `external_id` but correct record `id`. ([#5014](https://github.com/argilla-io/argilla/pull/5014))
 - Fixed error when searching all record response values. ([#5003](https://github.com/argilla-io/argilla/pull/5003))
+- Fixed SQLite connection settings not working correctly due to a outdated conditional. ([#5149](https://github.com/argilla-io/argilla/pull/5149))
 
 ## [1.29.0](https://github.com/argilla-io/argilla/compare/v1.28.0...v1.29.0)
 

argilla-server/src/argilla_server/database.py

@@ -45,7 +45,34 @@
 def set_sqlite_pragma(dbapi_connection, connection_record):
     if isinstance(dbapi_connection, AsyncAdapt_aiosqlite_connection):
         cursor = dbapi_connection.cursor()
+
+        # Enforce foreign key constraints
+        # https://www.sqlite.org/pragma.html#pragma_foreign_keys
+        # https://www.sqlite.org/foreignkeys.html
         cursor.execute("PRAGMA foreign_keys = ON")
+
+        # Journal mode WAL allows for greater concurrency (many readers + one writer)
+        # https://www.sqlite.org/pragma.html#pragma_journal_mode
+        cursor.execute("PRAGMA journal_mode = WAL")
+
+        # Set more relaxed level of database durability
+        # 2 = "FULL" (sync on every write), 1 = "NORMAL" (sync every 1000 written pages) and 0 = "NONE"
+        # https://www.sqlite.org/pragma.html#pragma_synchronous
+        cursor.execute("PRAGMA synchronous = NORMAL")
+
+        # Set the global memory map so all processes can share some data
+        # https://www.sqlite.org/pragma.html#pragma_mmap_size
+        # https://www.sqlite.org/mmap.html
+        cursor.execute("PRAGMA mmap_size = 134217728")  # 128 megabytes
+
+        # Impose a limit on the WAL file to prevent unlimited growth
+        # https://www.sqlite.org/pragma.html#pragma_journal_size_limit
+        cursor.execute("PRAGMA journal_size_limit = 67108864")  # 64 megabytes
+
+        # Set the local connection cache to 2000 pages
+        # https://www.sqlite.org/pragma.html#pragma_cache_size
+        cursor.execute("PRAGMA cache_size = 2000")
+
         cursor.close()
 
 

argilla-server/tests/unit/test_database.py

@@ -26,3 +26,8 @@ async def test_sqlite_pragma_settings(self, db: AsyncSession):
             return
 
         assert (await db.execute(text("PRAGMA foreign_keys"))).scalar() == 1
+        assert (await db.execute(text("PRAGMA journal_mode"))).scalar() == "wal"
+        assert (await db.execute(text("PRAGMA synchronous"))).scalar() == 1
+        assert (await db.execute(text("PRAGMA journal_size_limit"))).scalar() == 67108864
+        assert (await db.execute(text("PRAGMA mmap_size"))).scalar() == 134217728
+        assert (await db.execute(text("PRAGMA cache_size"))).scalar() == 2000