feat!: EXPOSED-320 Many-to-many relation with extra columns

An intermediate table is defined to link a many-to-many relation between 2 IdTables, with references defined using via(). If this intermediate table is defined with additional columns, these are not accessible through the linked entities. This PR refactors the InnerTableLink logic to include additional columns in generated SQL, which can be accessed as a regular field on one of the referencing entities. It also allows the possibility to access the additional data as a new entity type, which wraps the main child entity along with the additional fields. This is accomplished with the introduction of InnerTableLinkEntity.
JetBrains · Aug 15, 2024 · 5b4c5ed · 5b4c5ed
1 parent 6f2019d
commit 5b4c5ed
Show file tree

Hide file tree

Showing 7 changed files with 368 additions and 60 deletions.
diff --git a/documentation-website/Writerside/topics/Breaking-Changes.md b/documentation-website/Writerside/topics/Breaking-Changes.md
@@ -3,6 +3,22 @@
 ## 0.54.0
 
 * All objects that are part of the sealed class `ForUpdateOption` are now converted to `data object`.
+* Additional columns from intermediate tables (defined for use with DAO `via()` for many-to-many relations) are no longer ignored on batch insert of references.
+  These columns are now included in the generated SQL and a value will be required when setting references, unless column defaults are defined.
+
+  To continue to ignore these columns, use the non-infix version of `via()` and provide an empty list to `additionalColumns` (or a list of specific columns to include):
+```kotlin
+// given an intermediate table StarWarsFilmActors with extra columns that should be ignored
+class StarWarsFilm(id: EntityID<Int>) : IntEntity(id) {
+    companion object : IntEntityClass<StarWarsFilm>(StarWarsFilms)
+    // ...
+    var actors by Actor.via(
+        sourceColumn = StarWarsFilmActors.starWarsFilm,
+        targetColumn = StarWarsFilmActors.actor,
+        additionalColumns = emptyList()
+    )
+}
+```
 
 ## 0.51.0
 

diff --git a/documentation-website/Writerside/topics/Deep-Dive-into-DAO.md b/documentation-website/Writerside/topics/Deep-Dive-into-DAO.md
@@ -299,15 +299,15 @@ class User(id: EntityID<Int>) : IntEntity(id) {
     ...
 ```
 
-### many-to-many reference
+### Many-To-Many reference
 In some cases, a many-to-many reference may be required.
-Let's assume you want to add a reference to the following Actors table to the StarWarsFilm class:
+Let's assume you want to add a reference to the following `Actors` table to the `StarWarsFilm` class:
 ```kotlin
-object Actors: IntIdTable() {
+object Actors : IntIdTable() {
     val firstname = varchar("firstname", 50)
     val lastname = varchar("lastname", 50)
 }
-class Actor(id: EntityID<Int>): IntEntity(id) {
+class Actor(id: EntityID<Int>) : IntEntity(id) {
     companion object : IntEntityClass<Actor>(Actors)
     var firstname by Actors.firstname
     var lastname by Actors.lastname
@@ -325,21 +325,119 @@ Add a reference to `StarWarsFilm`:
 ```kotlin
 class StarWarsFilm(id: EntityID<Int>) : IntEntity(id) {
     companion object : IntEntityClass<StarWarsFilm>(StarWarsFilms)
-    ...
+    // ...
     var actors by Actor via StarWarsFilmActors
-    ...
+    // ...
 }
 ```
-Note: You can set up IDs manually inside a transaction like this:
+Note: You can set up IDs manually inside a transaction and set all referenced actors like this:
 ```kotlin
 transaction {
-    // only works with UUIDTable and UUIDEntity
-    StarWarsFilm.new (UUID.randomUUID()){
-        ...
-        actors = SizedCollection(listOf(actor))
+    StarWarsFilm.new(421) {
+        // ...
+        actors = SizedCollection(actor1, actor2)
+    }
+}
+```
+Now you can access all actors (and their fields) for a `StarWarsFilm` object, `film`, in the same way you would get any other field:
+```kotlin
+film.actors.first() // returns an Actor object
+film.actors.map { it.lastname } // returns a List<String>
+```
+If the intermediate table is defined with more than just the two reference columns, these additional columns can be accessed in two ways, both detailed below.
+
+Given a `StarWarsFilmActors` table with the extra column `roleName`:
+```kotlin
+object StarWarsFilmActors : Table() {
+    val starWarsFilm = reference("starWarsFilm", StarWarsFilms)
+    val actor = reference("actor", Actors)
+    val roleName = varchar("role_name", 64)
+    override val primaryKey = PrimaryKey(starWarsFilm, actor)
+}
+```
+**The first approach** assumes that the value stored in this extra column will be accessed from the `Actor` class:
+```kotlin
+class Actor(id: EntityID<Int>) : IntEntity(id) {
+    companion object : IntEntityClass<Actor>(Actors)
+    var firstname by Actors.firstname
+    var lastname by Actors.lastname
+    var roleName by StarWarsFilmActors.roleName
+}
+```
+This extra value can then be set, for example, when a new `Actor` is created or when it is provided to the parent entity's field,
+and accessed like any other field:
+```kotlin
+val actor1 = Actor.new {
+    firstname = "Harrison"
+    lastname = "Ford"
+    roleName = "Han Solo"
+}
+// or
+film.actors = SizedCollection(actor1, actor2.apply { roleName = "Ben Solo" })
+
+StarWarsFilm.all().first.actors.map { it.roleName }
+```
+**The second approach** assumes that the `Actor` class should not be given an extra field and that the extra value stored should
+be accessed through an object that holds both the child entity and the additional data.
+
+To both allow this and still take advantage of the underlying DAO cache, a new entity class has to be defined using `InnerTableLinkEntity`,
+which details how to get and set the additional column values from the intermediate table through two overrides:
+```kotlin
+class ActorWithRole(
+    val actor: Actor,
+    val roleName: String
+) : InnerTableLinkEntity<Int>(actor) {
+    override fun getInnerTableLinkValue(column: Column<*>): Any = when (column) {
+        StarWarsFilmActors.roleName -> roleName
+        else -> error("Column does not exist in intermediate table")
+    }
+
+    companion object : InnerTableLinkEntityClass<Int, ActorWithRole>(Actors) {
+        override fun createInstance(entityId: EntityID<Int>, row: ResultRow?) = row?.let {
+            ActorWithRole(Actor.wrapRow(it), it[StarWarsFilmActors.roleName])
+        } ?: ActorWithRole(Actor(entityId), "")
     }
 }
 ```
+The original entity class reference now looks like this:
+```kotlin
+class StarWarsFilm(id: EntityID<Int>) : IntEntity(id) {
+    companion object : IntEntityClass<StarWarsFilm>(StarWarsFilms)
+    // ...
+    var actors by ActorWithRole via StarWarsFilmActors
+}
+class Actor(id: EntityID<Int>) : IntEntity(id) {
+    companion object : IntEntityClass<Actor>(Actors)
+    var firstname by Actors.firstname
+    var lastname by Actors.lastname
+}
+```
+This extra value can then be set by providing a new `ActorWithRole` instance to the parent entity field, and accessed as before:
+```kotlin
+film.actors = SizedCollection(
+    ActorWithRole(actor1, "Han Solo"),
+    ActorWithRole(actor2, "Ben Solo")
+)
+
+StarWarsFilm.all().first.actors.map { it.roleName }
+```
+<note>
+If only some additional columns in the intermediate table should be used during batch insert, these can be specified by using
+<code>via()</code> with an argument provided to <code>additionalColumns</code>.
+<code-block>
+class StarWarsFilm(id: EntityID&lt;Int&gt;) : IntEntity(id) {
+    companion object : IntEntityClass&lt;StarWarsFilm&gt;(StarWarsFilms)
+    // ...
+    var actors by ActorWithRole.via(
+        sourceColumn = StarWarsFilmActors.starWarsFilm,
+        targetColumn = StarWarsFilmActors.actor,
+        additionalColumns = listOf(StarWarsFilmActors.roleName)
+    )
+}
+</code-block>
+Setting this parameter to an <code>emptyList()</code> means all additional columns will be ignored.
+</note>
+
 ### Parent-Child reference
 Parent-child reference is very similar to many-to-many version, but an intermediate table contains both references to the same table.
 Let's assume you want to build a hierarchical entity which could have parents and children. Our tables and an entity mapping will look like

diff --git a/exposed-dao/api/exposed-dao.api b/exposed-dao/api/exposed-dao.api
@@ -23,6 +23,7 @@ public class org/jetbrains/exposed/dao/Entity {
 	public static synthetic fun flush$default (Lorg/jetbrains/exposed/dao/Entity;Lorg/jetbrains/exposed/dao/EntityBatchUpdate;ILjava/lang/Object;)Z
 	public final fun getDb ()Lorg/jetbrains/exposed/sql/Database;
 	public final fun getId ()Lorg/jetbrains/exposed/dao/id/EntityID;
+	public fun getInnerTableLinkValue (Lorg/jetbrains/exposed/sql/Column;)Ljava/lang/Object;
 	public final fun getKlass ()Lorg/jetbrains/exposed/dao/EntityClass;
 	public final fun getReadValues ()Lorg/jetbrains/exposed/sql/ResultRow;
 	public final fun getValue (Lorg/jetbrains/exposed/dao/EntityFieldWithTransform;Lorg/jetbrains/exposed/dao/Entity;Lkotlin/reflect/KProperty;)Ljava/lang/Object;
@@ -43,8 +44,9 @@ public class org/jetbrains/exposed/dao/Entity {
 	public final fun setValue (Lorg/jetbrains/exposed/sql/CompositeColumn;Lorg/jetbrains/exposed/dao/Entity;Lkotlin/reflect/KProperty;Ljava/lang/Object;)V
 	public final fun set_readValues (Lorg/jetbrains/exposed/sql/ResultRow;)V
 	public final fun storeWrittenValues ()V
-	public final fun via (Lorg/jetbrains/exposed/dao/EntityClass;Lorg/jetbrains/exposed/sql/Column;Lorg/jetbrains/exposed/sql/Column;)Lorg/jetbrains/exposed/dao/InnerTableLink;
+	public final fun via (Lorg/jetbrains/exposed/dao/EntityClass;Lorg/jetbrains/exposed/sql/Column;Lorg/jetbrains/exposed/sql/Column;Ljava/util/List;)Lorg/jetbrains/exposed/dao/InnerTableLink;
 	public final fun via (Lorg/jetbrains/exposed/dao/EntityClass;Lorg/jetbrains/exposed/sql/Table;)Lorg/jetbrains/exposed/dao/InnerTableLink;
+	public static synthetic fun via$default (Lorg/jetbrains/exposed/dao/Entity;Lorg/jetbrains/exposed/dao/EntityClass;Lorg/jetbrains/exposed/sql/Column;Lorg/jetbrains/exposed/sql/Column;Ljava/util/List;ILjava/lang/Object;)Lorg/jetbrains/exposed/dao/InnerTableLink;
 }
 
 public final class org/jetbrains/exposed/dao/EntityBatchUpdate {
@@ -236,8 +238,8 @@ public abstract class org/jetbrains/exposed/dao/ImmutableEntityClass : org/jetbr
 }
 
 public final class org/jetbrains/exposed/dao/InnerTableLink : kotlin/properties/ReadWriteProperty {
-	public fun <init> (Lorg/jetbrains/exposed/sql/Table;Lorg/jetbrains/exposed/dao/id/IdTable;Lorg/jetbrains/exposed/dao/EntityClass;Lorg/jetbrains/exposed/sql/Column;Lorg/jetbrains/exposed/sql/Column;)V
-	public synthetic fun <init> (Lorg/jetbrains/exposed/sql/Table;Lorg/jetbrains/exposed/dao/id/IdTable;Lorg/jetbrains/exposed/dao/EntityClass;Lorg/jetbrains/exposed/sql/Column;Lorg/jetbrains/exposed/sql/Column;ILkotlin/jvm/internal/DefaultConstructorMarker;)V
+	public fun <init> (Lorg/jetbrains/exposed/sql/Table;Lorg/jetbrains/exposed/dao/id/IdTable;Lorg/jetbrains/exposed/dao/EntityClass;Lorg/jetbrains/exposed/sql/Column;Lorg/jetbrains/exposed/sql/Column;Ljava/util/List;)V
+	public synthetic fun <init> (Lorg/jetbrains/exposed/sql/Table;Lorg/jetbrains/exposed/dao/id/IdTable;Lorg/jetbrains/exposed/dao/EntityClass;Lorg/jetbrains/exposed/sql/Column;Lorg/jetbrains/exposed/sql/Column;Ljava/util/List;ILkotlin/jvm/internal/DefaultConstructorMarker;)V
 	public final fun getSourceColumn ()Lorg/jetbrains/exposed/sql/Column;
 	public final fun getTable ()Lorg/jetbrains/exposed/sql/Table;
 	public final fun getTarget ()Lorg/jetbrains/exposed/dao/EntityClass;
@@ -251,6 +253,17 @@ public final class org/jetbrains/exposed/dao/InnerTableLink : kotlin/properties/
 	public fun setValue (Lorg/jetbrains/exposed/dao/Entity;Lkotlin/reflect/KProperty;Lorg/jetbrains/exposed/sql/SizedIterable;)V
 }
 
+public abstract class org/jetbrains/exposed/dao/InnerTableLinkEntity : org/jetbrains/exposed/dao/Entity {
+	public fun <init> (Lorg/jetbrains/exposed/dao/Entity;)V
+	public abstract fun getInnerTableLinkValue (Lorg/jetbrains/exposed/sql/Column;)Ljava/lang/Object;
+	public final fun getWrapped ()Lorg/jetbrains/exposed/dao/Entity;
+}
+
+public abstract class org/jetbrains/exposed/dao/InnerTableLinkEntityClass : org/jetbrains/exposed/dao/EntityClass {
+	public fun <init> (Lorg/jetbrains/exposed/dao/id/IdTable;)V
+	protected abstract fun createInstance (Lorg/jetbrains/exposed/dao/id/EntityID;Lorg/jetbrains/exposed/sql/ResultRow;)Lorg/jetbrains/exposed/dao/InnerTableLinkEntity;
+}
+
 public abstract class org/jetbrains/exposed/dao/IntEntity : org/jetbrains/exposed/dao/Entity {
 	public fun <init> (Lorg/jetbrains/exposed/dao/id/EntityID;)V
 }

diff --git a/exposed-dao/src/main/kotlin/org/jetbrains/exposed/dao/Entity.kt b/exposed-dao/src/main/kotlin/org/jetbrains/exposed/dao/Entity.kt
@@ -78,6 +78,16 @@ open class Entity<ID : Comparable<ID>>(val id: EntityID<ID>) {
 
     private val referenceCache by lazy { HashMap<Column<*>, Any?>() }
 
+    private val writeInnerTableLinkValues by lazy { HashMap<Column<*>, Any?>() }
+
+    /**
+     * Returns the initial column-value mapping for an entity involved in an [InnerTableLink] relation
+     * before being flushed and inserted into the database.
+     *
+     * @sample org.jetbrains.exposed.sql.tests.shared.entities.ViaTests.ProjectWithApproval
+     */
+    open fun getInnerTableLinkValue(column: Column<*>): Any? = writeInnerTableLinkValues[column]
+
     internal fun isNewEntity(): Boolean {
         val cache = TransactionManager.current().entityCache
         return cache.inserts[klass.table]?.contains(this) ?: false
@@ -273,13 +283,18 @@ open class Entity<ID : Comparable<ID>>(val id: EntityID<ID>) {
     @Suppress("UNCHECKED_CAST", "USELESS_CAST")
     fun <T> Column<T>.lookup(): T = when {
         writeValues.containsKey(this as Column<out Any?>) -> writeValues[this as Column<out Any?>] as T
+        writeInnerTableLinkValues.containsKey(this) -> getInnerTableLinkValue(this) as T
         id._value == null && _readValues?.hasValue(this)?.not() ?: true -> defaultValueFun?.invoke() as T
         columnType.nullable -> readValues[this]
         else -> readValues[this]!!
     }
 
     operator fun <T> Column<T>.setValue(o: Entity<ID>, desc: KProperty<*>, value: T) {
         klass.invalidateEntityInCache(o)
+        if (this !in klass.table.columns) {
+            writeInnerTableLinkValues[this] = value
+            return
+        }
         val currentValue = _readValues?.getOrNull(this)
         if (writeValues.containsKey(this as Column<out Any?>) || currentValue != value) {
             val entityCache = TransactionManager.current().entityCache
@@ -337,14 +352,19 @@ open class Entity<ID : Comparable<ID>>(val id: EntityID<ID>) {
      *
      * @param sourceColumn The intermediate table's reference column for the child entity class.
      * @param targetColumn The intermediate table's reference column for the parent entity class.
+     * @param additionalColumns Any additional columns from the intermediate table that should be included when inserting.
+     * If left `null`, all columns additional to the [sourceColumn] and [targetColumn] will be included in the insert
+     * statement and will require a value if defaults are not defined. Provide an empty list as an argument if all
+     * additional columns should be ignored.
      * @sample org.jetbrains.exposed.sql.tests.shared.entities.ViaTests.NodesTable
      * @sample org.jetbrains.exposed.sql.tests.shared.entities.ViaTests.Node
      * @sample org.jetbrains.exposed.sql.tests.shared.entities.ViaTests.NodeToNodes
      */
     fun <TID : Comparable<TID>, Target : Entity<TID>> EntityClass<TID, Target>.via(
         sourceColumn: Column<EntityID<ID>>,
-        targetColumn: Column<EntityID<TID>>
-    ) = InnerTableLink(sourceColumn.table, this@Entity.id.table, this@via, sourceColumn, targetColumn)
+        targetColumn: Column<EntityID<TID>>,
+        additionalColumns: List<Column<*>>? = null
+    ) = InnerTableLink(sourceColumn.table, this@Entity.id.table, this@via, sourceColumn, targetColumn, additionalColumns)
 
     /**
      * Deletes this [Entity] instance, both from the cache and from the database.

diff --git a/exposed-dao/src/main/kotlin/org/jetbrains/exposed/dao/EntityCache.kt b/exposed-dao/src/main/kotlin/org/jetbrains/exposed/dao/EntityCache.kt
@@ -23,6 +23,7 @@ class EntityCache(private val transaction: Transaction) {
     internal val inserts = LinkedHashMap<IdTable<*>, MutableSet<Entity<*>>>()
     private val updates = LinkedHashMap<IdTable<*>, MutableSet<Entity<*>>>()
     internal val referrers = HashMap<Column<*>, MutableMap<EntityID<*>, SizedIterable<*>>>()
+    private val innerTableLinks by lazy { LinkedHashMap<IdTable<*>, MutableMap<Any, Entity<*>>>() }
 
     /**
      * The amount of entities to store in this [EntityCache] per [Entity] class.
@@ -55,7 +56,11 @@ class EntityCache(private val transaction: Transaction) {
             }
         }
 
-    private fun getMap(f: EntityClass<*, *>): MutableMap<Any, Entity<*>> = getMap(f.table)
+    private fun getMap(f: EntityClass<*, *>): MutableMap<Any, Entity<*>> = if (f is InnerTableLinkEntityClass<*, *>) {
+        innerTableLinks.getOrPut(f.table) { LimitedHashMap() }
+    } else {
+        getMap(f.table)
+    }
 
     private fun getMap(table: IdTable<*>): MutableMap<Any, Entity<*>> = data.getOrPut(table) {
         LimitedHashMap()
@@ -293,6 +298,7 @@ class EntityCache(private val transaction: Transaction) {
         inserts.clear()
         updates.clear()
         clearReferrersCache()
+        innerTableLinks.clear()
     }
 
     /** Clears this [EntityCache] of stored data that maps cached parent entities to their referencing child entities. */