Core: Add TableUtil to provide access to a table's format version

[nastra]

This is an alternative impl to #11587

[jbonofre]

protected is for test purpose. Should we add @VisibleForTesting or similar (just as indication) ?

[nastra]

this method isn't used for test purposes, but rather to get access to the underlying table in the subclass

[jbonofre]

Oh yes, right. I misread. Good to me then.

[jbonofre]

I agree with this approach. I just wonder if we should not define the format version just as table property (and set when using delete vectors for instance). It's more "hardcoded" but probably easier to evaluate (and fallback to V2 if the table property is not present).

[RussellSpitzer]

I would strongly prefer we keep it out of table properties. IMHO, Table properties are user facing hints or guides and the system should not rely on them for correct behavior. Having the correct format version is important for correct behavior so I would keep it out of properties.

[jbonofre]

OK, fair enough (good point about mutable/user facing aspect). I'm just a little "concerned" about the "complexity" of this method to get the format version (it remembers me the hack we did in JDBC Catalog 😄 ).

[RussellSpitzer]

I wish we had scala or even kotlin

[RussellSpitzer]

One of the reasons I want scala is I really want to just enumerate cases here. I would recommend we just go through all of our cases narrower then broader if we have exceptions so

If (PositionDeleteTable)
  return format version 
else if (MetaTable) {
  Sorry Brah
}
else if (HasTableOperations) {
   return format version
}
else {
 // Sorry Brah
}

Although I am curious why position delete table needs a format version?

[nastra]

Although I am curious why position delete table needs a format version?

@RussellSpitzer this is mainly for SparkPositionDeletesRewrite (which operates against the position deletes table). Basically when rewriting existing position deletes we need to know whether we need to rewrite them to V2 position deletes or to DVs by looking at the underlying format version of the table. A table that was upgraded to V3 can still have V2 position deletes, meaning that these would then have to be rewritten as DVs

[RussellSpitzer]

Shouldn't we just be looking at the underlying table then? Shouldn't the converter look at the base table rather than the metadata table?

Ie

formatVersion(metadataTable.baseTable)

Rather than

formatVersion(metadataTable)

Implicitly calling metadataTable.baseTable but only sometimes

[nastra]

I would recommend we just go through all of our cases narrower then broader if we have exceptions so

I would prefer that too, but the fact that SerializableMetadataTable is a subclass of SerializableTable which in turn implements HasTableOperations makes this more difficult and you still need to differentiate there

[nastra]

Shouldn't we just be looking at the underlying table then? Shouldn't the converter look at the base table rather than the metadata table?

I'm not fully sure I follow your comment. Do you mean the calling site should first check whether it's a metadata table before calling TableUtil.formatVersion(...)?

[RussellSpitzer]

Yes. But I also don't think the caller should need to check. Shouldn't the caller know what it's doing? Like if it is compacting DeleteVectors it knows it has a DeleteVectorMetadataTable and therefor it uses the parent table.

[nastra]

that would require changing a bunch of more places, because effectively we have a Broadcast<Table> in SparkPositionDeletesRewrite:

iceberg/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeletesRewrite.java

Lines 130 to 131 in 8351248

	Broadcast<Table> tableBroadcast =
	sparkContext.broadcast(SerializableTableWithSize.copyOf(table));

[RussellSpitzer]

That is a an argument for ease of current implementation. But do you think it's the right decision going forward to have a format version for position deletes metadata table and none of the other metadata tables?