Dependency conflicts on com.google.protobuf:protobuf-java, leading to inconsistent program behaviors

[HelloCoCooo]

Hi, in DataLink/dl-flinker/dl-flinker-hbasewriter98, there are mulptiple versions of library com.google.protobuf:protobuf-java. However, according to Maven's dependency management strategy: "first declaration wins", only com.google.protobuf:protobuf-java:2.5.0 can be loaded, and com.google.protobuf:protobuf-java:2.6.0 will be shadowed.

In total, there are 10 conflicting API pairs between these two library version.

As shown in the following figure, your project expects to invoke method <com.google.protobuf.CodedInputStream: readRawBytes(I)[B> in library com.google.protobuf:protobuf-java:2.6.0 (along the original dependency path). As it has been shadowed, this method defined in com.google.protobuf:protobuf-java:2.5.0 is actually forced to be referenced via the following invocation path (along the actual dependency path):

<com.ucar.datalink.flinker.plugin.writer.hbasewriter98.HbaseUtil: parseColumns(Ljava/util/List;)Ljava/util/List;> /home/wwww/wangSensor/unzip/DataLink-1.0.2-beta/dl-flinker/dl-flinker-hbasewriter98/target/classes
<com.mysql.cj.xdevapi.AbstractDataResult: next()Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.xdevapi.DbDocFactory: createFromProtocolEntity(Lcom/mysql/cj/protocol/ProtocolEntity;)Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.xdevapi.DbDocFactory: createFromProtocolEntity(Lcom/mysql/cj/protocol/ProtocolEntity;)Lcom/mysql/cj/xdevapi/DbDoc;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.protocol.x.XProtocolRow: getValue(ILcom/mysql/cj/result/ValueFactory;)Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.protocol.x.XProtocolDecoder: decodeSet([BIILcom/mysql/cj/result/ValueFactory;)Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.google.protobuf.CodedInputStream: readRawBytes(I)[B>

Although both of these conflicting libraries contain the referenced methods (with the same signature), they have different implementations. This issue will not cause runtime crashes, but it can introduce inconsistent semantic program hehaviors----

Code snippet of <com.google.protobuf.CodedInputStream: readRawBytes(I)[B> in com.google.protobuf:protobuf-java:2.6.0 (shadowed but expected to invoke method):

detailed method body

public byte[] readRawBytes(final int size) throws IOException {
    final int pos = bufferPos;
    if (size <= (bufferSize - pos) && size > 0) {
      bufferPos = pos + size;
      return Arrays.copyOfRange(buffer, pos, pos + size);
    } else {
      return readRawBytesSlowPath(size);
    }
  }

Code snippet of <com.google.protobuf.CodedInputStream: readRawBytes(I)[B> in com.google.protobuf:protobuf-java:2.5.0 (loaded version):

detailed method body

public byte[] readRawBytes(final int size) throws IOException {
    if (size < 0) {
      throw InvalidProtocolBufferException.negativeSize();
    }

    if (totalBytesRetired + bufferPos + size > currentLimit) {
      // Read to the end of the stream anyway.
      skipRawBytes(currentLimit - totalBytesRetired - bufferPos);
      // Then fail.
      throw InvalidProtocolBufferException.truncatedMessage();
    }

    if (size <= bufferSize - bufferPos) {
      // We have all the bytes we need already.
      final byte[] bytes = new byte[size];
      System.arraycopy(buffer, bufferPos, bytes, 0, size);
      bufferPos += size;
      return bytes;
    } else if (size < BUFFER_SIZE) {
      // Reading more bytes than are in the buffer, but not an excessive number
      // of bytes.  We can safely allocate the resulting array ahead of time.

      // First copy what we have.
      final byte[] bytes = new byte[size];
      int pos = bufferSize - bufferPos;
      System.arraycopy(buffer, bufferPos, bytes, 0, pos);
      bufferPos = bufferSize;

      // We want to use refillBuffer() and then copy from the buffer into our
      // byte array rather than reading directly into our byte array because
      // the input may be unbuffered.
      refillBuffer(true);

      while (size - pos > bufferSize) {
        System.arraycopy(buffer, 0, bytes, pos, bufferSize);
        pos += bufferSize;
        bufferPos = bufferSize;
        refillBuffer(true);
      }

      System.arraycopy(buffer, 0, bytes, pos, size - pos);
      bufferPos = size - pos;

      return bytes;
    } else {
      // The size is very large.  For security reasons, we can't allocate the
      // entire byte array yet.  The size comes directly from the input, so a
      // maliciously-crafted message could provide a bogus very large size in
      // order to trick the app into allocating a lot of memory.  We avoid this
      // by allocating and reading only a small chunk at a time, so that the
      // malicious message must actually *be* extremely large to cause
      // problems.  Meanwhile, we limit the allowed size of a message elsewhere.

      // Remember the buffer markers since we'll have to copy the bytes out of
      // it later.
      final int originalBufferPos = bufferPos;
      final int originalBufferSize = bufferSize;

      // Mark the current buffer consumed.
      totalBytesRetired += bufferSize;
      bufferPos = 0;
      bufferSize = 0;

      // Read all the rest of the bytes we need.
      int sizeLeft = size - (originalBufferSize - originalBufferPos);
      final List<byte[]> chunks = new ArrayList<byte[]>();

      while (sizeLeft > 0) {
        final byte[] chunk = new byte[Math.min(sizeLeft, BUFFER_SIZE)];
        int pos = 0;
        while (pos < chunk.length) {
          final int n = (input == null) ? -1 :
            input.read(chunk, pos, chunk.length - pos);
          if (n == -1) {
            throw InvalidProtocolBufferException.truncatedMessage();
          }
          totalBytesRetired += n;
          pos += n;
        }
        sizeLeft -= chunk.length;
        chunks.add(chunk);
      }

      // OK, got everything.  Now concatenate it all into one buffer.
      final byte[] bytes = new byte[size];

      // Start by copying the leftover bytes from this.buffer.
      int pos = originalBufferSize - originalBufferPos;
      System.arraycopy(buffer, originalBufferPos, bytes, 0, pos);

      // And now all the chunks.
      for (final byte[] chunk : chunks) {
        System.arraycopy(chunk, 0, bytes, pos, chunk.length);
        pos += chunk.length;
      }

      // Done.
      return bytes;
    }
  }

The detailed informantion of the remaining 9 conflicting API pairs can be found in the following attachment.
10 conflicting API pairs in project dl-flinker-hbasewriter98.txt

Dependency tree--

[INFO] com.ucar.datalink:dl-flinker-hbasewriter98:jar:1.0.2-beta
[INFO] +- com.ucar.datalink:dl-flinker-api:jar:1.0.2-beta:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] | +- com.google.guava:guava:jar:18.0:compile
[INFO] | +- com.alibaba:fastjson:jar:1.2.62:compile (version managed from 1.1.43)
[INFO] | +- joda-time:joda-time:jar:2.9.4:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.12:compile - version managed from 1.7.7; omitted for duplicate)
[INFO] | +- (ch.qos.logback:logback-classic:jar:1.1.3:compile - version managed from 1.0.13; omitted for duplicate)
[INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] | +- com.google.code.gson:gson:jar:2.8.0:compile
[INFO] | +- (org.apache.zookeeper:zookeeper:jar:3.4.14:compile - version managed from 3.4.6; omitted for conflict with 3.3.2)
[INFO] | +- com.github.sgroschupf:zkclient:jar:0.1:compile
[INFO] | | +- (org.apache.zookeeper:zookeeper:jar:3.4.14:compile - version managed from 3.3.3; omitted for duplicate)
[INFO] | | - log4j:log4j:jar:1.2.14:compile
[INFO] | +- org.springframework:spring:jar:2.5.6:compile
[INFO] | | - commons-logging:commons-logging:jar:1.1.1:compile
[INFO] | - mysql:mysql-connector-java:jar:8.0.11:compile
[INFO] | \ - (com.google.protobuf:protobuf-java:jar:2.6.0:compile - scope updated from runtime; omitted for duplicate)
[INFO] +- org.slf4j:slf4j-api:jar:1.7.12:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.1.3:compile
[INFO] | +- ch.qos.logback:logback-core:jar:1.1.3:compile
[INFO] | - (org.slf4j:slf4j-api:jar:1.7.12:compile - version managed from 1.7.7; omitted for duplicate)
[INFO] +- org.apache.hbase:hbase-client:jar:0.98.16.1-hadoop1:compile
[INFO] | +- org.apache.hbase:hbase-annotations:jar:0.98.16.1-hadoop1:compile
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile - omitted for duplicate)
[INFO] | | - (junit:junit:jar:4.12:test - version managed from 4.11; scope managed from compile; omitted for duplicate)
[INFO] | +- org.apache.hbase:hbase-common:jar:0.98.16.1-hadoop1:compile
[INFO] | | +- (org.apache.hbase:hbase-annotations:jar:0.98.16.1-hadoop1:compile - omitted for duplicate)
......
[INFO] | +- commons-codec:commons-codec:jar:1.7:compile
[INFO] | +- (commons-io:commons-io:jar:2.4:compile - version managed from 2.1; omitted for duplicate)
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- (com.google.guava:guava:jar:18.0:compile - version managed from 12.0.1; omitted for duplicate)
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | +- io.netty:netty:jar:3.10.6.Final:compile
......

Suggested solutions:

Solution: Declare version com.google.protobuf:protobuf-java:jar:2.6.0 as a direct dependency, to override the version 2.5.0 (based on Maven's nearest wins loading strategy).

Thanks.
Best regards,
Coco

[HelloCoCooo]

@lulu2panpan
@songwenbinasd
Could please help me check this issue?
May I pull a request to fix it?
Thanks again.