Skip to content

Commit

Permalink
known-issues
Browse files Browse the repository at this point in the history
  • Loading branch information
@mendonk
mendonk committed Sep 25, 2024
1 parent dbbe901 commit 73f7c8f
Showing 1 changed file with 144 additions and 42 deletions.
186 changes: 144 additions & 42 deletions modules/install-upgrade/pages/upgrade-guide.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@

Upgrading to Luna Streaming 3.1 should only be done from Luna Streaming 2.10.

Upgrade is fully supported for all the components (connectors included).
Upgrade is fully supported for all the components including connectors.

== Functional changes
== Functional impacts

This section describes changes in Luna Streaming 3.1 that may impact how your deployment functions.

=== Default system topics

Expand All @@ -18,11 +20,10 @@ Changes in Prometheus Metrics:
* Prometheus Metric type UNTYPED is renamed to UNKNOWN
* Metrics have been renamed because OpenMetrics's counter name needs a _total suffix

Renamed Metrics:

.Renamed Metrics
[cols="2,2"]
|===
|Before |After
|Luna Streaming 2.10 |Luna Streaming 3.1

|pulsar_expired_token_count
|pulsar_expired_token_total
Expand Down Expand Up @@ -109,19 +110,20 @@ Renamed Metrics:
|pulsar_txn_append_log_total
|===

==== Related PRs:
The following PRs have been merged to update metrics:

* https://github.com/apache/pulsar/pull/13785[#13785 - Bump prometheus client version from 0.5.0 to 0.15.0]
* https://github.com/apache/pulsar/pull/16581[#16581 - Rename Pulsar txn metrics to specify OpenMetrics]
* https://github.com/apache/pulsar/pull/16610[#16610 - Rename Pulsar schema metrics to specify OpenMetrics]
* https://github.com/apache/pulsar/pull/16611[#16611 - Rename Pulsar lb metrics to specify OpenMetrics]
* https://github.com/apache/pulsar/pull/16591[#16591 - Bump prometheus client version from 0.15.0 to 0.16.0]
* https://github.com/apache/pulsar/pull/17419[#17419 - Removed timestamp from all prometheus metrics.]

=== Pulsar-SQL

If you're upgrading Pulsar SQL from 2.11 or earlier, you should copy config files from `conf/presto` to `trino/conf`.
If you're upgrading Pulsar SQL from 2.11 or earlier, you should copy the config files from `conf/presto` to `trino/conf`.

If you're downgrading Pulsar SQL to 2.11 or earlier from newer versions, copy config files from `trino/conf` to `conf/presto`.
If you're downgrading Pulsar SQL to 2.11 or earlier from newer versions, copy the config files from `trino/conf` to `conf/presto`.

=== Other Functional Impacts

Expand Down Expand Up @@ -160,9 +162,9 @@ If you're downgrading Pulsar SQL to 2.11 or earlier from newer versions, copy co

== Configuration Impacts

=== Dropped in 3.1
=== Removed in 3.1

* https://github.com/apache/pulsar/pull/14506[#14506] removes `managedLedgerNumWorkerThreads` and instead passes the MetadataStore instance from PulsarService directly to the `ManagedLedgerFactory`.
* https://github.com/apache/pulsar/pull/14506[#14506] removes `managedLedgerNumWorkerThreads`. The `MetadataStore` instance is passed from the `PulsarService` directly to the `ManagedLedgerFactory`.

* The `conf/presto` directory has been removed.

Expand All @@ -171,7 +173,7 @@ If you're downgrading Pulsar SQL to 2.11 or earlier from newer versions, copy co
.`broker.conf` and `standalone.conf` values
[cols="1,1,1"]
|===
|Configuration |2.10 Default |3.1 Default
|Configuration |Luna Streaming 2.10 Default | Luna Streaming 3.1 Default

|Managed ledger cache eviction frequency
|`managedLedgerCacheEvictionFrequency=100.0`
Expand All @@ -182,59 +184,159 @@ If you're downgrading Pulsar SQL to 2.11 or earlier from newer versions, copy co
|`managedLedgerMaxUnackedRangesToPersistInZooKeeper=-1`
|===

== Changed in 3.1
=== Changed in 3.1

.`broker.conf` / `standalone.conf`
[cols="3*"]
.`broker.conf` and `standalone.conf` values
[cols="1,1,1"]
|===
|Configuration |2.10 |3.1
|systemTopicEnabled=
|Configuration |Luna Streaming 2.10 Default | Luna Streaming 3.1 Default

|`systemTopicEnabled`
Enable or disable system topic
|false
|true
|topicLevelPoliciesEnabled=
|`topicLevelPoliciesEnabled`
Enable or disable topic level policies (depends on system topic)
|false
|true
|supportedNamespaceBundleSplitAlgorithms=
|range_equally_divide,topic_count_equally_divide,specified_positions_divide
|range_equally_divide,topic_count_equally_divide,specified_positions_divide,flow_or_qps_equally_divide
|The direct memory usage weight when calculating new resource usage.
It only takes effect in the ThresholdShedder strategy.
loadBalancerDirectMemoryResourceWeight=
|`supportedNamespaceBundleSplitAlgorithms`
Supported algorithms for namespace bundle split
|`range_equally_divide`,`topic_count_equally_divide`,`specified_positions_divide`
|`range_equally_divide`,`topic_count_equally_divide`,`specified_positions_divide`,`flow_or_qps_equally_divide`
|`loadBalancerDirectMemoryResourceWeight`
Direct memory usage weight for calculating resource usage in `ThresholdShedder ` strategy
|1.0
|0
|For File System Storage, file system profile path
fileSystemProfilePath=
|../conf/filesystem_offload_core_site.xml
|conf/filesystem_offload_core_site.xml
|For Google Cloud Storage ledger offload, Max block size in bytes. (64MB by default, 5MB minimum)
gcsManagedLedgerOffloadMaxBlockSizeInBytes=
|`fileSystemProfilePath`
File System Storage profile path
|`../conf/filesystem_offload_core_site.xml`
|`conf/filesystem_offload_core_site.xml`
|`gcsManagedLedgerOffloadMaxBlockSizeInBytes`
Max block size in bytes for Google Cloud Storage ledger offload
|67108864
|134217728
|===


== Operational Impacts

=== Upgrade to jdk17
This section describes changes in Luna Streaming 3.1 that may impact how your deployment operates.

The lunastreaming - 3.1 uses jdk17. This changes Pulsar Server modules' javac release version to 17 except client (shared) modules[1].
=== Upgrade to JDK 17

The detail modification is described in the PIP-156, https://github.com/apache/pulsar/pull/15207[#15207]
Luna Streaming 3.1 uses JDK 17. This changes the Pulsar server module's javac release version to 17.

Client and client-server shared modules will remain at the target Java 8 release.

The modification is described in detail in PIP-156 in https://github.com/apache/pulsar/pull/15207[#15207].

=== Removed Python 2 support

Removed Python 2 from build scripts in lunastreamin-3.1. Instead Python3 used in the build image. Updated build image to ubuntu:20.04 (There is no Python 3.7 support in the old Ubuntu). Used python3 instead of python in the executable scripts[1].
Luna Streaming 3.1 removes Python 2 from build scripts.

Python3 is used in the build image.

The build image has been updated to ubuntu:20.04 as there is no Python 3.7 support in the old Ubuntu.

Executable scripts have been updated to use python3 instead of python.

The modification is described in detail in PIP-155 in https://github.com/apache/pulsar/pull/15376[#15376]

=== Updated Prometheus metrics

Prometheus metrics have been updated in Luna Streaming 3.1.

See <<Prometheus metrics>> for details.

==== Renamed Metrics

[cols="2,2"]
|===
|Before |After

|pulsar_expired_token_count
|pulsar_expired_token_total

|pulsar_authentication_success_count
|pulsar_authentication_success_total

|pulsar_authentication_failures_count
|pulsar_authentication_failures_total

|pulsar_source_received_total_1min
|pulsar_source_received_1min_total

|pulsar_source_written_total_1min
|pulsar_source_written_1min_total

|pulsar_source_source_total_1min
|pulsar_source_source_exceptions_1min_total

|pulsar_source_system_exceptions_total_1min
|pulsar_source_system_exceptions_1min_total

The detail modification is described in the PIP-155, https://github.com/apache/pulsar/pull/15376[#15376]
|pulsar_function_received_total_1min
|pulsar_function_received_1min_total

=== Metrics
|pulsar_function_user_exceptions_total_1min
|pulsar_function_user_exceptions_1min_total

|pulsar_function_system_exceptions_total_1min
|pulsar_function_system_exceptions_1min_total

|pulsar_function_processed_successfully_total_1min
|pulsar_function_processed_successfully_1min_total

* Bumped prometheus client version from 0.5.0 to 0.15.0. Around 25 Metrics names have been changed. #13785
* Renamed pulsar_txn* metrics, changed suffix from _count to _total #16581
* Renamed pulsar_schema* metrics, changed suffix from _count to _total #16610
* Renamed pulsar_lb* metrics, changed suffix from _count to _total #16611
* All metrics are listed in functional impact.
* Removed timestamp from all prometheus metrics. https://github.com/apache/pulsar/pull/17419[#17419]
|pulsar_sink_received_total_1min
|pulsar_sink_received_1min_total

|pulsar_sink_written_total_1min
|pulsar_sink_written_1min_total

|pulsar_sink_sink_exceptions_total_1min
|pulsar_sink_sink_exceptions_1min_total

|pulsar_sink_system_exceptions_total_1min
|pulsar_sink_system_exceptions_1min_total

|pulsar_lb_unload_broker_count
|pulsar_lb_unload_broker_total

|pulsar_lb_unload_bundle_count
|pulsar_lb_unload_bundle_total

|pulsar_lb_bundles_split_count
|pulsar_lb_bundles_split_total

|pulsar_schema_del_ops_failed_count
|pulsar_schema_del_ops_failed_total

|pulsar_schema_get_ops_failed_count
|pulsar_schema_get_ops_failed_total

|pulsar_schema_put_ops_failed_count
|pulsar_schema_put_ops_failed_total

|pulsar_schema_compatible_count
|pulsar_schema_compatible_total

|pulsar_schema_incompatible_count
|pulsar_schema_incompatible_total

|pulsar_txn_committed_count
|pulsar_txn_committed_total

|pulsar_txn_aborted_count
|pulsar_txn_aborted_total

|pulsar_txn_created_count
|pulsar_txn_created_total

|pulsar_txn_timeout_count
|pulsar_txn_timeout_total

|pulsar_txn_append_log_count
|pulsar_txn_append_log_total
|===

== Known Issues

Expand Down

0 comments on commit 73f7c8f

Please sign in to comment.