Next Major release?

[J-N-K]

I would like the AC (as stated on the Governance Page) to discuss a new major release.

This should include the following breaking changes that affect the whole openHAB system:

• Minimum requirement Java 17. While a lot of progress has been made for running openHAB 3 under Java 17 (Add support for running openHAB with Java 17 #1316), this is different from having Java 17 as minimum requirement. IMO the code could really benefit from new language features like records, improvements to the Stream interface or instanceof pattern matching. This is breaking only for running newer add-ons on older core-versions.
• I'm working on Introduce add-on metadata openhab-core#2058 which will greatly improve the the way information about add-ons is made available. Since this involves changes to all add-ons metadata, it will not be possible to use older versions of add-ons on newer core-versions or vice-versa. I would prefer not to have the "old" and "new" metadata in parallel as it adds quite some complexity to the implementation. The conversion of the add-ons in openhab-addons can probably be automated by a script, at least in a way that the add-ons work, without making use of the new features.
• Removal of all deprecated interfaces, classes and methods. We have accumulated quite a lot of @Deprecated annotations over time and a new major version would be a good time to remove them.
• Decouple API and implementation in more places. E.g. the MqttBrokerConnection or the AbstractWatchService/WatchQueueReader mix API and implementation (and should IMO be services anyway), making it very difficult to improve implementation details without breaking compatibility.

[rkoshak]

I don't have an opinion one way or the other. These changes all sound like good ideas. But from an end user's perspective, they are somewhat underwhelming when compared to the changes between OH 1 and OH 2 and then from OH 2 to OH 3. I don't know that that's a criteria worth considering but it will be something that will be noted by the end users.

One impact that would need to be addressed is that Nashorn will go away as a default. That means breaking changes to existing JavaScript transformations and a good deal of work will need to be done to migrate Blockly to "compile" to some other language. I'd assume that other language would be JS Scripting since it would be the least amount of work to migrate to that. However, that means that Blockly, which comes by default, has a dependency on an optional add-on. So JS Scripting will need to come by default too (if that's chosen as the new underlying language) or Blockly will need to be disabled until JS Scripting is installed.

[J-N-K]

I’m not sure either, that’s why I put it before the AC. And yes, the user-facing changes are not that big.

I agree that prior to a switch to Java 17 the Blockly issue has to be resolved. We could either add the Nashorn add-on to core (similar to the DSL scripting engine), suggest the installation during setup, disable Blockly if it is not available (that‘s a bad decision in my opinion for upgrades) or figure out if we can install it during the upgrade process.

I personally would vote for removing the DSL scripting engine from core and adding it to addons, too. That would keep core totally agnostic of the scripting engine.

A point for next major version when increasing the minimum Java version is: OH2 is Java 8, OH3 is Java 11 and OH4 is Java 17. That’s super-easy to understand. Combined with the incompatibility of the add-ons this makes sense for me.

[wborn]

There is still more to fix for full Java 17 compatibility according to the Xtext Masterplan. So it would be nice to know how well Xtext works when Java 17 is used compile time.

I think it is best to only update OH to a new major version if there are big user facing (breaking) changes. In the past we've also never guaranteed add-ons to work between any OH release. When we upgrade certain dependencies like UoM or switch to OSGi R8 (with Karaf 4.4.x) it will also break add-on compatibility again without noticable user changes.

Users will most likely also hesitate to immediately upgrade to a new major version (for good reasons). So there should be a big reward waiting for them if they finally decide to upgrade to a new major version. 🙂

[J-N-K]

Xtext is another issue. After reading quite some issues in their repository and the website (releases only if paid or needed by itemis' customers) , I'm not too sure we'll ever see Java 17 source compatibility with Xtext. There is a scheduled release 2.28 for beginning of December which has Java 17 support on the list, but that is also true for 2.27. There are also discussions about dropping LSP support to keep Java 8 compatibility: eclipse-xtext/xtext#1972. If that happens I would vote for a rewrite of the parsing code for textual configuration and drop Xtext completely (there is the issue that DSL rules depend on it).

For my test-branch I have raised source level to 17 and lowered it to 11 for the model bundles. That at least compiles fine and should not impose any problems as long as we keep the new features out of the API and use it only in the implementation.

My suggestion was not for the next release (which is probably 3.4), I was thinking about the long-term. But I'm fine with staying on 3.x, too.

[rkoshak]

I've been worried about Xtext for some time. It would indeed be a huge breaking change if we lost Rules DSL, up there with the removal of the 1.x Compatibility Layer. That certainly would be a big user facing change, but not really a good one, especially for long time users. Losing the LSP would be a big hit for lots of users too who rely on VSCode and the plugin.

I like the idea of moving Rules DSL to it's own add-on for lots of reasons. But I think we need something to be the default. Blockly with what ever underlies Blockly (Nashorn or whatever Blockly is upgraded to support) would be my vote. Right now that would be @wborn's add-on on the marketplace I think. Though I'd really like to see Blockly eventually move to one of the more modern languages, even if it's not JS Scripting.

I think it's clear that there will need to be a next version of OH at some point. The changes suggested above make sense to be part of it. So I guess it's a matter of when we start working towards that release?

[ghys]

I think carving out the Rules DSL into its own add-on and eventually making it opt-in (as a "legacy" option, if it remains compatible at all with mandated the JRE version) rather than bundled-in makes a lot of sense.

Since we couldn't possibly ship distributions with no scripting support, installing GraalVM by default in new installations as the default script engine (via inclusion in the standard package), the same way rrd4j is automatically installed as the default persistence service, seems a good option. It clearly has the most momentum (with modern, maintained versions of both the language - ECMAScript - as well as the runtime - GraalVM - and the openhab-js library) compared to the other candidates (reaslistically those are Groovy and JRuby, I'm excluding Jython on purpose).

Additionally novice-friendly Blockly can be made to work with it, IMHO important for new installations.
For upgrades, installing GraalVM automatically after Nashorn is dropped and installing a potential "legacy DSL scripting" add-on if necessary could be nice, but not a must - it could simply be a breaking change warning, installing add-ons is easy enough.

[digitaldan]

I would love to see Java 17 features available in addons (pattern matching switch statements look especially cool), but I'm not in the core enough to have an educated opinion on what the consequences might be, so i'll leave that to others.

I think making the DSL optional (even if its installed by default) sounds like a good plan for eventual phasing out. I also agree that JSScripting is the best option for a default scripting option, and is why I was so eager to see that included in our last major release (along with openhab-js).

[kaikreuzer]

Sorry for not commenting earlier, I lay in bed with feaver the last two days. :-(

I think most things have been said and I agree with them.
All suggested changes make sense, but I also see them as "technical" changes that do not necessarily mean that we have to switch to a new major version (as we do not treat the major versions as semver for code compatibility, but rather to signal major evolution towards the end user). Therefore, I wouldn't see a problem to change to "addon.xml"s, remove deprecated code and refactor certain services within an upcoming 3.4 release. As @wborn mentions, we so far never guaranteed a compatibility of older add-ons against a newer core. For existing users, all of these changes should have hardly any impact when updating their system to 3.4.

With Java 17, the situation is of course different as users would need to update their installed JVM, so here it makes sense to only do this step with a next major version. But this alone would imho not justify a new major version. As @wborn points out, we should make sure that a new major version does not only come with efforts and removed features for users, but really bring some cool new stuff that makes the version stick out as something new towards the user - and I am not yet sure what that could be, so I'm still reluctant to head for a new major version.

While I know that new language features are always desirable for developers, we were usually quite conservative with increasing the JVM minimal version. We did the move from 8 to 11 only after there were no security updates for Java 8 anymore and the switch was therefore unavoidable. Looking at the EOL dates of Java 11, this is still under full active support until Sept. 23 and with security patches until Sept. 26. My feeling would be that if we aim for an openHAB 4.0 by end of 2023, we are still on the safe side.

Wrt moving towards GraalVM, I agree that this makes sense on the long run as this is where the activity of the Java community is focussing on.

[J-N-K]

What is the reason to stay on outdated (even if still supported) Java versions?

• Java 17 is LTS and available for all major platforms.
• It has no other system requirements than Java 11.
• It's far more likely that a new installation has Java 17 than Java 11.
• Upgrading Java from 11 to 17 is really easy and upgrading openHAB itself also requires manual changes due to breaking changes, so it's just one more step.

The only reason would be that there are consumers of our code that want to stay on 11. But I know of no such distribution.

[ghys]

I'd say the reasons are more practical than ideological - probably everyone would welcome a switch to Java 17 if it weren't for Nashorn (there's a replacement for scripting in rules but not for JS transformations yet) and Xtext.

[J-N-K]

There is a replacement for the transformation (openhab/openhab-core#2883, which needs only a little fine-tuning openhab/openhab-core#2990) and a replacement for the profile is currently in review (openhab/openhab-core#2872). Both work together with every available scripting language, be it Nashorn, DSL or Jython. The issues with the profile PR are mostly related to the UI representation, your input would be very welcome, @ghys.

Xtext is a minor issue at the moment, because we can build the model-bundles with source-level 11 (openhab/openhab-core#2994). This only prevents using Java 17 features (like records) in exported API, but we could still benefit in all implementation classes.

[ghys]

The issues with the profile PR are mostly related to the UI representation, your input would be very welcome, @ghys.

Indeed - I'm just now seeing this discussion. I guess I'll have to adapt my workflow. I'll gather some thoughts and reply there.

[rkoshak]

What is the reason to stay on outdated (even if still supported) Java versions?

I'd say the biggest reason is it will cause a good deal of extra work for a number (not all) end users.

• Docker users will be best off. Just update the image and we are good to go.
• openHABian users are probably the next best off. openHABian can almost certainly be updated to install and configure Java 17.
• I don't know if apt/yum can do this or not. Based on the docs, when installing that way from scratch they expect Java to already installed so I suspect they will need to manually install Java 17 (either truly manually or via the package manager) and configure it as the default.
• Synology (and QNAP?) users who are not using Docker will be out of luck as it looks like Java 11 is as late as is supported.
• Everyone else will definitely need to install and configure Java 17 manually before upgrading openHAB.

I think the Nashorn going away issue is solvable with those PRs and including the Nashorn add-on by default so it really boils down to losing Synology support and making this upgrade more involved than is usual.

[J-N-K]

For apt it's just sudo apt install zulu-17 followed by sudo update-alternatives --config java and selecting the required default java version.

There is a very fine documentation for managing multiple java versions on Win/Mac/Linux: https://docs.azul.com/core/zulu-openjdk/manage-multiple-zulu-versions/linux

[J-N-K]

I did check that the latest Java on QNAP seems to be 8, so they already need to use tricks for running OH3 anyway. So the only issue left are users with Synology that do not use Docker (there are instructions on the net - at least in German - for installing Java 17, and it's only five commands or so).

IMO the benefits of Java 17 for the development process outweigh the hassle for those users. It seems that @kaikreuzer and @rkoshak would vote against, @ghys abstains (or has at least not expressed an opinion) and @wborn and @digitaldan are in favor of changing Java version after the release. What does this mean now?

[wborn]

Another option could be to use Java 17 for all source code but compile to Java 11 for compatibility.

Then you change the compiler source/target in the POM to something like:

    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>11</maven.compiler.target>

Note that Java 17 will also solve issues our users run into today, like:

• JVM crashes (openhab-core#2032)
• Missing crypto ciphers (openhab-addons#12861)

can't easily, upgrade to Java 17

If it is Linux based, the Zulu 17 JDK can be used on any system that has glibc 2.5 (or newer). glibc 2.5 has been available since 2006.

[kaikreuzer]

If it is an option to compile to Java11 without loosing the language features, this would indeed be the best compromise. It brings developers the language features, while not forcing the users to upgrade without seeing any direct benefit.

In general I share @rkoshak's opinion: I don't want to hinder any development progress, but am also concerned about imposing work to our userbase without offering any obvious benefits. We should not forget that there are many not that tech-savvy users out there, who simply will shy away from an update and stay with an outdated openHAB version for a long time or maybe moving away from it completely.

[J-N-K]

Unfortunately that will not work. I checked that it'll not work to use record if you set source to 17 and target and release to 11. The documentation also states that it is not possible:

Note: Merely setting the target option does not guarantee that your code actually runs on a JRE with the specified version. The pitfall is unintended usage of APIs that only exist in later JREs which would make your code fail at runtime with a linkage error. To avoid this issue, you can either configure the compiler's boot classpath to match the target JRE, or use the [Animal Sniffer Maven Plugin](http://www.mojohaus.org/animal-sniffer/animal-sniffer-maven-plugin/) to verify your code doesn't use unintended APIs, or better yet use the [release option supported since JDK 9](https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-release.html). In the same way, setting the source option does not guarantee that your code actually compiles on a JDK with the specified version. To compile your code with a specific JDK version, different than the one used to launch Maven, refer to the [Compile Using A Different JDK](https://maven.apache.org/plugins/maven-compiler-plugin/examples/compile-using-different-jdk.html) example.

But I still think that the burden on the user is next to nothing:

• For openHABian it should just be an option to switch the java version (similar to what we had when we switched from 8 to 11)
• Docker containers should automatically be upgraded (I don't have much experience with docker, but from my understanding that should be easy)
• Plain Debian/Ubuntu users need to type sudo apt upgrade openhab to upgrade. Additionally doing sudo apt install zulu-17 followed by sudo update-alternatives --config java` should not be that hard if you found your way to the console.
• Windows/Mac users need to install Java manually, too. Upgrading is in no way harder than upgrading openHAB. It's just one more step.
• It may be difficult in some specific setups, but in that case the installation will also be harder, so we can assume that users with such setups are more experienced.

That users will refrain from making an upgrade if the minimum Java version is increased is IMO not an argument. We have users running 3.0 not upgrading to 3.1 even though the Java version stayed the same (or they still run completely outdated 2.4 instead of 2.5, even if both require Java 8). Reasons for not upgrading are more that user fear the breaking changes we have in every release (prominent example for 3.3 will be netatmo, which requires a full re-configuration). Not bumping Java will only shift the problem to the future: If users don't upgrade because of a different Java version, they'll not upgrade for the same reason a year in the future.

On the other hand the upgrade allows to use language features that'll lead to cleaner code (and by that reduce bugs, which also affects the user) and reduce code complexity (which will allow us to deploy new features with less work for the developer, so more work can be done in the same time).

[rkoshak]

But I still think that the burden on the user is next to nothing:

Well, the burden on the end users of Bintray going away was next to nothing too. Yet we still have a steady stream of users coming to the forum running ancient versions of OH complaining they can't download add-ons any more. I think it's far to easy to underestimate the disruption caused by even minimal extra work like this. It's going to be people like me and rossko57 and and other helpers on the forum who will face the largest burden of them all.

Ultimately, the issue isn't that it's hard to do the upgrade. The issue is far too many users won't even know they need to do so. It fails the "idiot in a hurry" test. We have lots of those among our OH user base. These users are used to being on the release repo and blindly running apt upgrade periodically. Since it's six months between releases these users will be caught by surprise. Surprised users are usually angry users. And we don't have much to offer to ameliorate that surprise.

That's my concern. It's the surprise more than the actual amount of work and no user facing improvements to justify the surprise.

[J-N-K]

Can't fix stupid. But:

ALERT;BMW ConnectedDrive Binding: BMW replaced ConnectedDrive Smartphone App with MyBMW Application. Underlying API is now disabled and the Binding isn't working anymore. Switch to new introduced MyBMW Binding recommended.
ALERT;Daikin Binding: The channel 'specialmode-powerful' has been removed, and a r/w channel 'streamer' added. The channel 'specialmode' changed to read/write to control eco/powerful mode.
ALERT;DanfossAirUnit Binding: The type of the channel 'humidity' was updated. You may need to remove and create your things again in case your things were previously created using UI and you want to use this channel.
*ALERT;JS Scripting Automation: 'item.history.lastUpdate()' returns 'time.ZonedDateTime' instead of 'Date'
*ALERT;JS Scripting Automation: 'addItem(...)' and 'updateItem(...)' use new itemConfig as parameter and 'createItem(...)' was removed
ALERT;MQTT Binding: Support for the MQTT System Broker has been removed. Replace 'mqtt:systemBroker' things with 'mqtt:broker' things.
ALERT;Netatmo Binding: Things hierarchy and organization has changed as well as channel namings. You will need to remove and create again these things.
*ALERT;OmniLink Binding: The channel 'sysdate' has been renamed to 'system_date', and is now read only, to make these changes visible you may need to remove and recreate the `controller` thing. To synchronize the controller time there is now a new action synchronizeControllerTime that can be used with a time zone input (e.g. 'America/Denver').
*ALERT;Opentherm Gateway Binding: The 'otgw' Thing has been removed and split into a new 'openthermgateway' Bridge, and a new 'boiler' Thing. You will need to change your Thing and respective Item definitions.
*ALERT;RFXCOM Binding: The channel 'fanSpeed' of 'fan_lucci_dc' and 'fan_novy' has been renamed to 'fanSpeedControl', and 'fan_lucci_dc' has a new numeric channel 'fanSpeed'. You may need to remove and create your things again.
*ALERT;TapoControl Binding: L510_Series and L530_Series Things were renamed to L510 and L530 because of manufacturer changed naming with new HW-rev. You may need to remove and create again these things.
*ALERT;WLED Binding: Binding now uses Bridge and Thing structure. Delete and add a new bridge, and at least 1 thing for each segment you require.
*ALERT;Helios Binding: Firmware versions older than v2.21 are no longer supported.

This is the breaking changes list for 3.3.0. I marked some changes with a * that'll break the installation for users who just do apt upgrade. In contrast to changing to a new Java version (which will completely prevent starting openHAB), it might be unnoticed that something just stopped working and it might take weeks until the user finds out why something is behaving in a strange way. I would like to ask: what is the big user-facing benefit of re-naming channels and things-types or changing the return-type of .lastUpdate that breaking a user installation is justified. All of these changes are harder to fix than upgrading Java (e.g. it took me about 30 minutes to re-create the things and re-link all channels after the Netatmo change). And why was that argument not brought forward in these PR?

[ghys]

I like the clarity of tying Java versions to major releases.

To give a purely chronological perspective, the table below tries to summarize when these major releases happened along with that of the supported Java version:

OH Version	OH Release Date	Java Version	Java Release Date*	Java EoL Date*
2.0	Jan 2017	8	Mar 2014	Mar 2022
3.0	Dec 2020 (+~4y)	11	Sep 2018	Sep 2023
4.0	Mid-2023? (+~2.5y)	17	Sep 2021	Sep 2026

*EoL denotes end of active/"Premier" support, according to:
https://endoflife.date/java & https://www.oracle.com/java/technologies/java-se-support-roadmap.html

I would argue it's important to release a major version before the currently supported Java version reaches EoL (with some leeway, at least a few months), while also not jumping on a new version and let it mature first - for 2.0 & 3.0 the supported Java version had already been released for 2/3 years.
I think we ought to try releasing 4.0 with ideally major new features (not sure yet what that would be) and supporting Java 17 sooner than 4 years after 3.0 - the time gap between 2.0 & 3.0 - I suggested mid-2023 above to do it reasonably before Java 11 reaches EoL but it could be at the end of this year as well.

Edit: also note that 4.0.0-SNAPSHOT and milestone versions would already target Java 17 so there would be at least 6 months of Java 17 support before GA.

[hmerk]

Ok, please let me too put my two cents into this.
Looking for some really great new features as some kind of another reason for a new major release,
what about having a deep look at what was posted in the community regarding RBAC.

https://community.openhab.org/t/rbac-model-in-openhab-and-potential-security-vulnerability-found/136419

It sounded to me like the OP had a working solution ind we could try to implement this.
This is a long wanted feature for many users.
WDYT ?

[cdjackson]

"if this shouldn't be a change made at OH4", that's exactly why I proposed aiming at OH4: OH2 -> Java 8, OH3 -> Java 11, OH4 -> Java 17 is very easy to understand.

Sorry - I missed this then. I thought that the discussion was to change in OH3, but if I've misread this then that's fine with me and I think is a good approach 👍 .

[rkoshak]

And why was that argument not brought forward in these PR?

Those are all breaking changes in add-ons. Traditionally breaking changes in add-ons are treated as much less of a problem than breaking changes in core. As such, the reaction to them by the end users is much less subdued. Also, changes to any one binding is only going to affect a minority of users, further reducing the size of the angry mob.

But it's kind of pointless to argue it further. To summarize my thoughts, releasing an OH 4 where the only significant change is a move to Java 17 with no significant user facing improvements and/or features will have poor adoption rates and anger a number of users. People don't like to have work forced on them just to stay where they already are.

But I won't stand in the way of that happening.

I suggested mid-2023 above to do it reasonably before Java 11 reaches

That seems a reasonable timeframe. That basically gives a year to make the switch and get some new features included to make the switch a little more palatable.

what about having a deep look at what was posted in the community regarding RBAC.

That would be a big addition. Some other things I've seen talked about that could help pay for the disruption:

• significant and measurable improvements in RAM or other performance parameters ("hey! now you can run it on an RPi 0w!")
• independent binding versioning so users have more flexibility in the installation and use of add-ons that are not the exact same version as OH core (that will probably make the add-on devs happier but I can sell it to end users too)
• significant improvements/new features for text based configs like the concept of macros in .things files
• major additions to rules along the lines of adding time based actions to pure UI rules (e.g. adding a sleep between actions), passing data between conditions/actions, making Thing Actions callable from UI rules directly, etc.
• rework the semantic model or otherwise figure out to make it user definable or easier to extend and adjust
• REST API changes and changes to the core so that we can get a reasonable "code" tab implemented for Items like we have for everything else
• Maybe it's time for a new Item type or two

Most if not all of these already have issues (sometimes more than one) that have been open for quite some time and are kind of stuck. I'm not saying it has to be all or even any of these, but these are the sorts of fixes and features that will make it easier to sell an OH 4, ways to pay for the surprise. If we can move two or three of these (or things like these) over the finish line as part of an OH 4 it will be a lot easier to say upgrading to OH 4 is worth it and the upgrade pays for the surprise.

NOTE: I bet there are a ton of features and irksome things could be addressed for the add-on developers too. I get the feeling that some of them are feeling left out since most of the focus for awhile has been on end user facing stuff and not add-on API facing stuff.

Moving to a new OH version is an opportunity to be bold and audacious. Just moving Java versions feels like a big missed opportunity. An opportunity that won't come again for years (given past release schedules).

But again, if that's all we want to do, I won't stand in the way by voting no.

[digitaldan]

As much as i think Java 17 would be great to have, i am very ok waiting for 4.0 to have it. I agree the benefits are not so great as to offset the disruption to users this may cause (although i get a chuckle when asked if i can port a binding feature back to 2.x) . So my vote is to stay on 11 for now and plan for a major version change in 4.0 (which is not that far away).

[kaikreuzer]

I just learnt about another incompatibility wrt Java 17: I tried to update my production system to a 4.0.0-SNAPSHOT when I noticed that JDK 17 isn't available on macOS < 10.12 anymore - my Mac mini only runs macOS 10.11 and cannot be upgraded. 😢
So after 12 years running openHAB on this nice piece of hardware, I'll have to replace it. Not really our fault, but I thought it might be interesting to share that "Java = run everywhere" does not really hold true anymore, as newer JDKs are only built for newer OSes.

[digitaldan]

How about putting Debian on that ?
https://wiki.debian.org/MacMiniIntel

[kaikreuzer]

Well, I am running more things on it and am also using the MacTTS service. So I have chosen the quicker (while more expensive) road: I've ordered a more recent Mac mini...

[J-N-K]

Good to know. I was about to order an used MacMini to migrate from my old PC Engines APU.

[kaikreuzer]

FTR: I've chosen a late 2014 model which arrived today. I am running Monterey (macOS 12.6.2) on it and I am back in the game again with Java17 and openHAB 4.0.0-SNAPSHOT. 😄
Energy consumption is around 7W which is less than half of my old one - and it is blazingly fast in comparison to a RaspPi. So all in all a choice I can recommend!