Fix broken downtime comment sync #10000

yhabteab · 2024-02-12T13:51:29Z

All objects must be synced sorted by their load dependency. Otherwise, downtimes and/or comments might get synced before their respective Checkables, which will result in comments and downtimes being ignored by the other endpoint since it does not yet know about their checkables. Given that the runtime config updates event does not trigger a reload on the remote endpoint, these objects won't be synced again until the next reload.

~/master2/icinga2 (bundled-cluster-fixes ✗) ls prefix/var/lib/icinga2/api/packages/_api/*/conf.d/downtimes | wc -l
    3501

~/master2/icinga2 (bundled-cluster-fixes ✗) curl -sSku root:icinga 'https://localhost:5666/v1/objects/downtimes?pretty=1' | grep ' "attrs": {' | wc -l
    1501
~/master1/icinga2 (bundled-cluster-fixes ✗) curl -sSku root:icinga 'https://localhost:5665/v1/objects/downtimes?pretty=1' | grep ' "attrs": {' | wc -l
    3501

After master2 reload:

~/master2/icinga2 (bundled-cluster-fixes ✗) curl -sSku root:icinga 'https://localhost:5666/v1/objects/downtimes?pretty=1' | grep ' "attrs": {' | wc -l
    3501

closes #7786
closes #9873

TODO

lib/remote/apilistener-configsync.cpp

log1-c · 2024-07-22T09:56:32Z

Quick question: would this also fix the following issue:
Downtimes scheduled via API (sometimes) get synced/re-created with delay and are doubled #10078

Mordecaine · 2024-08-13T07:35:29Z

Quick question: would this also fix the following issue: Downtimes scheduled via API (sometimes) get synced/re-created with delay and are doubled #10078

It would be very helpful to get an answer to this.

yhabteab · 2024-08-13T08:21:07Z

Quick question: would this also fix the following issue: Downtimes scheduled via API (sometimes) get synced/re-created with delay and are doubled #10078

It would be very helpful to get an answer to this.

Hi, we don't know for sure whether this will fix #10078 as we still haven't identified exactly what is going wrong there, other than something is not working as expected. It's unlikely that this PR will fix #10078, but we can't tell you for sure until the cause for #10078 is identified.

dgiesselbach · 2024-08-22T10:33:12Z

@yhabteab When will this request be completed? Is there a timeline?

julianbrost · 2024-09-26T13:22:30Z

I believe similar problems will still exist for other types where there's no load_after dependency at the moment: many objects can refer to a time period, however, there's not a single load_after TimePeriod. There are more such examples: Host/Service -> *Command, Notification -> NotificationCommand (not necessarily a complete list).

lib/remote/apilistener-configsync.cpp

Al2Klimov

Yep, I've overseen an elephant, despite I like small diffs. Sorry😅

lib/remote/apilistener-configsync.cpp

Al2Klimov

LFTM

yhabteab · 2024-12-04T07:54:44Z

Fortunately, @julianbrost found out during testing that the objects are not being synchronised in the correct order, as they are supposed to be with this PR. Thus, I made some modifications in the DependencyGraph class to achieve the desired result without producing too many code changes.

Before

[2024-12-04 08:45:47 +0100] critical/config: Error: Validation failed for object 'dsl-groups' of type 'HostGroup'; Attribute 'groups': Object 'dns-groups' of type 'HostGroup' does not exist.
Location: in /Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf: 2:2-2:26
/Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf(1): object HostGroup "dsl-groups" {
/Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf(2):  groups = [ "dns-groups" ]
                                                                                                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^
/Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf(3):  version = 1733298340.337465
/Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf(4):  zone = "master"
[2024-12-04 08:45:47 +0100] critical/config: 1 error
[2024-12-04 08:45:47 +0100] critical/ApiListener: Could not create object 'dsl-groups':
[2024-12-04 08:45:47 +0100] critical/ApiListener: Error: Validation failed for object 'dsl-groups' of type 'HostGroup'; Attribute 'groups': Object 'dns-groups' of type 'HostGroup' does not exist.
Location: in /Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf: 2:2-2:26
/Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf(1): object HostGroup "dsl-groups" {
/Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf(2):  groups = [ "dns-groups" ]
                                                                                                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^
/Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf(3):  version = 1733298340.337465
/Users/yhabteab/ClionProjects/icinga2/prefix/var/lib/icinga2/api/packages/_api/354bb4c7-3d94-4df3-bba3-add86e471cd3/conf.d/hostgroups/dsl-groups.conf(4):  zone = "master"
[2024-12-04 08:45:47 +0100] information/ConfigObjectUtility: Created and activated object 'dns-groups' of type 'HostGroup'.

After

[2024-12-04 08:51:01 +0100] information/ApiListener: Copying file 'master//_etc/overflow.conf' from config sync staging to production zones directory.
[2024-12-04 08:51:01 +0100] information/ApiListener: Copying file 'master//_etc/services.conf' from config sync staging to production zones directory.
[2024-12-04 08:51:02 +0100] information/ConfigObjectUtility: Created and activated object 'dns-groups' of type 'HostGroup'.
[2024-12-04 08:51:02 +0100] information/ConfigObjectUtility: Created and activated object 'dsl-groups' of type 'HostGroup'.
[2024-12-04 08:51:03 +0100] information/Application: Got reload command: Starting new instance.

Al2Klimov

I'd try the below suggestions starting with "💡" myself. (I'm kind of "waiting" for #10254 and #9990.)

Al2Klimov · 2024-12-04T09:10:14Z

lib/base/dependencygraph.cpp

+	Edge edge(parent, child);
+	if (auto it(m_Dependencies.find(edge)); it != m_Dependencies.end()) {
+		if (it->count == 1) {
+			m_Dependencies.erase(edge);


Shouldn't you erase by iterator as you already have it?

Al2Klimov · 2024-12-04T09:10:42Z

lib/base/dependencygraph.cpp

-	m_Dependencies[child][parent]++;
+	Edge edge(parent, child, 1);
+	if (auto it(m_Dependencies.find(edge)); it != m_Dependencies.end()) {
+		m_Dependencies.modify(it, [](Edge& e) { e.count++; });


... just like here?

Al2Klimov · 2024-12-04T09:13:44Z

lib/base/dependencygraph.cpp

+	if (auto it(m_Dependencies.find(edge)); it != m_Dependencies.end()) {
+		m_Dependencies.modify(it, [](Edge& e) { e.count++; });
+	} else {
+		m_Dependencies.insert(edge);


Also I'm wondering whether the new map

either allows to pass an iterator as hint on insertion

or supports kind of insert-if-new-else-return-old-position op

Al2Klimov · 2024-12-04T09:18:31Z

lib/base/dependencygraph.hpp

+	};
+
+	using DependencyMap = boost::multi_index_container<
+		Edge, // The value type we want to sore in the container.


💡 Maybe the stuff mentioned above can be mitigated by using kind of a multimap?

Al2Klimov · 2024-12-04T09:19:48Z

lib/base/dependencygraph.hpp

 	static std::mutex m_Mutex;
-	static std::map<Object *, std::map<Object *, int> > m_Dependencies;
+	static DependencyMap m_Dependencies;


💡 And, while on it, I could even imagine two (multi?)maps – parent-child and child-parent.

Al2Klimov · 2024-12-04T09:22:53Z

lib/base/dependencygraph.hpp

+		// Example: A service object is dependent on a host object, thus the service object is the *parent* and
+		// the host object is the *child* one :).
+		Object* parent;
+		Object* child;


💡 And, while on it, theoretically we could make them ConfigObject*.

But only if that's not too complicated. 😅

Al2Klimov · 2024-12-04T09:26:40Z

lib/base/dependencygraph.cpp


 void DependencyGraph::AddDependency(Object *parent, Object *child)
 {
 	std::unique_lock<std::mutex> lock(m_Mutex);
-	m_Dependencies[child][parent]++;
+	Edge edge(parent, child, 1);


Suggested change

Edge edge(parent, child, 1);

Edge edge(parent, child);

Al2Klimov · 2024-12-04T09:28:21Z

lib/base/dependencygraph.hpp

+			{
+				size_t seed = 0;
+				boost::hash_combine(seed, edge.parent);
+				boost::hash_combine(seed, edge.child);


Oh, this is interesting! I just had XORed two std::hash-es, but good to know.

Al2Klimov · 2024-12-04T09:31:03Z

lib/base/dependencygraph.hpp

+		boost::multi_index::indexed_by<
+			// The first indexer is used for lookups by the Edge object itself, thus it
+			// needs its own hash function and comparison predicate.
+			boost::multi_index::hashed_non_unique<boost::multi_index::identity<Edge>, Edge::HashPredicate, Edge::HashPredicate>,


But why NON unique if you don't use this as multi-map anyway?

yhabteab added bug Something isn't working area/distributed Distributed monitoring (master, satellites, clients) ref/IP area/runtime Downtimes, comments, dependencies, events labels Feb 12, 2024

cla-bot bot added the cla/signed label Feb 12, 2024

yhabteab requested review from julianbrost and Al2Klimov February 12, 2024 13:52

Al2Klimov requested changes Feb 13, 2024

View reviewed changes

lib/remote/apilistener-configsync.cpp Outdated Show resolved Hide resolved

Al2Klimov mentioned this pull request Feb 14, 2024

WIP: Change the order of the API runtime object sync on reconnect #7786

Draft

Al2Klimov self-assigned this Feb 14, 2024

Al2Klimov added this to the 2.15.0 milestone Apr 5, 2024

Al2Klimov removed their assignment Aug 20, 2024

Al2Klimov self-requested a review August 20, 2024 11:16

julianbrost mentioned this pull request Aug 29, 2024

ConfigItem::CommitNewItems(): pre-sort types by their load dependencies once #10003

Closed

julianbrost mentioned this pull request Sep 12, 2024

Sort config types by their load dependencies once #10148

Merged

yhabteab force-pushed the broken-downtime-comment-sync branch from e3c289f to 78095b8 Compare September 16, 2024 07:21

yhabteab changed the base branch from master to enhanced-sort-types-by-load-dependencies September 16, 2024 07:23

yhabteab force-pushed the enhanced-sort-types-by-load-dependencies branch 4 times, most recently from cb4fe57 to eb97676 Compare September 20, 2024 14:18

Al2Klimov changed the base branch from enhanced-sort-types-by-load-dependencies to master September 20, 2024 15:30

Al2Klimov marked this pull request as draft September 20, 2024 15:31

yhabteab removed request for julianbrost and Al2Klimov September 26, 2024 13:30

yhabteab force-pushed the broken-downtime-comment-sync branch from 0435672 to 6f4d8c8 Compare November 4, 2024 15:25

yhabteab requested a review from julianbrost November 4, 2024 15:26

yhabteab force-pushed the broken-downtime-comment-sync branch 2 times, most recently from 1d62919 to 70ccb4c Compare November 4, 2024 16:16

yhabteab requested review from Al2Klimov and julianbrost and removed request for Al2Klimov and julianbrost November 4, 2024 16:17

Al2Klimov reviewed Dec 2, 2024

View reviewed changes

julianbrost reviewed Dec 2, 2024

View reviewed changes

lib/remote/apilistener-configsync.cpp Outdated Show resolved Hide resolved

lib/remote/apilistener-configsync.cpp Show resolved Hide resolved

lib/remote/apilistener-configsync.cpp Show resolved Hide resolved

yhabteab force-pushed the broken-downtime-comment-sync branch 2 times, most recently from 8645d1e to 1261254 Compare December 3, 2024 07:58

yhabteab requested review from julianbrost and Al2Klimov December 3, 2024 07:59

yhabteab force-pushed the broken-downtime-comment-sync branch from 1261254 to 163964f Compare December 3, 2024 10:58

Al2Klimov reviewed Dec 3, 2024

View reviewed changes

lib/remote/apilistener-configsync.cpp Outdated Show resolved Hide resolved

lib/remote/apilistener-configsync.cpp Show resolved Hide resolved

lib/remote/apilistener-configsync.cpp Outdated Show resolved Hide resolved

yhabteab force-pushed the broken-downtime-comment-sync branch 3 times, most recently from 845a61b to 2fd478a Compare December 3, 2024 14:39

yhabteab requested a review from Al2Klimov December 3, 2024 14:41

Al2Klimov previously approved these changes Dec 3, 2024

View reviewed changes

yhabteab dismissed Al2Klimov’s stale review via 3254f36 December 4, 2024 07:14

yhabteab force-pushed the broken-downtime-comment-sync branch from 2fd478a to 3254f36 Compare December 4, 2024 07:14

yhabteab added 3 commits December 4, 2024 08:16

DependencyGraph: Allow lookups by parent & child dependencies

f8e7546

DependencyGraph: Introduce GetChildren() method

de32fa6

ApiListener: Sync runtime configs in order

770b730

yhabteab force-pushed the broken-downtime-comment-sync branch from 3254f36 to 770b730 Compare December 4, 2024 07:16

yhabteab requested a review from Al2Klimov December 4, 2024 07:55

Al2Klimov reviewed Dec 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix broken downtime comment sync #10000

Fix broken downtime comment sync #10000

yhabteab commented Feb 12, 2024 •

edited

Loading

log1-c commented Jul 22, 2024

Mordecaine commented Aug 13, 2024 •

edited

Loading

yhabteab commented Aug 13, 2024

dgiesselbach commented Aug 22, 2024

julianbrost commented Sep 26, 2024

Al2Klimov left a comment

Al2Klimov left a comment

yhabteab commented Dec 4, 2024

Al2Klimov left a comment

Al2Klimov Dec 4, 2024

Al2Klimov Dec 4, 2024

Al2Klimov Dec 4, 2024

Al2Klimov Dec 4, 2024

Al2Klimov Dec 4, 2024

Al2Klimov Dec 4, 2024

Al2Klimov Dec 4, 2024

Al2Klimov Dec 4, 2024

Al2Klimov Dec 4, 2024

Al2Klimov Dec 4, 2024

Fix broken downtime comment sync #10000

Are you sure you want to change the base?

Fix broken downtime comment sync #10000

Conversation

yhabteab commented Feb 12, 2024 • edited Loading

TODO

log1-c commented Jul 22, 2024

Mordecaine commented Aug 13, 2024 • edited Loading

yhabteab commented Aug 13, 2024

dgiesselbach commented Aug 22, 2024

julianbrost commented Sep 26, 2024

Al2Klimov left a comment

Choose a reason for hiding this comment

Al2Klimov left a comment

Choose a reason for hiding this comment

yhabteab commented Dec 4, 2024

Before

After

Al2Klimov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yhabteab commented Feb 12, 2024 •

edited

Loading

Mordecaine commented Aug 13, 2024 •

edited

Loading