Configure HTTP-based ARP information fetching from Palo Alto PIO-OS firewalls using management profiles #3147

jorund1 · 2024-10-18T17:08:48Z

Deprecate the [paloaltoarp] section of ipdevpoll.conf in favor of using management profiles to configure HTTP-based fetching of ARP information from Palo Alto PIO-OS firewalls, analogous to how configuration of SNMP-based fetching is done.

The new management profile protocol to configure HTTP-based fetching has been given the name HTTP API~~HTTP REST API for now, but this name might be missing the mark since there really isn't anything REST-related about the management profile type.~~ Perhaps HTTP WITH API KEY, would be a more fitting name?

codecov · 2024-10-18T17:16:34Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 60.60%. Comparing base (05dfb18) to head (8503acf).
Report is 37 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3147      +/-   ##
==========================================
+ Coverage   60.45%   60.60%   +0.14%     
==========================================
  Files         605      605              
  Lines       43745    43725      -20     
  Branches       48       48              
==========================================
+ Hits        26448    26500      +52     
+ Misses      17285    17213      -72     
  Partials       12       12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

lunkwill42

This is shaping up really nicely!

I'm really glad you took the time to write a proper release notes entry, though I am not sure that we need to delve deeply into rationalizing why we are changing the palo alto API key config stuff. The bit about ARP being available only through the API I believe was explained in an earlier release note too.

You added a separate "make linter happy" commit, which should be unnecessary. These are formatting changes that your pre-commit hooks should have fixed for you in the original commits (you did install the pre-commit hooks, right?)

The "Fix unittests" commit is also probably best if squashed into the already existing commit that updates the tests.

lunkwill42 · 2024-10-24T08:07:10Z

python/nav/models/manage.py

+    def get_http_rest_management_profiles(
+            self, service: str
+    ) -> models.QuerySet:


This needs a docstring

Fixed! (Also moved functionality to the paloalto plugin.)

nav/python/nav/ipdevpoll/plugins/paloaltoarp.py

Lines 161 to 173 in 6c9d4b5

def _paloalto_profile_queryset(netbox: Netbox):

"""

Creates a Django queryset which when iterated yields JSON dictionaries

representing configurations for accessing Palo Alto ARP data of the given

netbox via HTTP. The keys in these dictionaries are the attribute-names of

the :py:class:`~nav.web.seeddb.page.management_profile.forms.HttpRestForm`

Django form.

"""

return NetboxProfile.objects.filter(

netbox_id=netbox.id,

profile__protocol=ManagementProfile.PROTOCOL_HTTP_API,

profile__configuration__contains={"service": "Palo Alto ARP"},

).values_list("profile__configuration", flat=True)

lunkwill42 · 2024-10-24T08:18:52Z

python/nav/ipdevpoll/plugins/paloaltoarp.py

+        for i, api_key in enumerate(api_keys):
+            arptable = yield self._do_request(ip, api_key)
+            if arptable is not None:
+                # process arpdata into an array of mappings
+                mappings = parse_arp(arptable.decode('utf-8'))
+                break
+            self._logger.info(
+                "Could not fetch ARP table from Paloalto device When using API key %d of %d",
+                i,
+                len(api_keys),
+            )


IMNSHO, the key iteration functionality presented here should not really be the concern of the _get_paloalto_arp_mappings() method. It should be factored out.

Moved the loop to the handle() function

nav/python/nav/ipdevpoll/plugins/paloaltoarp.py

Lines 55 to 67 in 6c9d4b5

@defer.inlineCallbacks

def handle(self):

"""Handle plugin business, return a deferred."""

self._logger.debug("Collecting IP/MAC mappings for Paloalto device")

configurations = yield self._get_paloalto_configurations(self.netbox)

for configuration in configurations:

mappings = yield self._get_paloalto_arp_mappings(

self.netbox.ip, configuration["api_key"]

)

if mappings:

yield self._process_data(mappings)

break

lunkwill42 · 2024-10-24T08:33:28Z

python/nav/ipdevpoll/plugins/paloaltoarp.py

-            netbox.sysname in cls.configured_devices
-            or str(netbox.ip) in cls.configured_devices
-        )
+        return netbox.get_http_rest_management_profiles("Palo Alto ARP").exists()


There's a big fat gotcha here: The can_handle() method runs inside the main event loop thread, so you should absolutely not run database queries from this method: They will block everything ipdevpoll is working on until the response comes back from the database.

If you need to access the database at this point, you should defer the lookup to a database thread instead. See

nav/python/nav/ipdevpoll/plugins/lldp.py

Lines 51 to 56 in 518e622

@classmethod

@defer.inlineCallbacks

def can_handle(cls, netbox):

daddy_says_ok = super(LLDP, cls).can_handle(netbox)

has_ifcs = yield run_in_thread(cls._has_interfaces, netbox)

defer.returnValue(has_ifcs and daddy_says_ok)

for a simple example of how.

Thanks for the catch, wrapped database queries appropriately

nav/python/nav/ipdevpoll/plugins/paloaltoarp.py

Lines 81 to 99 in bdc6e99

@classmethod

@db.synchronous_db_access

def _has_paloalto_configurations(cls, netbox: Netbox):

"""

Make a database request to check if the netbox has any management

profile that configures access to Palo Alto ARP data via HTTP

"""

queryset = _paloalto_profile_queryset(netbox)

return queryset.exists()

@classmethod

@db.synchronous_db_access

def _get_paloalto_configurations(cls, netbox: Netbox):

"""

Make a database request that fetches all management profiles of

the netbox that configures access to Palo Alto ARP data via HTTP

"""

queryset = _paloalto_profile_queryset(netbox)

return list(queryset)

lunkwill42 · 2024-10-24T08:58:27Z

tests/integration/ipdevpoll/plugins/paloaltoarp_test.py

+@pytest.fixture
+def paloalto_netbox_1234(db, client):
+    box = Netbox(
+        ip='127.0.0.1',
+        sysname='localhost.example.org',
+        organization_id='myorg',
+        room_id='myroom',
+        category_id='SRV',
+    )
+    box.save()
+    profile = ManagementProfile(
+        name="PaloAlto Test Management Profile",
+        protocol=ManagementProfile.PROTOCOL_SNMP,  # Correct protocol is set with HTTP POST below
+        configuration={
+            "version": 2,
+            "community": "public",
+            "write": False,
+        },
+    )
+    profile.save()
+
+
+    netbox_url = reverse("seeddb-netbox-edit", args=(box.id,))
+    management_profile_url = reverse(
+        "seeddb-management-profile-edit", args=(profile.id,)
+    )
+
+    # Manually sending this post request helps reveal regression bugs in case
+    # HTTPRestForm.service.choices keys are altered; because the post's thus
+    # invalid service field should then cause the django form cleaning stage to
+    # fail. (Changing the HTTPRestForm.choice map to use enums as keys instead
+    # of strings would enable static analysis to reveal this.)
+    client.post(
+        management_profile_url,
+        follow=True,
+        data={
+            "name": profile.name,
+            "description": profile.description,
+            "protocol": ManagementProfile.PROTOCOL_HTTP_REST,
+            "service": "Palo Alto ARP",
+            "api_key": "1234",
+        }
+    )
+
+    client.post(
+        netbox_url,
+        follow=True,
+        data={
+            "ip": box.ip,
+            "room": box.room_id,
+            "category": box.category_id,
+            "organization": box.organization_id,
+            "profiles": [profile.id],
+        },
+    )
+


According to the commit log, you seem to be arguing that this fixture is also a test for the web forms. I would advocate that it shouldn't be. Testing the web forms should be an explicit test function, with a name that makes it abundantly clear what kind of assertion is breaking when it fails.

For the purposes of testing the ipdevpoll plugin itself, it is much simpler for this fixture to just insert the required boxes/profiles directly into the database.

The fixture code as-is could/should be re-used in an explicit management profile form test :)

Totally agree, simplified this fixture and moved form-related parts to a new test file

nav/tests/integration/web/management_profile_test.py

Lines 11 to 61 in 6c9d4b5

class TestServiceChoices:

"""

The HttpApiForm Django form, which creates HTTP API management profiles,

stores a "service" key in the profile's configuration, used by other

parts of NAV to differentiate between HTTP services:

profile.configuration == {"service": "foo", ...}

The HttpApiForm follows the convention of other management profile forms

in not using named constants for ChoiceField values stored in the

configuration dict. To help make sure that other parts of NAV use the

same service values as those set by the form, we add tests below for

each service choice.

The tests first associates a service with a netbox through using the

form, thus asserting that the service is a valid choice. Then it checks

that the other parts of NAV accepts same service value as the one the

form accepted.

"""

tested_services = ["Palo Alto ARP"]

@pytest.mark.twisted

@pytest_twisted.inlineCallbacks

def test_paloalto_plugin_should_use_the_service_choice_used_in_form(

self, localhost, send_add_management_profile_form

):

can_handle = yield PaloaltoArp.can_handle(localhost)

assert not can_handle

# Check that "Palo Alto ARP" is accepted by the form

send_add_management_profile_form(

localhost,

protocol=ManagementProfile.PROTOCOL_HTTP_API,

service="Palo Alto ARP",

api_key="foo",

)

# Check that "Palo Alto ARP" is accepted by the plugin

can_handle = yield PaloaltoArp.can_handle(localhost)

assert can_handle

def test_all_service_choices_should_have_a_test(self):

all_services = HttpApiForm.service.choices

missing_tests = set(all_services) - set(self.tested_services)

error_message = (

"If you've added a service choice to HttpApiForm, please add an "

"integration test that checks that the service is correctly referenced "

f"in other parts of NAV (missing tests for {', '.join(missing_tests)})"

)

assert not missing_tests, error_message

Prior to this commit, the netboxes handled by the PaloaltoArp ipdevpoll plugin used to be configured in the `ipdevpoll.conf` configuration file, but since the netboxes the plugin wants to handle (i.e. collect Arp information from) already should reside in the NAV database, this configuration is now instead done through the SeedDB tool by assigning a HTTP_API ManagementProfile (with `service` set `Palo Alto ARP` and api_key set to some secret API key) to the netboxes to be handled. The prior way to configure the netboxes handled by the PaloAltoArp plugin implicitly only allowed one API key per netbox (both enforced in code but also by the configuration syntax). With ManagementProfiles, it is perfectly possible to assign multiple profiles (e.g. configurations) of the same type but with different parameters (e.g. API keys) to the same netbox. Hence the new way to configure the netboxes allow many API keys per netbox. Thus the semantics of the plugin must change a little: For any given netbox, the plugin now assumes there may be multiple API keys, and uses the ARP results of first API key for which the _do_request method returns a successful response. IMPORTANT: This commit removes the ability to configure the netboxes handled by the PaloaltoArp plugin the NAV version 5.10 - 5.11 way through the `[paloaltoarp]` section in the `ipdevpoll.conf` configuration file.

sonarcloud · 2024-11-08T10:38:33Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

lunkwill42 requested changes Oct 24, 2024

View reviewed changes

jorund1 force-pushed the rest-api-management-profile branch 2 times, most recently from e6d1949 to de813f0 Compare November 8, 2024 07:07

jorund1 added 2 commits November 8, 2024 08:09

Add HTTP_API protocol option in ManagementProfile

d9a5f9c

Add Django form for configuring HTTP_API ManagementProfiles in SeedDB

548900c

jorund1 force-pushed the rest-api-management-profile branch 3 times, most recently from bdc6e99 to 2449d89 Compare November 8, 2024 09:05

jorund1 added 2 commits November 8, 2024 11:34

Update PaloaltoArp documentation

8503acf

jorund1 force-pushed the rest-api-management-profile branch from 2449d89 to 8503acf Compare November 8, 2024 10:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure HTTP-based ARP information fetching from Palo Alto PIO-OS firewalls using management profiles #3147

Configure HTTP-based ARP information fetching from Palo Alto PIO-OS firewalls using management profiles #3147

jorund1 commented Oct 18, 2024 •

edited

Loading

codecov bot commented Oct 18, 2024 •

edited

Loading

lunkwill42 left a comment

lunkwill42 Oct 24, 2024

jorund1 Nov 8, 2024 •

edited

Loading

lunkwill42 Oct 24, 2024

jorund1 Nov 8, 2024 •

edited

Loading

lunkwill42 Oct 24, 2024

jorund1 Nov 8, 2024

lunkwill42 Oct 24, 2024

jorund1 Nov 8, 2024

sonarcloud bot commented Nov 8, 2024

	def _paloalto_profile_queryset(netbox: Netbox):
	"""
	Creates a Django queryset which when iterated yields JSON dictionaries
	representing configurations for accessing Palo Alto ARP data of the given
	netbox via HTTP. The keys in these dictionaries are the attribute-names of
	the :py:class:`~nav.web.seeddb.page.management_profile.forms.HttpRestForm`
	Django form.
	"""
	return NetboxProfile.objects.filter(
	netbox_id=netbox.id,
	profile__protocol=ManagementProfile.PROTOCOL_HTTP_API,
	profile__configuration__contains={"service": "Palo Alto ARP"},
	).values_list("profile__configuration", flat=True)

	@defer.inlineCallbacks
	def handle(self):
	"""Handle plugin business, return a deferred."""
	self._logger.debug("Collecting IP/MAC mappings for Paloalto device")

	configurations = yield self._get_paloalto_configurations(self.netbox)
	for configuration in configurations:
	mappings = yield self._get_paloalto_arp_mappings(
	self.netbox.ip, configuration["api_key"]
	)
	if mappings:
	yield self._process_data(mappings)
	break

	@classmethod
	@defer.inlineCallbacks
	def can_handle(cls, netbox):
	daddy_says_ok = super(LLDP, cls).can_handle(netbox)
	has_ifcs = yield run_in_thread(cls._has_interfaces, netbox)
	defer.returnValue(has_ifcs and daddy_says_ok)

	@classmethod
	@db.synchronous_db_access
	def _has_paloalto_configurations(cls, netbox: Netbox):
	"""
	Make a database request to check if the netbox has any management
	profile that configures access to Palo Alto ARP data via HTTP
	"""
	queryset = _paloalto_profile_queryset(netbox)
	return queryset.exists()

	@classmethod
	@db.synchronous_db_access
	def _get_paloalto_configurations(cls, netbox: Netbox):
	"""
	Make a database request that fetches all management profiles of
	the netbox that configures access to Palo Alto ARP data via HTTP
	"""
	queryset = _paloalto_profile_queryset(netbox)
	return list(queryset)

	class TestServiceChoices:
	"""
	The HttpApiForm Django form, which creates HTTP API management profiles,
	stores a "service" key in the profile's configuration, used by other
	parts of NAV to differentiate between HTTP services:

	profile.configuration == {"service": "foo", ...}

	The HttpApiForm follows the convention of other management profile forms
	in not using named constants for ChoiceField values stored in the
	configuration dict. To help make sure that other parts of NAV use the
	same service values as those set by the form, we add tests below for
	each service choice.

	The tests first associates a service with a netbox through using the
	form, thus asserting that the service is a valid choice. Then it checks
	that the other parts of NAV accepts same service value as the one the
	form accepted.
	"""

	tested_services = ["Palo Alto ARP"]

	@pytest.mark.twisted
	@pytest_twisted.inlineCallbacks
	def test_paloalto_plugin_should_use_the_service_choice_used_in_form(
	self, localhost, send_add_management_profile_form
	):
	can_handle = yield PaloaltoArp.can_handle(localhost)
	assert not can_handle

	# Check that "Palo Alto ARP" is accepted by the form
	send_add_management_profile_form(
	localhost,
	protocol=ManagementProfile.PROTOCOL_HTTP_API,
	service="Palo Alto ARP",
	api_key="foo",
	)

	# Check that "Palo Alto ARP" is accepted by the plugin
	can_handle = yield PaloaltoArp.can_handle(localhost)
	assert can_handle

	def test_all_service_choices_should_have_a_test(self):
	all_services = HttpApiForm.service.choices
	missing_tests = set(all_services) - set(self.tested_services)
	error_message = (
	"If you've added a service choice to HttpApiForm, please add an "
	"integration test that checks that the service is correctly referenced "
	f"in other parts of NAV (missing tests for {', '.join(missing_tests)})"
	)
	assert not missing_tests, error_message

Configure HTTP-based ARP information fetching from Palo Alto PIO-OS firewalls using management profiles #3147

Are you sure you want to change the base?

Configure HTTP-based ARP information fetching from Palo Alto PIO-OS firewalls using management profiles #3147

Conversation

jorund1 commented Oct 18, 2024 • edited Loading

codecov bot commented Oct 18, 2024 • edited Loading

Codecov Report

lunkwill42 left a comment

Choose a reason for hiding this comment

lunkwill42 Oct 24, 2024

Choose a reason for hiding this comment

jorund1 Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

lunkwill42 Oct 24, 2024

Choose a reason for hiding this comment

jorund1 Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

lunkwill42 Oct 24, 2024

Choose a reason for hiding this comment

jorund1 Nov 8, 2024

Choose a reason for hiding this comment

lunkwill42 Oct 24, 2024

Choose a reason for hiding this comment

jorund1 Nov 8, 2024

Choose a reason for hiding this comment

sonarcloud bot commented Nov 8, 2024

Quality Gate passed

jorund1 commented Oct 18, 2024 •

edited

Loading

codecov bot commented Oct 18, 2024 •

edited

Loading

jorund1 Nov 8, 2024 •

edited

Loading

jorund1 Nov 8, 2024 •

edited

Loading