Update for NetBox 4.0.7; Add more logging, error checking; Fix #134, …

[NicJames2378]

Verified working with:

• NetBox 4.0.7
  • hosted in Docker
  • Running on CentOS Stream 8
• Python 3.12.4
  • Running under Windows 11 22H2

[brandan-schmitz]

I tried running your PR and I encountered a few issues. I outlined what I did to fix those here.

[brandan-schmitz]

It also seems that some of the changes for retries upon connection errors is not working 100% as I have had instances where the Netbox server gets temporarily overloaded and returns a 502 connection error and the importer is not catching this and it causes it to crash. Previous versions would simply continue to the next request.

Traceback (most recent call last):
  File "/app/nb-dt-import.py", line 68, in <module>
    main()
  File "/app/nb-dt-import.py", line 30, in main
    netbox.create_device_types(device_types)
  File "/app/netbox_api.py", line 138, in create_device_types
    self.device_types.create_interfaces(device_type["interfaces"], dt.id)
  File "/app/netbox_api.py", line 281, in create_interfaces
    existing_interfaces = {str(item): item for item in self.netbox.dcim.interface_templates.filter(
  File "/app/netbox_api.py", line 281, in <dictcomp>
    existing_interfaces = {str(item): item for item in self.netbox.dcim.interface_templates.filter(
  File "/usr/local/lib/python3.9/site-packages/pynetbox/core/response.py", line 128, in __next__
    next(self.response), self.endpoint.api, self.endpoint
  File "/usr/local/lib/python3.9/site-packages/pynetbox/core/query.py", line 292, in get
    req = self._make_call(add_params=add_params)
  File "/usr/local/lib/python3.9/site-packages/pynetbox/core/query.py", line 259, in _make_call
    raise RequestError(req)
pynetbox.core.query.RequestError: The request failed with code 502 Bad Gateway but more specific details were not returned in json. Check the NetBox Logs or investigate this exception's error attribute.

[NicJames2378]

I'm having trouble getting the 502 issue to reproduce on a fresh test environment, so that one may have to wait a week or so for me to get back to my dev environment. To assist me in reproducing, could you provide the NetBox logs from the time around the import? Additionally, can you confirm NetBox is still usable from the web both before and after performing an import? This is to verify that is a connectivity issue to the importer and not an issue in NetBox itself going offline.

As an alternative possibility, it looks like this may possibly be solved by amending netbox_api.py on line 164 to read as except (http.client.RemoteDisconnected, requests.exceptions.ConnectionError, pynetbox.RequestError) as e: instead. This will only help if the issue is caused by the connection dropping after the device type is created but before the interfaces/ports/etc are. I didn't account for nor encounter this possibility initially, so it's possible I failed to catch a disconnection.

Let me know if you try the latter, and if it works I'll update the pull request with the new code.