Commit Graph

242 Commits

Author SHA1 Message Date
Oliver Gorwits
6b80626a29 better job queue stats in web 2023-10-27 15:32:06 +01:00
Oliver Gorwits
2536bfc12f fix bug in worker runner 2023-09-27 14:16:17 +01:00
Oliver Gorwits
3a160206a1 make sure internal plugin asciibetical load order is preserved 2023-09-27 14:13:40 +01:00
Oliver Gorwits
993edd0c6a ACL support for scheduled jobs (#1106)
implements #580
2023-09-27 12:03:49 +01:00
Oliver Gorwits
a8db56fa8c add internal worker plugin to set fqdn for cli actions 2023-09-26 09:27:23 +01:00
Oliver Gorwits
3a820c06e4 #1084 move slow skiplist build to the first job on running backend 2023-09-05 22:55:33 +01:00
Oliver Gorwits
82e084a87f #1095 fix with multiple netdisco-do -d, snmpfastdiscover runs multiple times 2023-08-20 21:39:23 +01:00
Oliver Gorwits
bb15abce58 allow deferrable_actions exceptions to device deferrals 2023-08-12 19:51:05 +01:00
Oliver Gorwits
a78870c49f Revert "revert log from internal worker so other failed worker log appears"
This reverts commit 6757bc2d94.
2023-08-11 14:41:26 +01:00
Oliver Gorwits
6757bc2d94 revert log from internal worker so other failed worker log appears 2023-08-11 13:48:08 +01:00
Oliver Gorwits
c4ca7fa1da make sure internal workers run before others 2023-08-11 13:47:47 +01:00
Oliver Gorwits
0ff88a8be2 fix debug messages using -> in wrong place 2023-08-11 13:31:28 +01:00
Oliver Gorwits
f01f03caa4 check loadmibs has run before running snapshot 2023-08-11 13:29:10 +01:00
Oliver Gorwits
5efcdd7707 better status reporting from SNMPFastDiscover 2023-08-11 13:28:55 +01:00
Oliver Gorwits
b9041f87b2 fix bug with internal actions overriding job actions 2023-08-11 13:28:28 +01:00
Oliver Gorwits
21afacccc4 #1087 netdisco-do show handles qualified MIB leafs 2023-08-11 10:24:05 +01:00
Oliver Gorwits
9eb537a4c1 #910 implement import of snmpwalk and more robust snapshot handling (#1086)
* initial work

* initial work

* initial work

* some fixes for time and Layer3 weird spec

* store again the snapshot after update for specific

* resolve the enums

* monkeypatch SNMP::translateObj to avoid hardware exception on macOS

* only save cache to db in the late phase worker

* no need to check for cache on transport, can just go ahead and try

* use database only for oidmap instead of netdisco-mibs

* rewrite device snapshot to gather loaded mib leafs only

* remove old walker code from snapshot worker

* allow snmp browser to work without snapshot

* only store snapshot leafs which the device responded on

* refactor to separate snapshot work from snmp transport work

* refactor to separate snapshot work from snmp transport work

* allow typeahead on MIB qualified leafs

* fixes for snmpwalk input after previous refactor

* add the extra stuff SNMP::Info device class uses into snapshot

* better width for snmp search box

* fix css for snmp options

* add spinner while snmp loading

* add spinner while snmp loading

* add spinner while snmp loading

* support SNMP::Info device class or named MIBs as extra on snapshot

* add final tidy and bug fix
2023-08-10 22:27:02 +01:00
Oliver Gorwits
7afae0b9b2 avoid tags worker if device not in storage 2023-07-27 07:12:08 +01:00
Oliver Gorwits
e2ce601782 fix double encoding on JSON UTF-8 custom fields 2023-07-25 12:42:20 +01:00
Oliver Gorwits
47c4226788 refactor CustomFields so with_custom_fields is available on backend 2023-07-20 16:51:04 +01:00
Oliver Gorwits
31489ba550 #1072 clean custom_fields keys on rediscover 2023-07-20 12:49:40 +01:00
Oliver Gorwits
f8222e7890 #1072 clean custom_fields keys on rediscover 2023-07-19 22:18:00 +01:00
Oliver Gorwits
0378e01af7 skip set tags if none to set 2023-07-19 17:46:20 +01:00
Oliver Gorwits
4b99f0bb9c #1064 implement setting of tags on discovery based on config 2023-07-19 11:24:41 +01:00
Oliver Gorwits
71aa261737 fix skip long SNMP timeout for initial discover 2023-07-15 10:14:01 +01:00
Oliver Gorwits
eac55a581d #1066 implement snmp_try_slow_connect and fast device discover from queue 2023-07-14 17:37:52 +01:00
Oliver Gorwits
2cbb68889a #975 RBAC for port control with new portctl_by_role setting 2023-06-27 22:52:04 +01:00
Oliver Gorwits
b89e8913b6 #843 redux - also refresh API token for getapikey 2023-06-26 16:54:40 +01:00
Ambroise
130fa4aebe Change the way to retrieve IP address of interface (#1026)
Use by default the ipTable MIB entry to retrieve the IP address of interfaces and fallback to the index method as previously.
Fixe the issue with the case where the index is not really an IP address (seen in Fortinet with VDOM).

Related to issue #1016
2023-05-31 14:06:00 +01:00
Oliver Gorwits
9355f5c2b9 Refactored ACL support with multi-object compare
Squashed commit of the following:

commit 4081e22202693bd7c4ea00e95daad8e628c6fd5a
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon May 29 21:02:07 2023 +0100

    large rename of check_acl* to acl_matches*

commit 3cfa284ddd24d68765c255578cc5c184afbdcd83
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Fri May 19 20:39:03 2023 +0100

    update permission doc

commit 8c7bb93cc5e9fafb770f98f446e45cbd94b14894
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Wed May 17 21:50:07 2023 +0100

    migrate most check_acl_only to acl_matches_only

commit c47f699f2a22f08f2f3e093ed0f24c891e6f9a82
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Wed May 17 21:39:19 2023 +0100

    rename check_acl* to be acl_matches*

commit a884a22c3ab1f3262118c3a47ed8e25b0b0a7336
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun May 14 16:50:42 2023 +0100

    update macsuck_no_deviceports to use acl_matches

commit 8c256af728721329b64d071fa529dfc844073ac6
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun May 7 22:54:33 2023 +0100

    update hide_deviceports to use acl_matches multi @things

commit cd5d9978aba1da459be4fed4500f395df13f7784
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun May 7 22:53:38 2023 +0100

    check_acl fix to allow all @things to offer a property before fallback to missing as empty string

commit 1a3ab9a7646e9f994f03126d45fc36e9e5a13ed5
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 2 15:31:17 2023 +0100

    add ignore_deviceports to portproperties discover; improve comments

commit 51385ce89458dc939587dae902fda431719c22c9
Merge: b97c07d2 3f8ffe78
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 2 15:21:48 2023 +0100

    Merge branch 'master' into og-acl_multidict

commit b97c07d237d750c1d9eb3095d8ff3908512eac2a
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 25 14:37:53 2023 +0000

    add support for arrayref of items, and unblessed hash, to check_acl
2023-05-29 21:32:07 +01:00
Oliver Gorwits
3f8ffe787f hooks only queued if their trigger job is successful (done) 2023-04-30 22:59:20 +01:00
Oliver Gorwits
949aeb9eea delete hook (#1032)
* make log_message optional in delete_device

* add hooks support to delete job

* make delete job high prio

* web delete now queues job instead of inline delete

* move web logging into web package and remove userlog from device delete helper

* submit delete job for expire device instead of inline delete

* fixes to get web submit form for delete device to work

* enable delete hook functionality
2023-04-30 22:58:42 +01:00
Oliver Gorwits
5c1e753c14 #1013 undef error in Nodes interfaces 2023-03-29 11:17:06 +01:00
Oliver Gorwits
1d988bbf7c implement ignore_layers, force_macsuck, force_arpnip config settings (#1002)
* implementation of ignore_layers, force_macsuck, force_arpnip and macwalk

* use new WalkJobs view to get devices needing macsuck

* also new query for discoverall, arpwalk, nbtwalk

* faux record has a last_defer stamp so we can see when the backend started

* fix typo
2023-03-15 14:44:42 +00:00
Oliver Gorwits
062895df10 allow 3min for port last_change compare to uptime, do not assume wrapped 2023-03-15 08:07:45 +00:00
Oliver Gorwits
2848877d74 #977 add comment to netdisco-do renumber about device_identity 2023-03-07 20:15:54 +00:00
Oliver Gorwits
4bb9b25ae9 #901 node_ip time_last can be before time_first 2023-03-03 15:34:16 +00:00
Oliver Gorwits
d9a384d1d0 #949 replace other occurances of now() with LOCALTIMESTAMP 2023-03-03 14:17:38 +00:00
Oliver Gorwits
29402f0726 #949 replace now() with LOCALTIMESTAMP and cast to_timestamp() as ::timestamp 2023-03-03 14:08:10 +00:00
Oliver Gorwits
de1ec0dc67 #981 inspect ports lastchange for uptime wrap only after filtering ports 2023-02-27 18:00:21 +00:00
Oliver Gorwits
83b49f8ece add IEEE8023-LAG-MIB to snapshot retrieval 2022-12-13 15:29:10 +00:00
Oliver Gorwits
22210b26b7 keep port custom_fields in place across device discovery 2022-12-11 16:54:07 +00:00
Oliver Gorwits
97f40539e9 update num_ports to number of actually stored interfaces
also change update_or_insert to just update as the code never runs unless
in_storage anyway
2022-12-11 16:37:33 +00:00
Oliver Gorwits
9eb6aaebfc set layer2 support after successful macsuck, similar to arpnip does for l3 2022-11-25 21:54:03 +00:00
Oliver Gorwits
826e1db39d API Endpoints to submit arpnip and macsuck results (#942)
* Add macsuck worker to collect various PortAccessEntity (NAC) attributes

* Incorporate PAE feedback on #937

 * missing Result/Device.pm column added
 * pae_is... columns instead of pae_capabilities
 * moved most code to Util/PortAccessEntity.pm so the update can
   be done in discover and macsuck

* Refactor PAE attributes during discover as separate Plugin

* PortAccessEntity: don't use device->dns in log string

* Fix "Experimental keys on scalar is now forbidden" test failure

* Revamp pae_control and add missing attribute

 - device.pae_control (text) is now device.pae_is_enabled (bool)
 - also store pae_authconfig_port_control (port mode auto/force(un)Auth)

* Fix "Experimental keys on scalar is now forbidden" test failure

 - ... again because of botched merge
 - at least perlgolfed away a set of curly braces

* Update PortAccessEntity.pm

* Incorporate @ollyg PR feedback

* allow actions without transport to run when there are also no creds

* initial refactor for separate gather, process, store phases for macsuck

* factor out the vlan sanity check

* additional help with log of action workers

* cleanup logic in check macsuck

* refactor to make main phases only

* some fixes

* implement file slurp. amazingly the whole thing works

* remove outdated noop from test

* treat error as critical, use cancel to suppress further drivers

* big refactor to share mac sanity code to both paths

* fix inverted logic on vlan sanity filter

* some code tidy

* fix error in default value

* fix for vlan 0 nodes input from cli

* ensure imported MACs are IEEE format

* add api endpoint, no useful return status yet

* exit status if error from nodes PUT

* suppress other networked workers when direct workers are active

* better log showing worker

* fix status recording to get first error or last done message

* implement arpnip API PUT

* avoid package redeclaration error

* make sure write API methods require admin status

* add doc for passing JSON data to arpnip and macsuck

* update manifest

* remove option to do jobs in web handler; all by queue now

* use job entry timestamp for offline queued jobs

* fix store username and IP on api PUT

* never de-duplicate user-submitted jobs; never reset DeviceSkip for offline jobs

* myworker no longer needed

* make logic cleaner

Co-authored-by: Christian Ramseyer <ramseyer@netnea.com>
2022-11-25 15:24:23 +00:00
Oliver Gorwits
0e9ff81cf5 Add worker to collect various PortAccessEntity (NAC) attributes (PR #937, partially implements #887)
* Add macsuck worker to collect various PortAccessEntity (NAC) attributes

* Incorporate PAE feedback on #937

 * missing Result/Device.pm column added
 * pae_is... columns instead of pae_capabilities
 * moved most code to Util/PortAccessEntity.pm so the update can
   be done in discover and macsuck

* Refactor PAE attributes during discover as separate Plugin

* PortAccessEntity: don't use device->dns in log string

* Fix "Experimental keys on scalar is now forbidden" test failure

* Revamp pae_control and add missing attribute

 - device.pae_control (text) is now device.pae_is_enabled (bool)
 - also store pae_authconfig_port_control (port mode auto/force(un)Auth)

* Fix "Experimental keys on scalar is now forbidden" test failure

 - ... again because of botched merge
 - at least perlgolfed away a set of curly braces

* Update PortAccessEntity.pm

* Incorporate @ollyg PR feedback

Co-authored-by: Christian Ramseyer <ramseyer@netnea.com>
2022-11-04 10:03:26 +01:00
Oliver Gorwits
2cb2809dd9 #935 Macsuck forces ports up/up - but a forwarding table may be present on a port that is down 2022-10-18 12:49:24 +01:00
Oliver Gorwits
f9f06032f0 revert change as check_acl_only is the right way to go 2022-09-30 21:55:50 +01:00
Oliver Gorwits
d6d877a406 rename setting macsuck_no_deviceport to macsuck_no_deviceports 2022-09-24 22:35:23 +01:00
Oliver Gorwits
790c51b257 #925 implement ignore_deviceports and hide_deviceports 2022-09-20 20:32:35 +01:00