Commit Graph

152 Commits

Author SHA1 Message Date
Oliver Gorwits
826e1db39d API Endpoints to submit arpnip and macsuck results (#942)
* Add macsuck worker to collect various PortAccessEntity (NAC) attributes

* Incorporate PAE feedback on #937

 * missing Result/Device.pm column added
 * pae_is... columns instead of pae_capabilities
 * moved most code to Util/PortAccessEntity.pm so the update can
   be done in discover and macsuck

* Refactor PAE attributes during discover as separate Plugin

* PortAccessEntity: don't use device->dns in log string

* Fix "Experimental keys on scalar is now forbidden" test failure

* Revamp pae_control and add missing attribute

 - device.pae_control (text) is now device.pae_is_enabled (bool)
 - also store pae_authconfig_port_control (port mode auto/force(un)Auth)

* Fix "Experimental keys on scalar is now forbidden" test failure

 - ... again because of botched merge
 - at least perlgolfed away a set of curly braces

* Update PortAccessEntity.pm

* Incorporate @ollyg PR feedback

* allow actions without transport to run when there are also no creds

* initial refactor for separate gather, process, store phases for macsuck

* factor out the vlan sanity check

* additional help with log of action workers

* cleanup logic in check macsuck

* refactor to make main phases only

* some fixes

* implement file slurp. amazingly the whole thing works

* remove outdated noop from test

* treat error as critical, use cancel to suppress further drivers

* big refactor to share mac sanity code to both paths

* fix inverted logic on vlan sanity filter

* some code tidy

* fix error in default value

* fix for vlan 0 nodes input from cli

* ensure imported MACs are IEEE format

* add api endpoint, no useful return status yet

* exit status if error from nodes PUT

* suppress other networked workers when direct workers are active

* better log showing worker

* fix status recording to get first error or last done message

* implement arpnip API PUT

* avoid package redeclaration error

* make sure write API methods require admin status

* add doc for passing JSON data to arpnip and macsuck

* update manifest

* remove option to do jobs in web handler; all by queue now

* use job entry timestamp for offline queued jobs

* fix store username and IP on api PUT

* never de-duplicate user-submitted jobs; never reset DeviceSkip for offline jobs

* myworker no longer needed

* make logic cleaner

Co-authored-by: Christian Ramseyer <ramseyer@netnea.com>
2022-11-25 15:24:23 +00:00
Oliver Gorwits
0e9ff81cf5 Add worker to collect various PortAccessEntity (NAC) attributes (PR #937, partially implements #887)
* Add macsuck worker to collect various PortAccessEntity (NAC) attributes

* Incorporate PAE feedback on #937

 * missing Result/Device.pm column added
 * pae_is... columns instead of pae_capabilities
 * moved most code to Util/PortAccessEntity.pm so the update can
   be done in discover and macsuck

* Refactor PAE attributes during discover as separate Plugin

* PortAccessEntity: don't use device->dns in log string

* Fix "Experimental keys on scalar is now forbidden" test failure

* Revamp pae_control and add missing attribute

 - device.pae_control (text) is now device.pae_is_enabled (bool)
 - also store pae_authconfig_port_control (port mode auto/force(un)Auth)

* Fix "Experimental keys on scalar is now forbidden" test failure

 - ... again because of botched merge
 - at least perlgolfed away a set of curly braces

* Update PortAccessEntity.pm

* Incorporate @ollyg PR feedback

Co-authored-by: Christian Ramseyer <ramseyer@netnea.com>
2022-11-04 10:03:26 +01:00
Oliver Gorwits
d6d877a406 rename setting macsuck_no_deviceport to macsuck_no_deviceports 2022-09-24 22:35:23 +01:00
Oliver Gorwits
1d5f3ce316 #929 implement first half of multi tenancy with tenant_databases setting 2022-09-24 19:01:16 +01:00
Oliver Gorwits
790c51b257 #925 implement ignore_deviceports and hide_deviceports 2022-09-20 20:32:35 +01:00
Oliver Gorwits
f4dde980ff #914 change from silent_ports to macsuck_no_deviceport 2022-08-16 09:27:50 +01:00
Oliver Gorwits
6a27a2b3b1 #914 initial implementation of silent_ports 2022-08-15 22:26:38 +01:00
Oliver Gorwits
7aca60a348 bug - fe80::/10 is not LOCAL ADDRESS like ::1/128 2022-08-15 11:26:49 -04:00
Oliver Gorwits
3684451f2a add ifindex column for show 2022-08-14 21:13:27 +01:00
Oliver Gorwits
a004db2951 add discover_routed_neighbors setting default true 2022-08-10 17:31:04 -04:00
Oliver Gorwits
3d81f3e5f6 make hide1002 and vlan_names true sidebar options and move to port properties section 2022-08-03 06:22:51 +01:00
Oliver Gorwits
89ada080da #900 add option to hide vlan 1002-1005 2022-08-02 21:41:48 +01:00
Oliver Gorwits
aad4f0ec33 #902 change order of device ports cols to put duplex after speed 2022-08-01 08:44:19 +01:00
Oliver Gorwits
7566e64886 #882 option to display vlan names instead of numbers in device ports table 2022-07-26 09:49:07 +01:00
Oliver Gorwits
796aa1c6f0 rename port options for remote inventory and always show LLDP IP with not contacted device details 2022-07-25 16:43:40 +01:00
Oliver Gorwits
07e8520bac #881 new addpseudodevice worker action 2022-07-04 22:02:47 +01:00
Oliver Gorwits
2e8384df5a fix bugs to allow bulkwalk_no snapshot, and index with trailing zeros 2021-11-25 11:37:28 +00:00
Oliver Gorwits
6545fff0c2 allow contact|location|portname|snapshot to run on pseudo devices 2021-11-06 22:32:35 +00:00
Oliver Gorwits
dc1f76c1aa Feature to gather SNMP Walk, use as Pseudo Device, and Browse Objects
* fix anomalous name

* add gather worker

* fix encoding of binary storage

* store results back to job

* now parsing mbis report to translate

* fix the broken report parser

* rename gather to snapshot

* implement walk code copied from SNMP::Info

* can now bulkwalk and parse mibs report and store resolved walk in cache

* add func/glob aliasing broken

* better aliasing

* implement aliasing from globals and funcs

* fix regexp for matching netdisco-mibs report

* fake cache entry for all ND2 methods called, add comments

* also save to logs/snapshots/IP

* add doc for netdisco-do

* add is_pseudo column to device table

* support for loading cache for pseudo devices

* check for hrSystemUptime as well as sysUpTime for snmp connect

* display pseudo devices with yellow pill for name

* color all cells for layers for pseudo

* no need to b64 encode binary data in scalars as we b64 whole thing after

* tweaked uptime check

* store snapshot to database instead of Job

* expose snapshots in device details tab

* small ux improvements on snap download

* fixes for errors in subnet mask searching

* hide snapshot management for pseudo devices

* update to use new netdisco-mibs object cache

* update for new format oids file

* start of work on loading walk into db for browsing

* store values and meta

* add auto increment col and oid index to browser

* start web plugin for browser

* add virtual search for oid children

* have all oid in separte table (60 seconds load on my laptop)

* rename table and add relation

* store oid as int array

* fix sql for children

* make jstree start working

* working very slow tree expand

* fix to work when first displaying tree

* store both oid and oid_parts

* simplify SQL to speed up (more complicated perl)

* fix sql bug, add better index, prettify tree

* render the snmp node detail

* add node template, make scrollable, pretty print data values (insecure)

* store munge hint

* some dubious code to munge the data

* make sure to filter by IP on device_browser

* make safer the rendering of value data (but need to come back to key ordering)

* fix sorting on object values

* limit the opening of child nodes to keep response good and unclutter

* factor out the munge and make safer

* reject unknown mungers

* show the munger and option (not working) to change

* additional js for munge select

* complete custom munge

* change so that saving to database is only at CLI and on request

* hide snmp tab if no browser rows in the db

* add helpful message when no browser rows for the device

* stub handler for search and add recurse control

* working search

* minor ui fixes

* implement typeahead for leaf search

* limit rows in typeahead

* make sure device_browser is visited in delete and renumber

* add requirements for this branch

* update manifest

* make sure node search and typeahead are restricted to current device only
2021-11-06 07:47:29 +00:00
Oliver Gorwits
202d02a64a #827 add ignore_interface_types setting 2021-10-12 13:21:07 +01:00
Oliver Gorwits
3c3b9f286c better Content-Security-Policy 2021-10-06 17:56:46 +01:00
Oliver Gorwits
381f412df9 clickjacking prevention via X-Frame-Options and Content-Security-Policy headers 2021-10-06 16:44:36 +01:00
Oliver Gorwits
43ce26ef3d Allow port_control_reasons setting to actually work 2021-09-03 21:17:11 +01:00
Oliver Gorwits
5f3ae3bf10 Config setting portctl_nowaps to prevent port control on WAP ports 2021-09-03 06:22:16 +01:00
Oliver Gorwits
e953e3945f ACLs portctl_no and portctl_only for limiting port control targets 2021-09-03 06:15:20 +01:00
Oliver Gorwits
8509cd96cc #778 exec hooks were not configured to be high priority 2021-07-14 13:15:40 +01:00
nick n
83a946d36e replace unicode nonbreaking space with ascii space (#794) 2021-06-23 22:00:33 +01:00
Oliver Gorwits
351927d384 add arpnip and macsuck hooks and exec hook implementation 2020-12-30 20:38:45 +00:00
Oliver Gorwits
669eec46db Implement Hooks per #726 2020-12-23 11:18:38 +00:00
Oliver Gorwits
31fc6e1ee5 #734 Multiple TACACS+/RADIUS servers now supported 2020-07-10 22:18:29 +01:00
Oliver Gorwits
fc46384fc6 #227 Do not overwrite partially retrieved data with null or incorrect values
this and two related commits implements:
- setting snmp_field_protection which can be used to protect any
  database field from having a null value set
- extra device discover worker which prefetches SNMP interfaces and
  performs some sanity checks and comparison against the DB
the two checks will return job status ERROR with CANCELLED if they
see anything fishy.
2020-07-06 19:49:57 +01:00
Oliver Gorwits
7e51fe2295 try to safeguard some snmp leaf gather failures 2020-07-01 21:21:09 +01:00
Oliver Gorwits
da33478504 try to make swagger safer with send_file 2020-04-19 11:22:28 +01:00
Oliver Gorwits
4f0117e22c many fixes to swagger hosting to allow path relocation (part complete) 2020-04-18 17:26:04 +01:00
Oliver Gorwits
dff26abc5c API implementation (#712)
* initial v0 creator

* working json api for generic reports

* add require login

* move report swagger into plugin, and set new default layout of noop

* require proper role and also use new util func

* start to tidy authn

* some work on cleaning up web authn

* clean up the authN checks

* fix bug

* fix the auth for api

* fixes to json handling

* set swagger sort order

* enable most reports for api endpoints

* fix doc

* add paramters to reports

* add missed report

* allow api_parameters in reports config

* reorganise api

* add vlan search

* add port search

* make sure to enable layout processing

* add device search

* add v1 to api paths

* add Node Search

* support api_responses

* add device object search; fix spurious ports field in device result class

* handle some plugins just returning undef if search fails

* errors from api seamlessley

* fix error in date range default

* more sensible default for prefix

* change order of endpoints in swagger-ui

* all db row classes can now TO_JSON

* add device_port api endpoint

* add device ports endpoint

* do not expand docs

* add swagger ui json tree formatter

* add all relations from Device table

* add port relations

* add nodes retrieve on device or vlan

* rename to GetAPIKey

* update config for previous commit
2020-04-15 21:15:52 +01:00
Oliver Gorwits
ab6305b4cf default access list for device_auth should be ipv4 and ipv6 2020-02-12 16:55:16 +00:00
Oliver Gorwits
4030000b34 #687 Add an option to set main page after logging in 2019-12-21 22:24:19 +00:00
Oliver Gorwits
deb9b62c7f Enforce escaping on all template content 2019-09-23 14:22:00 +01:00
Oliver Gorwits
5f378a39ea better column titles 2019-09-23 14:05:04 +01:00
Christian Ramseyer
8ffdff3bbc Add speed_admin to device ports tab (#650) (#653)
* Add speed_admin to device ports tab (#650)

* move from device_port_properties to device_port table, and tidy up
2019-09-23 13:58:26 +01:00
pyro3d
73e9f0bf0d Add DOCSIS/Cable Modem Discovery (#646)
* Add DOCSIS/Cable Modem Discovery
2019-09-18 16:19:18 -07:00
Oliver Gorwits
542837d266 #591 domain_suffix can be list and supports (simple) regexp
Squashed commit of the following:

commit 975e4c6afc
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue Sep 3 13:35:26 2019 +0100

    also support regexp in domain_suffix

commit 0a2b5c8fa2
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue Sep 3 13:17:17 2019 +0100

    fix rancid, graph, and nodemonitor

commit 6d881de3ff
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue Sep 3 13:11:54 2019 +0100

    improve docs and set default domain_suffix to be list

commit 1dcafc08a8
Merge: 9a752e02 14ac69dc
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue Sep 3 13:00:39 2019 +0100

    Merge branch 'master' into og-multiple-domain-suffix

commit 9a752e0298
Merge: c836619f 82a99ea9
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue Sep 3 09:45:25 2019 +0100

    Merge branch 'master' into og-multiple-domain-suffix

commit c836619f8c
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Thu Jun 13 07:52:45 2019 +0100

    hokey fix for nodes with domains

commit fed14bd810
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Thu Jun 13 07:02:09 2019 +0100

    basic implementation, rancid graph and nodemonitor missing
2019-09-03 13:35:43 +01:00
Oliver Gorwits
262fa88009 #585 make toastr notifications less annoying 2019-05-27 12:53:56 +01:00
Oliver Gorwits
8f1f35abab fix inconsistency in titlebar global search when port is selected 2019-04-16 17:10:13 +01:00
nick n
fd4752897d netdisco rancid updates (#558)
* typo

* make this a link

* netdisco-rancid-export has been replaced

* rancid worker config

* netdisco-rancid-export now a no-op

* note debugging option
2019-04-05 01:28:30 +02:00
nick n
3f407685b1 unneeded question marks 2019-03-23 02:07:17 +01:00
Oliver Gorwits
46d8c52a08 Implement changes for API authentication and Swagger UI (#541)
* initial token-based-api login handler

* add token schema and validation

* initial import of pyro3d api code

* basic Swagger spec support

* Merge in working copy of API/Device.pm

* Fix some error handling for API/Device.pm

* Break out utility functions into separate file, to allow other api portions to use

* Add NodeIP support.

* Add nodeip plugin to config

* remove double define of "plugin:" (#448)

disclaimer: i did not test this is any way, came across it when looking for something else.

* only AuthZ header for api use, and alway regen key on login

* use RFC7235

* workaround for Swagger plugin weird response body

* do not autodiscover swagger routes

* code formatting only

* move api util to utils area

* initial full swagger spec for nodeip search

* add api user role and fix api auth failure response

* update version of swagger-ui to 3.20.3

* add more openapi defs

* fixes to SQL and api spec

* clean up subs

* improvements to login/logout for API

* make api logout work

* add openapi tags to group operations

* allow api params to be generated from DBIC schema spec

* remove API calls for nodes and devices

* remove some poor assumptions about api calls

* tidy up

* remove DDP

* make login and logout similar

* example of api call being handled by ajax call

* make the branch authonly
2019-03-17 20:27:19 +00:00
Oliver Gorwits
9b39d1e452 rename CLI Transport to SSH 2019-03-13 22:23:38 +00:00
Oliver Gorwits
d44a8f56ea Integrate netdisco-sshcollector into Worker::Plugin architecture (#489) (#535)
* update changes and SNMP::Info dep

* Integrate netdisco-sshcollector into Worker::Plugin architecture (#489)

* Initial integration of sshcollector into Worker::Plugin architecture

 * add NodesBySSH.pm
 * update Build.PL and config.yml to integrate the new module

* Further integration of sshcollector into Worker::Plugin architecture

 * added App::Netdisco::Transport::CLI loosely based on ::SNMP counterpart
 * switched to the more prevalent two-space tabs style
 * removed various TBD items, some new ones

* Further steps to integration of sshcollector into Worker::Plugin architecture

 * cleaned up code
 * added various error handling
 * warning for bin/netdisco-sshcollector deprecation
 * device_auth allows passing master_opts to Net::OpenSSH
 * netdisco-do -D also toggles Net::OpenSSH debug

* Merged NodesBySSH.pm into Nodes.pm

 * see https://github.com/netdisco/netdisco/pull/489#pullrequestreview-205603516

* Further integration of sshcollector into Worker::Plugin architecture

 * add snmp_arpnip_also option to sshcollector device_auth
 * cleanup code

* Remove big TBD: comment from CLI.pm as doc is updated now

* add transport/cli.pm to manifest

* revert some changes to allow simpler merging

* silent exit legacy script unless explicitly requested

* move ssh code into Transport, part one

* rewrite the CLI transport to provide an API

* merge in og-get_external_credentials

Squashed commit of the following:

commit 3fe8f383a7
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 11 17:07:42 2019 +0000

    add debug lines and tested

commit 3249739e42
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 11 16:54:11 2019 +0000

    change config name to get_credentials

commit e78558397a
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 11 16:51:11 2019 +0000

    separate out generic device auth to DeviceAuth module

commit 249f05165f
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Wed Mar 6 18:43:31 2019 +0000

    release 2.040007

commit e3af64df77
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Wed Mar 6 18:42:47 2019 +0000

    #521-redux fix wifi date search

commit 48857ae300
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 4 12:03:31 2019 +0000

    release 2.040006

commit e09dab5362
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 4 11:39:12 2019 +0000

    #527 update List::MoreUtils version requirement

commit 6e7de3fff3
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 4 09:59:41 2019 +0000

    release 2.040005

commit 0c98318a45
Author: Oliver Gorwits <oliver@spike.local>
Date:   Mon Mar 4 09:57:18 2019 +0000

    #526 fix discover syntax bug

commit e9efc45182
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 14:56:48 2019 +0000

    release 2.040004

commit 6cdfd80d10
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 14:34:00 2019 +0000

    allow undiscovered neighbors report to use discover_{waps,phones} setting

commit ac381e0802
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 14:13:20 2019 +0000

    #506 was a red herring

commit b83e614c85
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 13:00:36 2019 +0000

    make discover_{phones,waps} work with LLDP capabilities as well

commit 189d234b55
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 12:47:38 2019 +0000

    check discover_no_type and friends earlier on in neighbors list build

commit 9c956466f3
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 12:32:07 2019 +0000

    also update default config for new discover_phones and discover_waps settings

commit 09d29954d2
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 12:26:50 2019 +0000

    #512 fix regression in phone/wap discovery exclusion

commit 2bae91f1b6
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 12:01:34 2019 +0000

    rename match_devicetype() to match_to_setting()

commit 57cb6ddb70
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 09:19:39 2019 +0000

    fix for over-eager fix to #506

commit ef560fb59a
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 22:41:40 2019 +0000

    #506 relax device renumber so it works for an alias

commit 7a8bcb094e
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 22:23:39 2019 +0000

    #521 Search Node Date Range not working

commit a643820a62
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 21:54:27 2019 +0000

    #428 Port-Channels not showing in netmap

commit 5ba5bcd295
Merge: e7aacddb a1f95028
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 20:04:11 2019 +0000

    Merge branch 'master' of github.com:netdisco/netdisco

commit e7aacddbc6
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 20:01:05 2019 +0000

    #498 Map with VLAN filter omits unconnected devices

commit a1f95028ca
Author: nick n <39005454+inphobia@users.noreply.github.com>
Date:   Sat Mar 2 19:54:22 2019 +0100

    catch up with changes

    noticed that rc-sshcollector-core received updates to changes, add them here as well.

    didn't mention #499 & #522

commit ce1b847cea
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 18:47:44 2019 +0000

    fix bug showing no nodes when only one matches in netmap

commit 78e30a7926
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 16:28:15 2019 +0000

    #500 filtering in device/ports on native vlan duplicates entries

commit 9952f0c6c7
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 15:02:12 2019 +0000

    #499 netdisco-do renumber reports wrong ip (inphobia)

commit ca3fd8f466
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 15:00:18 2019 +0000

    #505 device renumber should update device port properties and device skips

commit 1265bc8470
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 14:52:21 2019 +0000

    #520 catch slave ports defined without a master

commit d4c7579c10
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 14:47:49 2019 +0000

    #522 TypeAhead.pm can reference empty data (inphobia)

commit 77decc23b7
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 14:45:37 2019 +0000

    #514 inconsistent results in ip inventory (inphobia)

commit 3f211650b8
Author: nick n <39005454+inphobia@users.noreply.github.com>
Date:   Fri Mar 1 12:34:42 2019 +0100

    last pieces for db schema upgrade

    last piece of #510

* import legacy sshcollector config

* add default use_legacy_sshcollector config

* remove unneeded deps

* various fixes and now tested

* enable sshcollector platform tests
2019-03-12 18:50:24 +00:00
Oliver Gorwits
b9218d91c8 Support for external command to provide device_auth stanza (#532)
* separate out generic device auth to DeviceAuth module

* change config name to get_credentials

* add debug lines and tested
2019-03-12 13:13:06 +00:00