Squashed commit of the following:
commit 4081e22202693bd7c4ea00e95daad8e628c6fd5a
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon May 29 21:02:07 2023 +0100
large rename of check_acl* to acl_matches*
commit 3cfa284ddd24d68765c255578cc5c184afbdcd83
Author: Oliver Gorwits <oliver@cpan.org>
Date: Fri May 19 20:39:03 2023 +0100
update permission doc
commit 8c7bb93cc5e9fafb770f98f446e45cbd94b14894
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed May 17 21:50:07 2023 +0100
migrate most check_acl_only to acl_matches_only
commit c47f699f2a22f08f2f3e093ed0f24c891e6f9a82
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed May 17 21:39:19 2023 +0100
rename check_acl* to be acl_matches*
commit a884a22c3ab1f3262118c3a47ed8e25b0b0a7336
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun May 14 16:50:42 2023 +0100
update macsuck_no_deviceports to use acl_matches
commit 8c256af728721329b64d071fa529dfc844073ac6
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun May 7 22:54:33 2023 +0100
update hide_deviceports to use acl_matches multi @things
commit cd5d9978aba1da459be4fed4500f395df13f7784
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun May 7 22:53:38 2023 +0100
check_acl fix to allow all @things to offer a property before fallback to missing as empty string
commit 1a3ab9a7646e9f994f03126d45fc36e9e5a13ed5
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 2 15:31:17 2023 +0100
add ignore_deviceports to portproperties discover; improve comments
commit 51385ce89458dc939587dae902fda431719c22c9
Merge: b97c07d2 3f8ffe78
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 2 15:21:48 2023 +0100
Merge branch 'master' into og-acl_multidict
commit b97c07d237d750c1d9eb3095d8ff3908512eac2a
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Mar 25 14:37:53 2023 +0000
add support for arrayref of items, and unblessed hash, to check_acl
* make log_message optional in delete_device
* add hooks support to delete job
* make delete job high prio
* web delete now queues job instead of inline delete
* move web logging into web package and remove userlog from device delete helper
* submit delete job for expire device instead of inline delete
* fixes to get web submit form for delete device to work
* enable delete hook functionality
* implementation of ignore_layers, force_macsuck, force_arpnip and macwalk
* use new WalkJobs view to get devices needing macsuck
* also new query for discoverall, arpwalk, nbtwalk
* faux record has a last_defer stamp so we can see when the backend started
* fix typo
* Add macsuck worker to collect various PortAccessEntity (NAC) attributes
* Incorporate PAE feedback on #937
* missing Result/Device.pm column added
* pae_is... columns instead of pae_capabilities
* moved most code to Util/PortAccessEntity.pm so the update can
be done in discover and macsuck
* Refactor PAE attributes during discover as separate Plugin
* PortAccessEntity: don't use device->dns in log string
* Fix "Experimental keys on scalar is now forbidden" test failure
* Revamp pae_control and add missing attribute
- device.pae_control (text) is now device.pae_is_enabled (bool)
- also store pae_authconfig_port_control (port mode auto/force(un)Auth)
* Fix "Experimental keys on scalar is now forbidden" test failure
- ... again because of botched merge
- at least perlgolfed away a set of curly braces
* Update PortAccessEntity.pm
* Incorporate @ollyg PR feedback
* allow actions without transport to run when there are also no creds
* initial refactor for separate gather, process, store phases for macsuck
* factor out the vlan sanity check
* additional help with log of action workers
* cleanup logic in check macsuck
* refactor to make main phases only
* some fixes
* implement file slurp. amazingly the whole thing works
* remove outdated noop from test
* treat error as critical, use cancel to suppress further drivers
* big refactor to share mac sanity code to both paths
* fix inverted logic on vlan sanity filter
* some code tidy
* fix error in default value
* fix for vlan 0 nodes input from cli
* ensure imported MACs are IEEE format
* add api endpoint, no useful return status yet
* exit status if error from nodes PUT
* suppress other networked workers when direct workers are active
* better log showing worker
* fix status recording to get first error or last done message
* implement arpnip API PUT
* avoid package redeclaration error
* make sure write API methods require admin status
* add doc for passing JSON data to arpnip and macsuck
* update manifest
* remove option to do jobs in web handler; all by queue now
* use job entry timestamp for offline queued jobs
* fix store username and IP on api PUT
* never de-duplicate user-submitted jobs; never reset DeviceSkip for offline jobs
* myworker no longer needed
* make logic cleaner
Co-authored-by: Christian Ramseyer <ramseyer@netnea.com>
* Add macsuck worker to collect various PortAccessEntity (NAC) attributes
* Incorporate PAE feedback on #937
* missing Result/Device.pm column added
* pae_is... columns instead of pae_capabilities
* moved most code to Util/PortAccessEntity.pm so the update can
be done in discover and macsuck
* Refactor PAE attributes during discover as separate Plugin
* PortAccessEntity: don't use device->dns in log string
* Fix "Experimental keys on scalar is now forbidden" test failure
* Revamp pae_control and add missing attribute
- device.pae_control (text) is now device.pae_is_enabled (bool)
- also store pae_authconfig_port_control (port mode auto/force(un)Auth)
* Fix "Experimental keys on scalar is now forbidden" test failure
- ... again because of botched merge
- at least perlgolfed away a set of curly braces
* Update PortAccessEntity.pm
* Incorporate @ollyg PR feedback
Co-authored-by: Christian Ramseyer <ramseyer@netnea.com>
* started pseudodevice rework
* give them a type, os, os_ver and num_ports
* os_ver will be the netdisco version in which they were created
* give their ports the same value for device_port.descr as device_port.port
* add db update to fill out those fields if they are null, which is most likely
* update last_discover if you make changes to the device
* num_ports will not be updates by the db script, neither will device_port.descr
* use device.name, not device.dns for pseudo
* at least try and get reverse dns for pseudo, works if there is one
* add db revision to manifest
* take dns no config into account
* new meta
* new version
Co-authored-by: Oliver Gorwits <ollyg@users.noreply.github.com>
* fix anomalous name
* add gather worker
* fix encoding of binary storage
* store results back to job
* now parsing mbis report to translate
* fix the broken report parser
* rename gather to snapshot
* implement walk code copied from SNMP::Info
* can now bulkwalk and parse mibs report and store resolved walk in cache
* add func/glob aliasing broken
* better aliasing
* implement aliasing from globals and funcs
* fix regexp for matching netdisco-mibs report
* fake cache entry for all ND2 methods called, add comments
* also save to logs/snapshots/IP
* add doc for netdisco-do
* add is_pseudo column to device table
* support for loading cache for pseudo devices
* check for hrSystemUptime as well as sysUpTime for snmp connect
* display pseudo devices with yellow pill for name
* color all cells for layers for pseudo
* no need to b64 encode binary data in scalars as we b64 whole thing after
* tweaked uptime check
* store snapshot to database instead of Job
* expose snapshots in device details tab
* small ux improvements on snap download
* fixes for errors in subnet mask searching
* hide snapshot management for pseudo devices
* update to use new netdisco-mibs object cache
* update for new format oids file
* start of work on loading walk into db for browsing
* store values and meta
* add auto increment col and oid index to browser
* start web plugin for browser
* add virtual search for oid children
* have all oid in separte table (60 seconds load on my laptop)
* rename table and add relation
* store oid as int array
* fix sql for children
* make jstree start working
* working very slow tree expand
* fix to work when first displaying tree
* store both oid and oid_parts
* simplify SQL to speed up (more complicated perl)
* fix sql bug, add better index, prettify tree
* render the snmp node detail
* add node template, make scrollable, pretty print data values (insecure)
* store munge hint
* some dubious code to munge the data
* make sure to filter by IP on device_browser
* make safer the rendering of value data (but need to come back to key ordering)
* fix sorting on object values
* limit the opening of child nodes to keep response good and unclutter
* factor out the munge and make safer
* reject unknown mungers
* show the munger and option (not working) to change
* additional js for munge select
* complete custom munge
* change so that saving to database is only at CLI and on request
* hide snmp tab if no browser rows in the db
* add helpful message when no browser rows for the device
* stub handler for search and add recurse control
* working search
* minor ui fixes
* implement typeahead for leaf search
* limit rows in typeahead
* make sure device_browser is visited in delete and renumber
* add requirements for this branch
* update manifest
* make sure node search and typeahead are restricted to current device only
* initial v0 creator
* working json api for generic reports
* add require login
* move report swagger into plugin, and set new default layout of noop
* require proper role and also use new util func
* start to tidy authn
* some work on cleaning up web authn
* clean up the authN checks
* fix bug
* fix the auth for api
* fixes to json handling
* set swagger sort order
* enable most reports for api endpoints
* fix doc
* add paramters to reports
* add missed report
* allow api_parameters in reports config
* reorganise api
* add vlan search
* add port search
* make sure to enable layout processing
* add device search
* add v1 to api paths
* add Node Search
* support api_responses
* add device object search; fix spurious ports field in device result class
* handle some plugins just returning undef if search fails
* errors from api seamlessley
* fix error in date range default
* more sensible default for prefix
* change order of endpoints in swagger-ui
* all db row classes can now TO_JSON
* add device_port api endpoint
* add device ports endpoint
* do not expand docs
* add swagger ui json tree formatter
* add all relations from Device table
* add port relations
* add nodes retrieve on device or vlan
* rename to GetAPIKey
* update config for previous commit
* We meant well but it turns out that the array unnest and join is
actually very slow, as the join arguments do not get pushed down
into the CTE (in Postgres 9/10 at least, later versions remove some
of the optimization barriers in that specifc type of query)
* This caused a seq scan on both device and device_port, and the query
is executed many times during macsuck
* The query is now rewritten to use ANY (macaddr[]) and without CTE,
which seems to be around 20x faster
Hi @ollyg! Unfortunately I have found some issues with the code we
finally released in #680:
* get_port_macs expects an array ref but values() returns array,
so the code was never called due to the return unless... check
* When fw_mac_list had exactly two entries, only the second value
was bound as a scalar to the parameter. This is probably due
to the shorthand bind formats described in
https://metacpan.org/pod/DBIx::Class::ResultSet#DBIC-BIND-VALUES,
but I'm not a 100% on this.
* return unless now checks for an entry in the list, with the old
check the statement was also executed for empty lists
In cases where only the device(_port)?.mac lookup worked for uplink
detection, users of 02.044005 - 010 might get a lot of uplinks not
labeled as such.
* Reduce macsuck bandwidth usage to database
get_port_macs transfers a full list of all device_port.(mac,ip) in every macsuck.
With 8k devices and 40k interfaces it takes up around 15MB. Transferring them 8k times
during an 1h macsuck cycle requires bandwidth in the 300 to 400 mbit/s range.
This patch changes get_port_macs to be called inside walk_fwtable and only transfer
the macs found in the current target device/vlan.
