Squashed commit of the following:
commit 4081e22202693bd7c4ea00e95daad8e628c6fd5a
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon May 29 21:02:07 2023 +0100
large rename of check_acl* to acl_matches*
commit 3cfa284ddd24d68765c255578cc5c184afbdcd83
Author: Oliver Gorwits <oliver@cpan.org>
Date: Fri May 19 20:39:03 2023 +0100
update permission doc
commit 8c7bb93cc5e9fafb770f98f446e45cbd94b14894
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed May 17 21:50:07 2023 +0100
migrate most check_acl_only to acl_matches_only
commit c47f699f2a22f08f2f3e093ed0f24c891e6f9a82
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed May 17 21:39:19 2023 +0100
rename check_acl* to be acl_matches*
commit a884a22c3ab1f3262118c3a47ed8e25b0b0a7336
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun May 14 16:50:42 2023 +0100
update macsuck_no_deviceports to use acl_matches
commit 8c256af728721329b64d071fa529dfc844073ac6
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun May 7 22:54:33 2023 +0100
update hide_deviceports to use acl_matches multi @things
commit cd5d9978aba1da459be4fed4500f395df13f7784
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun May 7 22:53:38 2023 +0100
check_acl fix to allow all @things to offer a property before fallback to missing as empty string
commit 1a3ab9a7646e9f994f03126d45fc36e9e5a13ed5
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 2 15:31:17 2023 +0100
add ignore_deviceports to portproperties discover; improve comments
commit 51385ce89458dc939587dae902fda431719c22c9
Merge: b97c07d2 3f8ffe78
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 2 15:21:48 2023 +0100
Merge branch 'master' into og-acl_multidict
commit b97c07d237d750c1d9eb3095d8ff3908512eac2a
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Mar 25 14:37:53 2023 +0000
add support for arrayref of items, and unblessed hash, to check_acl
* make log_message optional in delete_device
* add hooks support to delete job
* make delete job high prio
* web delete now queues job instead of inline delete
* move web logging into web package and remove userlog from device delete helper
* submit delete job for expire device instead of inline delete
* fixes to get web submit form for delete device to work
* enable delete hook functionality
* implementation of ignore_layers, force_macsuck, force_arpnip and macwalk
* use new WalkJobs view to get devices needing macsuck
* also new query for discoverall, arpwalk, nbtwalk
* faux record has a last_defer stamp so we can see when the backend started
* fix typo
* Add macsuck worker to collect various PortAccessEntity (NAC) attributes
* Incorporate PAE feedback on #937
* missing Result/Device.pm column added
* pae_is... columns instead of pae_capabilities
* moved most code to Util/PortAccessEntity.pm so the update can
be done in discover and macsuck
* Refactor PAE attributes during discover as separate Plugin
* PortAccessEntity: don't use device->dns in log string
* Fix "Experimental keys on scalar is now forbidden" test failure
* Revamp pae_control and add missing attribute
- device.pae_control (text) is now device.pae_is_enabled (bool)
- also store pae_authconfig_port_control (port mode auto/force(un)Auth)
* Fix "Experimental keys on scalar is now forbidden" test failure
- ... again because of botched merge
- at least perlgolfed away a set of curly braces
* Update PortAccessEntity.pm
* Incorporate @ollyg PR feedback
* allow actions without transport to run when there are also no creds
* initial refactor for separate gather, process, store phases for macsuck
* factor out the vlan sanity check
* additional help with log of action workers
* cleanup logic in check macsuck
* refactor to make main phases only
* some fixes
* implement file slurp. amazingly the whole thing works
* remove outdated noop from test
* treat error as critical, use cancel to suppress further drivers
* big refactor to share mac sanity code to both paths
* fix inverted logic on vlan sanity filter
* some code tidy
* fix error in default value
* fix for vlan 0 nodes input from cli
* ensure imported MACs are IEEE format
* add api endpoint, no useful return status yet
* exit status if error from nodes PUT
* suppress other networked workers when direct workers are active
* better log showing worker
* fix status recording to get first error or last done message
* implement arpnip API PUT
* avoid package redeclaration error
* make sure write API methods require admin status
* add doc for passing JSON data to arpnip and macsuck
* update manifest
* remove option to do jobs in web handler; all by queue now
* use job entry timestamp for offline queued jobs
* fix store username and IP on api PUT
* never de-duplicate user-submitted jobs; never reset DeviceSkip for offline jobs
* myworker no longer needed
* make logic cleaner
Co-authored-by: Christian Ramseyer <ramseyer@netnea.com>
* Add macsuck worker to collect various PortAccessEntity (NAC) attributes
* Incorporate PAE feedback on #937
* missing Result/Device.pm column added
* pae_is... columns instead of pae_capabilities
* moved most code to Util/PortAccessEntity.pm so the update can
be done in discover and macsuck
* Refactor PAE attributes during discover as separate Plugin
* PortAccessEntity: don't use device->dns in log string
* Fix "Experimental keys on scalar is now forbidden" test failure
* Revamp pae_control and add missing attribute
- device.pae_control (text) is now device.pae_is_enabled (bool)
- also store pae_authconfig_port_control (port mode auto/force(un)Auth)
* Fix "Experimental keys on scalar is now forbidden" test failure
- ... again because of botched merge
- at least perlgolfed away a set of curly braces
* Update PortAccessEntity.pm
* Incorporate @ollyg PR feedback
Co-authored-by: Christian Ramseyer <ramseyer@netnea.com>
* started pseudodevice rework
* give them a type, os, os_ver and num_ports
* os_ver will be the netdisco version in which they were created
* give their ports the same value for device_port.descr as device_port.port
* add db update to fill out those fields if they are null, which is most likely
* update last_discover if you make changes to the device
* num_ports will not be updates by the db script, neither will device_port.descr
* use device.name, not device.dns for pseudo
* at least try and get reverse dns for pseudo, works if there is one
* add db revision to manifest
* take dns no config into account
* new meta
* new version
Co-authored-by: Oliver Gorwits <ollyg@users.noreply.github.com>
* fix anomalous name
* add gather worker
* fix encoding of binary storage
* store results back to job
* now parsing mbis report to translate
* fix the broken report parser
* rename gather to snapshot
* implement walk code copied from SNMP::Info
* can now bulkwalk and parse mibs report and store resolved walk in cache
* add func/glob aliasing broken
* better aliasing
* implement aliasing from globals and funcs
* fix regexp for matching netdisco-mibs report
* fake cache entry for all ND2 methods called, add comments
* also save to logs/snapshots/IP
* add doc for netdisco-do
* add is_pseudo column to device table
* support for loading cache for pseudo devices
* check for hrSystemUptime as well as sysUpTime for snmp connect
* display pseudo devices with yellow pill for name
* color all cells for layers for pseudo
* no need to b64 encode binary data in scalars as we b64 whole thing after
* tweaked uptime check
* store snapshot to database instead of Job
* expose snapshots in device details tab
* small ux improvements on snap download
* fixes for errors in subnet mask searching
* hide snapshot management for pseudo devices
* update to use new netdisco-mibs object cache
* update for new format oids file
* start of work on loading walk into db for browsing
* store values and meta
* add auto increment col and oid index to browser
* start web plugin for browser
* add virtual search for oid children
* have all oid in separte table (60 seconds load on my laptop)
* rename table and add relation
* store oid as int array
* fix sql for children
* make jstree start working
* working very slow tree expand
* fix to work when first displaying tree
* store both oid and oid_parts
* simplify SQL to speed up (more complicated perl)
* fix sql bug, add better index, prettify tree
* render the snmp node detail
* add node template, make scrollable, pretty print data values (insecure)
* store munge hint
* some dubious code to munge the data
* make sure to filter by IP on device_browser
* make safer the rendering of value data (but need to come back to key ordering)
* fix sorting on object values
* limit the opening of child nodes to keep response good and unclutter
* factor out the munge and make safer
* reject unknown mungers
* show the munger and option (not working) to change
* additional js for munge select
* complete custom munge
* change so that saving to database is only at CLI and on request
* hide snmp tab if no browser rows in the db
* add helpful message when no browser rows for the device
* stub handler for search and add recurse control
* working search
* minor ui fixes
* implement typeahead for leaf search
* limit rows in typeahead
* make sure device_browser is visited in delete and renumber
* add requirements for this branch
* update manifest
* make sure node search and typeahead are restricted to current device only
* initial v0 creator
* working json api for generic reports
* add require login
* move report swagger into plugin, and set new default layout of noop
* require proper role and also use new util func
* start to tidy authn
* some work on cleaning up web authn
* clean up the authN checks
* fix bug
* fix the auth for api
* fixes to json handling
* set swagger sort order
* enable most reports for api endpoints
* fix doc
* add paramters to reports
* add missed report
* allow api_parameters in reports config
* reorganise api
* add vlan search
* add port search
* make sure to enable layout processing
* add device search
* add v1 to api paths
* add Node Search
* support api_responses
* add device object search; fix spurious ports field in device result class
* handle some plugins just returning undef if search fails
* errors from api seamlessley
* fix error in date range default
* more sensible default for prefix
* change order of endpoints in swagger-ui
* all db row classes can now TO_JSON
* add device_port api endpoint
* add device ports endpoint
* do not expand docs
* add swagger ui json tree formatter
* add all relations from Device table
* add port relations
* add nodes retrieve on device or vlan
* rename to GetAPIKey
* update config for previous commit
* We meant well but it turns out that the array unnest and join is
actually very slow, as the join arguments do not get pushed down
into the CTE (in Postgres 9/10 at least, later versions remove some
of the optimization barriers in that specifc type of query)
* This caused a seq scan on both device and device_port, and the query
is executed many times during macsuck
* The query is now rewritten to use ANY (macaddr[]) and without CTE,
which seems to be around 20x faster
Hi @ollyg! Unfortunately I have found some issues with the code we
finally released in #680:
* get_port_macs expects an array ref but values() returns array,
so the code was never called due to the return unless... check
* When fw_mac_list had exactly two entries, only the second value
was bound as a scalar to the parameter. This is probably due
to the shorthand bind formats described in
https://metacpan.org/pod/DBIx::Class::ResultSet#DBIC-BIND-VALUES,
but I'm not a 100% on this.
* return unless now checks for an entry in the list, with the old
check the statement was also executed for empty lists
In cases where only the device(_port)?.mac lookup worked for uplink
detection, users of 02.044005 - 010 might get a lot of uplinks not
labeled as such.
* Reduce macsuck bandwidth usage to database
get_port_macs transfers a full list of all device_port.(mac,ip) in every macsuck.
With 8k devices and 40k interfaces it takes up around 15MB. Transferring them 8k times
during an 1h macsuck cycle requires bandwidth in the 300 to 400 mbit/s range.
This patch changes get_port_macs to be called inside walk_fwtable and only transfer
the macs found in the current target device/vlan.
this patch resets all pseudo devices to have no layer3 support but adds a
feature to the pseudo devices admin panel to enable layer3 support. it also
changes arpnip and arpwalk behaviour to always permit the action if layer3
is available (ignoring the vendor).
documentation will need updating to tell users to create pseudo devices
with layer3 support when they want to arpnip an unsupported platform.
arpnip with ssh/cli against a supported platform (one that can be discovered)
will continue to work normally.
Squashed commit of the following:
commit 9dad5be81d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 09:03:53 2019 +0100
allow pseudo with layer 3 to run arpnip
commit 7d97943fcd
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:59:10 2019 +0100
allow pseudo devices with layer 2/3 capability
commit d1fdf574e3
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:55:41 2019 +0100
move pseudo and layer checks to is_able from is_able_now
commit e0f72ef67d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:51:42 2019 +0100
ports defaults to one
commit 86ba01270c
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:50:45 2019 +0100
add tooltip for arpnip toggle
commit cdd2470228
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:34:46 2019 +0100
simplify template
commit 46236d68ea
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 23:53:56 2019 +0100
a fix up for pseudo devices which need layer 3
commit 016d249efc
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 20:37:11 2019 +0100
do not wrap buttons
commit 1ec1402e0c
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 20:33:03 2019 +0100
implement user settable layer-three service for pseudo devices
commit a267efa3d8
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 18:39:22 2019 +0100
only set layer if successful action
commit b108be5e23
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 18:32:19 2019 +0100
should defer SNMP against pseudo devices
commit 897ba3a629
Merge: e0ddbaaba7348900
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 14:54:36 2019 +0100
Merge branch 'master' into og-pseudo-vs-cli-arpnip
commit e0ddbaab08
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon Aug 26 11:35:13 2019 +0100
as last commit, for discover
commit 61f9c89040
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Aug 25 23:55:38 2019 +0100
move pseudo and layer checks into is_*able functions
commit 8b010d4023
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Aug 25 18:38:11 2019 +0100
any device completing macsuck/arpnip must have that layer
commit a11bce7863
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Aug 25 18:33:27 2019 +0100
clean up device layers
commit d2661bff61
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Aug 25 18:18:02 2019 +0100
first make arpnip behave like other jobs towards pseudo devices
