* fix anomalous name
* add gather worker
* fix encoding of binary storage
* store results back to job
* now parsing mbis report to translate
* fix the broken report parser
* rename gather to snapshot
* implement walk code copied from SNMP::Info
* can now bulkwalk and parse mibs report and store resolved walk in cache
* add func/glob aliasing broken
* better aliasing
* implement aliasing from globals and funcs
* fix regexp for matching netdisco-mibs report
* fake cache entry for all ND2 methods called, add comments
* also save to logs/snapshots/IP
* add doc for netdisco-do
* add is_pseudo column to device table
* support for loading cache for pseudo devices
* check for hrSystemUptime as well as sysUpTime for snmp connect
* display pseudo devices with yellow pill for name
* color all cells for layers for pseudo
* no need to b64 encode binary data in scalars as we b64 whole thing after
* tweaked uptime check
* store snapshot to database instead of Job
* expose snapshots in device details tab
* small ux improvements on snap download
* fixes for errors in subnet mask searching
* hide snapshot management for pseudo devices
* update to use new netdisco-mibs object cache
* update for new format oids file
* start of work on loading walk into db for browsing
* store values and meta
* add auto increment col and oid index to browser
* start web plugin for browser
* add virtual search for oid children
* have all oid in separte table (60 seconds load on my laptop)
* rename table and add relation
* store oid as int array
* fix sql for children
* make jstree start working
* working very slow tree expand
* fix to work when first displaying tree
* store both oid and oid_parts
* simplify SQL to speed up (more complicated perl)
* fix sql bug, add better index, prettify tree
* render the snmp node detail
* add node template, make scrollable, pretty print data values (insecure)
* store munge hint
* some dubious code to munge the data
* make sure to filter by IP on device_browser
* make safer the rendering of value data (but need to come back to key ordering)
* fix sorting on object values
* limit the opening of child nodes to keep response good and unclutter
* factor out the munge and make safer
* reject unknown mungers
* show the munger and option (not working) to change
* additional js for munge select
* complete custom munge
* change so that saving to database is only at CLI and on request
* hide snmp tab if no browser rows in the db
* add helpful message when no browser rows for the device
* stub handler for search and add recurse control
* working search
* minor ui fixes
* implement typeahead for leaf search
* limit rows in typeahead
* make sure device_browser is visited in delete and renumber
* add requirements for this branch
* update manifest
* make sure node search and typeahead are restricted to current device only
this patch resets all pseudo devices to have no layer3 support but adds a
feature to the pseudo devices admin panel to enable layer3 support. it also
changes arpnip and arpwalk behaviour to always permit the action if layer3
is available (ignoring the vendor).
documentation will need updating to tell users to create pseudo devices
with layer3 support when they want to arpnip an unsupported platform.
arpnip with ssh/cli against a supported platform (one that can be discovered)
will continue to work normally.
Squashed commit of the following:
commit 9dad5be81d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 09:03:53 2019 +0100
allow pseudo with layer 3 to run arpnip
commit 7d97943fcd
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:59:10 2019 +0100
allow pseudo devices with layer 2/3 capability
commit d1fdf574e3
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:55:41 2019 +0100
move pseudo and layer checks to is_able from is_able_now
commit e0f72ef67d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:51:42 2019 +0100
ports defaults to one
commit 86ba01270c
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:50:45 2019 +0100
add tooltip for arpnip toggle
commit cdd2470228
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Sep 3 08:34:46 2019 +0100
simplify template
commit 46236d68ea
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 23:53:56 2019 +0100
a fix up for pseudo devices which need layer 3
commit 016d249efc
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 20:37:11 2019 +0100
do not wrap buttons
commit 1ec1402e0c
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 20:33:03 2019 +0100
implement user settable layer-three service for pseudo devices
commit a267efa3d8
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 18:39:22 2019 +0100
only set layer if successful action
commit b108be5e23
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 18:32:19 2019 +0100
should defer SNMP against pseudo devices
commit 897ba3a629
Merge: e0ddbaaba7348900
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Sep 1 14:54:36 2019 +0100
Merge branch 'master' into og-pseudo-vs-cli-arpnip
commit e0ddbaab08
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon Aug 26 11:35:13 2019 +0100
as last commit, for discover
commit 61f9c89040
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Aug 25 23:55:38 2019 +0100
move pseudo and layer checks into is_*able functions
commit 8b010d4023
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Aug 25 18:38:11 2019 +0100
any device completing macsuck/arpnip must have that layer
commit a11bce7863
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Aug 25 18:33:27 2019 +0100
clean up device layers
commit d2661bff61
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Aug 25 18:18:02 2019 +0100
first make arpnip behave like other jobs towards pseudo devices
* initial token-based-api login handler
* add token schema and validation
* initial import of pyro3d api code
* basic Swagger spec support
* Merge in working copy of API/Device.pm
* Fix some error handling for API/Device.pm
* Break out utility functions into separate file, to allow other api portions to use
* Add NodeIP support.
* Add nodeip plugin to config
* remove double define of "plugin:" (#448)
disclaimer: i did not test this is any way, came across it when looking for something else.
* only AuthZ header for api use, and alway regen key on login
* use RFC7235
* workaround for Swagger plugin weird response body
* do not autodiscover swagger routes
* code formatting only
* move api util to utils area
* initial full swagger spec for nodeip search
* add api user role and fix api auth failure response
* update version of swagger-ui to 3.20.3
* add more openapi defs
* fixes to SQL and api spec
* clean up subs
* improvements to login/logout for API
* make api logout work
* add openapi tags to group operations
* allow api params to be generated from DBIC schema spec
* remove API calls for nodes and devices
* remove some poor assumptions about api calls
* tidy up
* remove DDP
* make login and logout similar
* example of api call being handled by ajax call
* make the branch authonly
also link to manual topology from device ports
fix tooltip reinit on datatables paging
Squashed commit of the following:
commit 85d52ed174
Merge: 66e3536607446ed7
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 31 08:59:19 2017 +0000
Merge branch 'master' into og-newnetmap
commit 66e353669d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 31 00:16:49 2017 +0000
add docs link
commit 809f1ede76
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 31 00:11:29 2017 +0000
node links to other devices
commit 187e1ea803
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 30 23:58:56 2017 +0000
add node infostring
commit 6453f99b28
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 30 23:41:19 2017 +0000
link infostrings!
commit f3ba260142
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 30 22:42:09 2017 +0000
maxnoderadius is now based on dynamic size calc
commit 0af415e391
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 30 22:33:22 2017 +0000
fix to get aggregate ports in matching order on left and right side
commit 3f36121a4c
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 30 22:07:28 2017 +0000
initial aggregate link summing
commit e5b96b0781
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 30 14:55:00 2017 +0000
tidy and refactor
commit 79f22a47b3
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 30 13:42:58 2017 +0000
get rid of d3 v3/v4 and only have v3
commit 18c3f16c8f
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 30 12:54:45 2017 +0000
fullscreen exit now works
commit 93f8181fc8
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 30 08:58:04 2017 +0000
implement show link speeds on links
commit 72961f172d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Thu Dec 28 23:51:26 2017 +0000
add speed to retrieved devicelinks
commit ae5496310e
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Dec 26 11:21:12 2017 +0000
remove annealing on label change
commit 885cb92bd3
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Dec 26 11:15:21 2017 +0000
reorder options
commit 015a1debe0
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Dec 26 11:11:01 2017 +0000
fix d3-force debug link messing up svg text elements
commit 10ccaa78b1
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Dec 26 10:52:47 2017 +0000
show ips functionality; improvement to maximise icon; bugfix d3-force label wrap
commit 71a1a20909
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon Dec 25 13:21:23 2017 +0000
move labels on neighbor view
commit 9578215349
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon Dec 25 13:13:13 2017 +0000
force running spinner
commit 6618fba548
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 23:13:33 2017 +0000
fix neighbors pinning
commit 938848551e
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 18:12:04 2017 +0000
implement dynamic sizing option
commit e4b19be5d7
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 13:25:00 2017 +0000
zoom to device on neighbor show; do not save positions on neighbor show
commit 58b576b464
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 13:10:48 2017 +0000
better neighbors/save interaction
commit 2c77660f2f
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 13:03:46 2017 +0000
fix labels on checkboxes to be clickable
commit 2c2dc62cda
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 12:50:26 2017 +0000
prevent save positions when in neighbor view mode
commit 78c558b5be
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 12:37:51 2017 +0000
add tooltip to show why host groups radio is disabled
commit 7000d4b0eb
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 12:12:29 2017 +0000
allow port control role to edit manual topology
commit 547d2934cb
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 12:09:40 2017 +0000
working save/restore node positions with groups
commit bf7e231ffe
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 24 10:09:26 2017 +0000
node colors!
commit e23594b041
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 23 23:42:17 2017 +0000
wooo device group selection is working
commit b1b22c9811
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 23 22:31:23 2017 +0000
send full query str from netmap for json
commit 9a13bb7f9b
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 23 22:06:24 2017 +0000
implement host_group_displaynames and fix vlan save bug
commit 35ed34ed2f
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 23 17:52:14 2017 +0000
when neighbors only can restrict the links search
commit f9e5ec02ab
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 23 17:44:41 2017 +0000
tweak mapshow
commit 9f9971ff4e
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 23 00:47:17 2017 +0000
better sidebar look and layout
commit 1def41da67
Author: Oliver Gorwits <oliver@cpan.org>
Date: Thu Dec 21 23:54:44 2017 +0000
sidebar with many options!!
commit 4203cbb225
Merge: 7c12df6b7db65932
Author: Oliver Gorwits <oliver@cpan.org>
Date: Thu Dec 21 20:14:55 2017 +0000
Merge branch 'master' into og-newnetmap
commit 7c12df6b83
Author: Oliver Gorwits <oliver@cpan.org>
Date: Thu Dec 21 00:13:03 2017 +0000
refresh tooltips when the datatables table is updated
commit 081b89d5dc
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed Dec 20 23:59:15 2017 +0000
shortcut from device port to manual topology
commit 9520438960
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed Dec 20 23:34:44 2017 +0000
tighten the sql for neighbor links
commit d6b0275931
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed Dec 20 23:26:22 2017 +0000
tweak layout again
commit 9e5e2f0ed5
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed Dec 20 23:18:57 2017 +0000
fullscreen mode for netmap
commit aba359d2a9
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed Dec 20 22:05:51 2017 +0000
save and load netmap positions for same device groups
commit b7cd0251ff
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Dec 19 23:24:47 2017 +0000
basic positions upload
commit 786115ebda
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Dec 19 22:13:16 2017 +0000
working group drag
commit 91f98d4f33
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon Dec 18 23:00:16 2017 +0000
drag all selected nodes
commit ba48eee5f7
Merge: 60412cd67f8e3edf
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon Dec 18 20:14:20 2017 +0000
Merge branch 'master' into og-newnetmap
commit 60412cd6e6
Merge: 39ca14917dc79f72
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 17 20:23:41 2017 +0000
Merge branch 'master' into og-newnetmap
commit 39ca1491b0
Merge: 69622582c8806bea
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 17 19:52:49 2017 +0000
Merge branch 'master' into og-newnetmap
commit 6962258223
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 17 19:52:26 2017 +0000
bleed versions
commit 444e82b8a3
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 16 15:33:41 2017 +0000
use d3 v4 only
commit 01cd7a908d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sat Dec 16 15:22:08 2017 +0000
two d3 connected
commit 854f37fc94
Author: Oliver Gorwits <oliver@cpan.org>
Date: Thu Dec 14 21:34:49 2017 +0000
more fiddling with d3
commit 61a3a9e06d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Thu Dec 14 19:15:21 2017 +0000
working worker
commit ebadebe69a
Author: Oliver Gorwits <oliver@cpan.org>
Date: Thu Dec 14 07:57:55 2017 +0000
borken worker
commit c327b28513
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed Dec 13 13:45:03 2017 +0000
fix node center
commit 33bd9d39da
Author: Oliver Gorwits <oliver@cpan.org>
Date: Wed Dec 13 01:14:50 2017 +0000
improvements to graphing
commit 6178bf2298
Merge: 1c627eaf925940c9
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Dec 12 23:44:48 2017 +0000
Merge branch 'master' into og-newnetmap
commit 1c627eaf3c
Merge: 77ca8f96000cbaa1
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue Dec 12 23:38:00 2017 +0000
Merge branch 'master' into og-newnetmap
commit 77ca8f96e1
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon Dec 11 00:09:08 2017 +0000
working rendering using d3-force plugin
commit 187265fc03
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun Dec 10 23:19:23 2017 +0000
initial work on moving to d3-force-apex-plugin - json data
we had situations where the manager would start workers on the same job,
either because of race conditions or because at the time of queueing it wasn't
known that the jobs were targeting the same device (due to device aliases).
this commit removes duplicate jobs, reduces the need for locking on the job
queue, and makes use of lldpRemChassisId to try to deduplicate jobs before
they are started. in effect we have several goes to prevent duplicate jobs:
1. at neighbor discovery time we try to skip queueing same lldpRemChassisId
2. at job selection we 'error out' jobs with same profile as job selected
3. at job selection we check for running job with same profile as selected
4. the job manager process also checks for duplicate job profiles
5. at job lock we abort if the job was 'errored out'
all together this seems to work well. a test on a large university network of
303 devices (four core routers and the rest edge routers, runing VRF with many
duplicate identities), ~1200 subnets, ~50k hosts, resulted in no DB deadlock
or contention and a complete discover+arpnip+macsuck (909 jobs) in ~3 minutes
(with ~150 duplicate jobs identified and skipped).
This commit adds a table 'device_skip' that is used to restrict job queue
searches to avoid jobs that are not permitted on this backend via *_no ACLs,
or jobs on devices that have previously encountered multiple SNMP timeouts.
When the backend loads or a device is added, a row is added to the table if
that device should not be polled on this backend (together with the job
actions which are to be skipped/denied). When a device SNMP connect fails a
counter in the same row (or a new row) is incremented.
There is also a new report 'SNMP Connect Failures' to show the devices with
non-zero SNMP connect failure counters. A configurable limit in the setting
'max_deferrals' is used to set the threshold of no longer polling the device.
To reset the deferrals/failures count, restart the Netdisco backend (which
regenerates 'device_skip' cache entries).
Squashed commit of the following:
commit b5e32c219d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 20:55:14 2017 +0100
show all failed connections in report
commit ffce3cee84
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 20:12:39 2017 +0100
only resolve fqdn once
commit cc4f680f01
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 20:10:20 2017 +0100
Revert "only resolve fqdn once"
This reverts commit 3d136a54de.
commit d8d082b30e
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 20:09:05 2017 +0100
a report to show SNMP failures
commit 3d136a54de
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 19:37:58 2017 +0100
only resolve fqdn once
commit 4550b8a84c
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 17:27:43 2017 +0100
skipover now implicit from deferrals/actionset; fix sql where logic with better correlation
commit b51edbccd2
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 16:11:29 2017 +0100
only abort lock if action matches badactions
commit 415559b24f
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 13:56:42 2017 +0100
set skipover true when adding to actionset
commit 1086f2c467
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 13:50:56 2017 +0100
fix empty actionset
commit 31962580b8
Merge: 9b2e993e6808133b
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 13:25:08 2017 +0100
Merge branch 'og-device_skip' of github.com:netdisco/netdisco into og-device_skip
commit 6808133bdb
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 13:19:54 2017 +0100
in-job checks for acls are required for netdisco-do foreground actions
commit 3944dd7813
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 13:18:30 2017 +0100
avoid extra device lookup
commit 9b2e993e0f
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 12:31:36 2017 +0100
also delete device_skip rows when deleting device
commit b55854e91d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 11:34:27 2017 +0100
actions in device_skip table are now an array/set
commit 5e126eef07
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 09:36:33 2017 +0100
typo
commit 44266f2767
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 09:14:25 2017 +0100
*able checks within jobs should not be necessary with skiplist
commit e7c22e7d11
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 08:58:57 2017 +0100
increment deferrals field when job is deferred
commit 88ae9c00ba
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 08:40:27 2017 +0100
turn connect fail into defer
commit eac1857043
Author: Oliver Gorwits <oliver@cpan.org>
Date: Tue May 23 08:26:59 2017 +0100
rename failures column to be deferrals
commit 96ed444bbb
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon May 22 22:52:51 2017 +0100
set up list of jobs the backend instance should skip
commit 3a0019296d
Author: Oliver Gorwits <oliver@cpan.org>
Date: Mon May 22 22:01:50 2017 +0100
separate out is_*able last_* checks
commit cf8589aba2
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun May 21 22:35:38 2017 +0100
change from ignore to skip name
commit ed193356f8
Author: Oliver Gorwits <oliver@cpan.org>
Date: Sun May 21 14:52:33 2017 +0100
device_ignore table to track devices to skip in polling
