This commit adds a table 'device_skip' that is used to restrict job queue searches to avoid jobs that are not permitted on this backend via *_no ACLs, or jobs on devices that have previously encountered multiple SNMP timeouts. When the backend loads or a device is added, a row is added to the table if that device should not be polled on this backend (together with the job actions which are to be skipped/denied). When a device SNMP connect fails a counter in the same row (or a new row) is incremented. There is also a new report 'SNMP Connect Failures' to show the devices with non-zero SNMP connect failure counters. A configurable limit in the setting 'max_deferrals' is used to set the threshold of no longer polling the device. To reset the deferrals/failures count, restart the Netdisco backend (which regenerates 'device_skip' cache entries). Squashed commit of the following: commitb5e32c219dAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 20:55:14 2017 +0100 show all failed connections in report commitffce3cee84Author: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 20:12:39 2017 +0100 only resolve fqdn once commitcc4f680f01Author: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 20:10:20 2017 +0100 Revert "only resolve fqdn once" This reverts commit3d136a54de. commitd8d082b30eAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 20:09:05 2017 +0100 a report to show SNMP failures commit3d136a54deAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 19:37:58 2017 +0100 only resolve fqdn once commit4550b8a84cAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 17:27:43 2017 +0100 skipover now implicit from deferrals/actionset; fix sql where logic with better correlation commitb51edbccd2Author: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 16:11:29 2017 +0100 only abort lock if action matches badactions commit415559b24fAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 13:56:42 2017 +0100 set skipover true when adding to actionset commit1086f2c467Author: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 13:50:56 2017 +0100 fix empty actionset commit31962580b8Merge:9b2e993e6808133bAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 13:25:08 2017 +0100 Merge branch 'og-device_skip' of github.com:netdisco/netdisco into og-device_skip commit6808133bdbAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 13:19:54 2017 +0100 in-job checks for acls are required for netdisco-do foreground actions commit3944dd7813Author: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 13:18:30 2017 +0100 avoid extra device lookup commit9b2e993e0fAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 12:31:36 2017 +0100 also delete device_skip rows when deleting device commitb55854e91dAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 11:34:27 2017 +0100 actions in device_skip table are now an array/set commit5e126eef07Author: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 09:36:33 2017 +0100 typo commit44266f2767Author: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 09:14:25 2017 +0100 *able checks within jobs should not be necessary with skiplist commite7c22e7d11Author: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 08:58:57 2017 +0100 increment deferrals field when job is deferred commit88ae9c00baAuthor: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 08:40:27 2017 +0100 turn connect fail into defer commiteac1857043Author: Oliver Gorwits <oliver@cpan.org> Date: Tue May 23 08:26:59 2017 +0100 rename failures column to be deferrals commit96ed444bbbAuthor: Oliver Gorwits <oliver@cpan.org> Date: Mon May 22 22:52:51 2017 +0100 set up list of jobs the backend instance should skip commit3a0019296dAuthor: Oliver Gorwits <oliver@cpan.org> Date: Mon May 22 22:01:50 2017 +0100 separate out is_*able last_* checks commitcf8589aba2Author: Oliver Gorwits <oliver@cpan.org> Date: Sun May 21 22:35:38 2017 +0100 change from ignore to skip name commited193356f8Author: Oliver Gorwits <oliver@cpan.org> Date: Sun May 21 14:52:33 2017 +0100 device_ignore table to track devices to skip in polling
143 lines
3.2 KiB
Perl
143 lines
3.2 KiB
Perl
package App::Netdisco::JobQueue;
|
|
|
|
use Dancer qw/:moose :syntax :script/;
|
|
|
|
use Module::Load ();
|
|
Module::Load::load
|
|
'App::Netdisco::JobQueue::' . setting('workers')->{queue} => ':all';
|
|
|
|
use base 'Exporter';
|
|
our @EXPORT = ();
|
|
our @EXPORT_OK = qw/
|
|
jq_getsome
|
|
jq_getsomep
|
|
jq_locked
|
|
jq_queued
|
|
jq_prime_skiplist
|
|
jq_log
|
|
jq_userlog
|
|
jq_lock
|
|
jq_defer
|
|
jq_complete
|
|
jq_insert
|
|
jq_delete
|
|
/;
|
|
our %EXPORT_TAGS = ( all => \@EXPORT_OK );
|
|
|
|
=head1 NAME
|
|
|
|
App::Netdisco::JobQueue
|
|
|
|
=head1 DESCRIPTION
|
|
|
|
Interface for Netdisco job queue.
|
|
|
|
There are no default exports, however the C<:all> tag will export all
|
|
subroutines.
|
|
|
|
=head1 EXPORT_OK
|
|
|
|
=head2 jq_getsome( $num? )
|
|
|
|
Returns a list of randomly selected queued jobs. Default is to return one job,
|
|
unless C<$num> is provided. Jobs are returned as objects which implement the
|
|
Netdisco job instance interface (see below).
|
|
|
|
=head2 jq_getsomep( $num? )
|
|
|
|
Same as C<jq_getsome> but for high priority jobs.
|
|
|
|
=head2 jq_locked()
|
|
|
|
Returns the list of jobs currently booked out to this processing node (denoted
|
|
by the local hostname). Jobs are returned as objects which implement the
|
|
Netdisco job instance interface (see below).
|
|
|
|
=head2 jq_queued( $job_type )
|
|
|
|
Returns a list of IP addresses of devices which currently have a job of the
|
|
given C<$job_type> queued (e.g. C<discover>, C<arpnip>, etc).
|
|
|
|
=head2 jq_prime_skiplist()
|
|
|
|
Sets up a table of hints for the backend daemon manager to help avoid picking
|
|
jobs from the queue that it cannot process due to C<*_no> configuration
|
|
settings.
|
|
|
|
=head2 jq_log()
|
|
|
|
Returns a list of the most recent 50 jobs in the queue. Jobs are returned as
|
|
objects which implement the Netdisco job instance interface (see below).
|
|
|
|
=head2 jq_userlog( $user )
|
|
|
|
Returns a list of jobs which have been entered into the queue by the passed
|
|
C<$user>. Jobs are returned as objects which implement the Netdisco job
|
|
instance interface (see below).
|
|
|
|
=head2 jq_lock( $job )
|
|
|
|
Marks a job in the queue as booked out to this processing node (denoted by the
|
|
local hostname). The C<$job> parameter must be an object which implements the
|
|
Netdisco job instance interface (see below).
|
|
|
|
Returns true if successful else returns false.
|
|
|
|
=head2 jq_defer( $job )
|
|
|
|
Marks a job in the queue as available for taking. This is usually done after a
|
|
job is booked but the processing node changes its mind and decides to return
|
|
the job to the queue. The C<$job> parameter must be an object which implements
|
|
the Netdisco job instance interface (see below).
|
|
|
|
Returns true if successful else returns false.
|
|
|
|
=head2 jq_complete( $job )
|
|
|
|
Marks a job as complete. The C<$job> parameter must be an object which
|
|
implements the Netdisco job instance interface (see below). The queue item's
|
|
status, log and finished fields will be updated from the passed C<$job>.
|
|
|
|
Returns true if successful else returns false.
|
|
|
|
=head2 jq_insert( \%job | [ \%job, \%job ...] )
|
|
|
|
Adds the passed jobs to the queue.
|
|
|
|
=head2 jq_delete( $id? )
|
|
|
|
If passed the ID of a job, deletes it from the queue. Otherwise deletes ALL
|
|
jobs from the queue.
|
|
|
|
=head1 Job Instance Interface
|
|
|
|
=head2 id (auto)
|
|
|
|
=head2 entered
|
|
|
|
=head2 started
|
|
|
|
=head2 finished
|
|
|
|
=head2 device
|
|
|
|
=head2 port
|
|
|
|
=head2 action
|
|
|
|
=head2 subaction or extra
|
|
|
|
=head2 status
|
|
|
|
=head2 username
|
|
|
|
=head2 userip
|
|
|
|
=head2 log
|
|
|
|
=head2 debug
|
|
|
|
=cut
|
|
|
|
true;
|