Avoid lock/defer of jobs deined via ACL

This commit adds a table 'device_skip' that is used to restrict job queue
searches to avoid jobs that are not permitted on this backend via *_no ACLs,
or jobs on devices that have previously encountered multiple SNMP timeouts.

When the backend loads or a device is added, a row is added to the table if
that device should not be polled on this backend (together with the job
actions which are to be skipped/denied). When a device SNMP connect fails a
counter in the same row (or a new row) is incremented.

There is also a new report 'SNMP Connect Failures' to show the devices with
non-zero SNMP connect failure counters. A configurable limit in the setting
'max_deferrals' is used to set the threshold of no longer polling the device.

To reset the deferrals/failures count, restart the Netdisco backend (which
regenerates 'device_skip' cache entries).

Squashed commit of the following:

commit b5e32c219d
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 20:55:14 2017 +0100

    show all failed connections in report

commit ffce3cee84
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 20:12:39 2017 +0100

    only resolve fqdn once

commit cc4f680f01
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 20:10:20 2017 +0100

    Revert "only resolve fqdn once"

    This reverts commit 3d136a54de.

commit d8d082b30e
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 20:09:05 2017 +0100

    a report to show SNMP failures

commit 3d136a54de
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 19:37:58 2017 +0100

    only resolve fqdn once

commit 4550b8a84c
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 17:27:43 2017 +0100

    skipover now implicit from deferrals/actionset; fix sql where logic with better correlation

commit b51edbccd2
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 16:11:29 2017 +0100

    only abort lock if action matches badactions

commit 415559b24f
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 13:56:42 2017 +0100

    set skipover true when adding to actionset

commit 1086f2c467
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 13:50:56 2017 +0100

    fix empty actionset

commit 31962580b8
Merge: 9b2e993e 6808133b
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 13:25:08 2017 +0100

    Merge branch 'og-device_skip' of github.com:netdisco/netdisco into og-device_skip

commit 6808133bdb
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 13:19:54 2017 +0100

    in-job checks for acls are required for netdisco-do foreground actions

commit 3944dd7813
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 13:18:30 2017 +0100

    avoid extra device lookup

commit 9b2e993e0f
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 12:31:36 2017 +0100

    also delete device_skip rows when deleting device

commit b55854e91d
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 11:34:27 2017 +0100

    actions in device_skip table are now an array/set

commit 5e126eef07
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 09:36:33 2017 +0100

    typo

commit 44266f2767
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 09:14:25 2017 +0100

    *able checks within jobs should not be necessary with skiplist

commit e7c22e7d11
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 08:58:57 2017 +0100

    increment deferrals field when job is deferred

commit 88ae9c00ba
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 08:40:27 2017 +0100

    turn connect fail into defer

commit eac1857043
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Tue May 23 08:26:59 2017 +0100

    rename failures column to be deferrals

commit 96ed444bbb
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon May 22 22:52:51 2017 +0100

    set up list of jobs the backend instance should skip

commit 3a0019296d
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon May 22 22:01:50 2017 +0100

    separate out is_*able last_* checks

commit cf8589aba2
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun May 21 22:35:38 2017 +0100

    change from ignore to skip name

commit ed193356f8
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun May 21 14:52:33 2017 +0100

    device_ignore table to track devices to skip in polling

lib/App/Netdisco/Backend/Worker/Interactive/DeviceActions.pm

@@ -23,7 +23,7 @@ sub _set_device_generic {
 
   # snmp connect using rw community
   my $info = snmp_connect_rw($ip)
-    or return job_error("Failed to connect to device [$ip] to update $slot");
+    or return job_defer("Failed to connect to device [$ip] to update $slot");
 
   my $method = 'set_'. $slot;
   my $rv = $info->$method($data);

lib/App/Netdisco/Backend/Worker/Interactive/PortActions.pm

@@ -75,7 +75,7 @@ sub _set_port_generic {
   if ($device->vendor ne 'netdisco') {
       # snmp connect using rw community
       my $info = snmp_connect_rw($ip)
-        or return job_error("Failed to connect to device [$ip] to control port");
+        or return job_defer("Failed to connect to device [$ip] to control port");
 
       my $iid = get_iid($info, $port)
         or return job_error("Failed to get port ID for [$pn] from [$ip]");
@@ -128,7 +128,7 @@ sub power {
 
   # snmp connect using rw community
   my $info = snmp_connect_rw($ip)
-    or return job_error("Failed to connect to device [$ip] to control port");
+    or return job_defer("Failed to connect to device [$ip] to control power");
 
   my $powerid = get_powerid($info, $port)
     or return job_error("Failed to get power ID for [$pn] from [$ip]");

lib/App/Netdisco/Backend/Worker/Manager.pm

@@ -8,7 +8,8 @@ use App::Netdisco::Util::Backend;
 use Role::Tiny;
 use namespace::clean;
 
-use App::Netdisco::JobQueue qw/jq_locked jq_getsome jq_getsomep jq_lock/;
+use App::Netdisco::JobQueue
+  qw/jq_locked jq_getsome jq_getsomep jq_lock jq_prime_skiplist/;
 
 sub worker_begin {
   my $self = shift;
@@ -19,6 +20,9 @@ sub worker_begin {
 
   debug "entering Manager ($wid) worker_begin()";
 
+  # rebuild device skip hints
+  jq_prime_skiplist;
+
   # requeue jobs locally
   debug "mgr ($wid): searching for jobs booked to this processing node";
   my @jobs = jq_locked;

lib/App/Netdisco/Backend/Worker/Poller/Arpnip.pm

@@ -1,7 +1,7 @@
 package App::Netdisco::Backend::Worker::Poller::Arpnip;
 
 use App::Netdisco::Core::Arpnip 'do_arpnip';
-use App::Netdisco::Util::Device 'is_arpnipable';
+use App::Netdisco::Util::Device 'is_arpnipable_now';
 
 use Role::Tiny;
 use namespace::clean;
@@ -9,7 +9,7 @@ use namespace::clean;
 with 'App::Netdisco::Backend::Worker::Poller::Common';
 
 sub arpnip_action { \&do_arpnip }
-sub arpnip_filter { \&is_arpnipable }
+sub arpnip_filter { \&is_arpnipable_now }
 sub arpnip_layer { 3 }
 
 sub arpwalk { (shift)->_walk_body('arpnip', @_) }

lib/App/Netdisco/Backend/Worker/Poller/Common.pm

@@ -64,7 +64,7 @@ sub _single_body {
 
   my $snmp = snmp_connect($device);
   if (!defined $snmp) {
-      return job_error("$job_type failed: could not SNMP connect to $host");
+      return job_defer("$job_type failed: could not SNMP connect to $host");
   }
 
   unless ($snmp->has_layer( $job_layer )) {

lib/App/Netdisco/Backend/Worker/Poller/Device.pm

@@ -3,7 +3,7 @@ package App::Netdisco::Backend::Worker::Poller::Device;
 use Dancer qw/:moose :syntax :script/;
 
 use App::Netdisco::Util::SNMP 'snmp_connect';
-use App::Netdisco::Util::Device qw/get_device is_discoverable/;
+use App::Netdisco::Util::Device qw/get_device is_discoverable_now/;
 use App::Netdisco::Core::Discover ':all';
 use App::Netdisco::Backend::Util ':all';
 use App::Netdisco::JobQueue qw/jq_queued jq_insert/;
@@ -54,13 +54,13 @@ sub discover {
       return job_done("discover skipped: $host is pseudo-device");
   }
 
-  unless (is_discoverable($device->ip)) {
+  unless (is_discoverable_now($device)) {
       return job_defer("discover deferred: $host is not discoverable");
   }
 
   my $snmp = snmp_connect($device);
   if (!defined $snmp) {
-      return job_error("discover failed: could not SNMP connect to $host");
+      return job_defer("discover failed: could not SNMP connect to $host");
   }
 
   store_device($device, $snmp);

lib/App/Netdisco/Backend/Worker/Poller/Macsuck.pm

@@ -1,7 +1,7 @@
 package App::Netdisco::Backend::Worker::Poller::Macsuck;
 
 use App::Netdisco::Core::Macsuck 'do_macsuck';
-use App::Netdisco::Util::Device 'is_macsuckable';
+use App::Netdisco::Util::Device 'is_macsuckable_now';
 
 use Role::Tiny;
 use namespace::clean;
@@ -9,7 +9,7 @@ use namespace::clean;
 with 'App::Netdisco::Backend::Worker::Poller::Common';
 
 sub macsuck_action { \&do_macsuck }
-sub macsuck_filter { \&is_macsuckable }
+sub macsuck_filter { \&is_macsuckable_now }
 sub macsuck_layer { 2 }
 
 sub macwalk { (shift)->_walk_body('macsuck', @_) }

lib/App/Netdisco/Backend/Worker/Poller/Nbtstat.pm

@@ -5,7 +5,7 @@ use Dancer::Plugin::DBIC 'schema';
 
 use App::Netdisco::Core::Nbtstat qw/nbtstat_resolve_async store_nbt/;
 use App::Netdisco::Util::Node 'is_nbtstatable';
-use App::Netdisco::Util::Device qw/get_device is_discoverable/;
+use App::Netdisco::Util::Device qw/get_device is_macsuckable/;
 use App::Netdisco::Backend::Util ':all';
 
 use NetAddr::IP::Lite ':lower';
@@ -29,8 +29,8 @@ sub nbtstat  {
     or job_error("nbtstat failed: unable to interpret device parameter");
   my $host = $device->ip;
 
-  unless (is_discoverable($device->ip)) {
-      return job_defer("nbtstat deferred: $host is not discoverable");
+  unless (is_macsuckable($device)) {
+      return job_defer("nbtstat deferred: $host is not macsuckable");
   }
 
   # get list of nodes on device

lib/App/Netdisco/DB.pm

@@ -11,7 +11,7 @@ __PACKAGE__->load_namespaces(
 );
 
 our # try to hide from kwalitee
-  $VERSION = 41; # schema version used for upgrades, keep as integer
+  $VERSION = 42; # schema version used for upgrades, keep as integer
 
 use Path::Class;
 use File::ShareDir 'dist_dir';

lib/App/Netdisco/DB/Result/Admin.pm

@@ -56,6 +56,36 @@ __PACKAGE__->set_primary_key("job");
 
 # You can replace this text with custom code or comments, and it will be preserved on regeneration
 
+=head1 RELATIONSHIPS
+
+=head2 device_skips( $backend?, $max_deferrals? )
+
+Retuns the set of C<device_skip> entries which apply to this job. They match
+the device IP, current backend, and job action.
+
+You probably want to use the ResultSet method C<skipped> which completes this
+query with a C<backend> host and C<max_deferrals> parameters (or sensible
+defaults).
+
+=cut
+
+__PACKAGE__->has_many( device_skips => 'App::Netdisco::DB::Result::DeviceSkip',
+  sub {
+    my $args = shift;
+    return {
+      "$args->{foreign_alias}.backend" => { '=' => \'?' },
+      "$args->{foreign_alias}.device"
+        => { -ident => "$args->{self_alias}.device" },
+      -or => [
+        { "$args->{foreign_alias}.actionset"
+            => { '@>' => \"string_to_array($args->{self_alias}.action,'')" } },
+        { "$args->{foreign_alias}.deferrals" => { '>=' => \'?' } },
+      ],
+    };
+  },
+  { cascade_copy => 0, cascade_update => 0, cascade_delete => 0 }
+);
+
 =head1 METHODS
 
 =head2 summary

lib/App/Netdisco/DB/Result/DeviceSkip.pm

@@ -0,0 +1,55 @@
+use utf8;
+package App::Netdisco::DB::Result::DeviceSkip;
+
+use strict;
+use warnings;
+
+use List::MoreUtils ();
+
+use base 'DBIx::Class::Core';
+__PACKAGE__->table("device_skip");
+__PACKAGE__->add_columns(
+  "backend",
+  { data_type => "text", is_nullable => 0 },
+  "device",
+  { data_type => "inet", is_nullable => 0 },
+  "actionset",
+  { data_type => "text[]", is_nullable => 0, default_value => '{}' },
+  "deferrals",
+  { data_type => "integer", is_nullable => 1, default_value => '0' },
+);
+
+__PACKAGE__->set_primary_key("backend", "device");
+
+__PACKAGE__->add_unique_constraint(
+  device_skip_pkey => [qw/backend device/]);
+
+=head1 METHODS
+
+=head2 increment_deferrals
+
+Increments the C<deferrals> field in the row, only if the row is in storage.
+There is a race in the update, but this is not worrying for now.
+
+=cut
+
+sub increment_deferrals {
+  my $row = shift;
+  return unless $row->in_storage;
+  return $row->update({ deferrals => (($row->deferrals || 0) + 1) });
+}
+
+=head2 add_to_actionset
+
+=cut
+
+sub add_to_actionset {
+  my ($row, @badactions) = @_;
+  return unless $row->in_storage;
+  return unless scalar @badactions;
+  return $row->update({ actionset =>
+    [ sort (List::MoreUtils::uniq( @{ $row->actionset || [] }, @badactions )) ]
+  });
+}
+
+1;

lib/App/Netdisco/DB/ResultSet.pm

@@ -5,8 +5,11 @@ use warnings;
 
 use base 'DBIx::Class::ResultSet';
 
-__PACKAGE__->load_components(
-    qw{Helper::ResultSet::SetOperations Helper::ResultSet::Shortcut});
+__PACKAGE__->load_components(qw/
+  Helper::ResultSet::SetOperations
+  Helper::ResultSet::Shortcut
+  Helper::ResultSet::CorrelateRelationship
+/);
 
 =head1 ADDITIONAL METHODS
 

lib/App/Netdisco/DB/ResultSet/Admin.pm

@@ -4,12 +4,34 @@ use base 'App::Netdisco::DB::ResultSet';
 use strict;
 use warnings;
 
+use Net::Domain 'hostfqdn';
+
 __PACKAGE__->load_components(qw/
   +App::Netdisco::DB::ExplicitLocking
 /);
 
 =head1 ADDITIONAL METHODS
 
+=head2 skipped
+
+Retuns a correlated subquery for the set of C<device_skip> entries that apply
+to some jobs. They match the device IP, current backend, and job action.
+
+Pass the C<backend> FQDN (or the current host will be used as a default), and
+the C<max_deferrals> (or 10 will be used as the default).
+
+=cut
+
+sub skipped {
+  my ($rs, $backend, $max_deferrals) = @_;
+  $backend ||= (hostfqdn || 'localhost');
+  $max_deferrals ||= 10;
+
+  return $rs->correlate('device_skips')->search(undef, {
+    bind => [[deferrals => $max_deferrals], [backend => $backend]],
+  });
+}
+
 =head2 with_times
 
 This is a modifier for any C<search()> (including the helpers below) which

lib/App/Netdisco/DB/ResultSet/Device.pm

@@ -596,9 +596,14 @@ sub delete {
       )->delete;
   }
 
-  $schema->resultset('Admin')->search({
-    device => { '-in' => $devices->as_query },
-  })->delete;
+  foreach my $set (qw/
+    Admin
+    DeviceSkip
+  /) {
+      $schema->resultset($set)->search(
+        { device => { '-in' => $devices->as_query } },
+      )->delete;
+  }
 
   $schema->resultset('Topology')->search({
     -or => [

lib/App/Netdisco/JobQueue.pm

@@ -13,6 +13,7 @@ our @EXPORT_OK = qw/
   jq_getsomep
   jq_locked
   jq_queued
+  jq_prime_skiplist
   jq_log
   jq_userlog
   jq_lock
@@ -57,6 +58,12 @@ Netdisco job instance interface (see below).
 Returns a list of IP addresses of devices which currently have a job of the
 given C<$job_type> queued (e.g. C<discover>, C<arpnip>, etc).
 
+=head2 jq_prime_skiplist()
+
+Sets up a table of hints for the backend daemon manager to help avoid picking
+jobs from the queue that it cannot process due to C<*_no> configuration
+settings.
+
 =head2 jq_log()
 
 Returns a list of the most recent 50 jobs in the queue. Jobs are returned as

lib/App/Netdisco/JobQueue/PostgreSQL.pm

@@ -3,7 +3,11 @@ package App::Netdisco::JobQueue::PostgreSQL;
 use Dancer qw/:moose :syntax :script/;
 use Dancer::Plugin::DBIC 'schema';
 
+use App::Netdisco::Util::Device
+  qw/is_discoverable is_macsuckable is_arpnipable/;
 use App::Netdisco::Backend::Job;
+
+
 use Net::Domain 'hostfqdn';
 use Module::Load ();
 use Try::Tiny;
@@ -15,26 +19,35 @@ our @EXPORT_OK = qw/
   jq_getsomep
   jq_locked
   jq_queued
-  jq_log
-  jq_userlog
+  jq_prime_skiplist
   jq_lock
   jq_defer
   jq_complete
+  jq_log
+  jq_userlog
   jq_insert
   jq_delete
 /;
 our %EXPORT_TAGS = ( all => \@EXPORT_OK );
 
+# this can take a few seconds - only do it once
+our $fqdn = undef;
+
 sub _getsome {
   my ($num_slots, $where) = @_;
   return () if ((!defined $num_slots) or ($num_slots < 1));
   return () if ((!defined $where) or (ref {} ne ref $where));
 
-  my $rs = schema('netdisco')->resultset('Admin')
-    ->search(
-      { status => 'queued', %$where },
-      { order_by => 'random()', rows => $num_slots },
-    );
+  my $fqdn ||= (hostfqdn || 'localhost');
+  my $jobs = schema('netdisco')->resultset('Admin');
+
+  my $rs = $jobs->search({
+    status => 'queued',
+    device => { '-not_in' =>
+      $jobs->skipped($fqdn, setting('workers')->{'max_deferrals'})
+           ->columns('device')->as_query },
+    %$where,
+  }, { order_by => 'random()', rows => $num_slots });
 
   my @returned = ();
   while (my $job = $rs->next) {
@@ -61,7 +74,7 @@ sub jq_getsomep {
 }
 
 sub jq_locked {
-  my $fqdn = hostfqdn || 'localhost';
+  my $fqdn ||= (hostfqdn || 'localhost');
   my @returned = ();
 
   my $rs = schema('netdisco')->resultset('Admin')
@@ -83,26 +96,65 @@ sub jq_queued {
   })->get_column('device')->all;
 }
 
-sub jq_log {
-  return schema('netdisco')->resultset('Admin')->search({}, {
-    order_by => { -desc => [qw/entered device action/] },
-    rows => 50,
-  })->with_times->hri->all;
+# given a device, tests if any of the primary acls applies
+# returns a list of job actions to be denied/skipped on this host.
+sub _get_denied_actions {
+  my $device = shift;
+  my @badactions = ();
+
+  push @badactions, ('discover', @{ setting('job_prio')->{high} })
+    if not is_discoverable($device);
+
+  push @badactions, (qw/macsuck nbtstat/)
+    if not is_macsuckable($device);
+
+  push @badactions, 'arpnip'
+    if not is_arpnipable($device);
+
+  return @badactions;
 }
 
-sub jq_userlog {
-  my $user = shift;
-  return schema('netdisco')->resultset('Admin')->search({
-    username => $user,
-    finished => { '>' => \"(now() - interval '5 seconds')" },
-  })->with_times->all;
+sub jq_prime_skiplist {
+  my $fqdn ||= (hostfqdn || 'localhost');
+  my @devices = schema('netdisco')->resultset('Device')->all;
+  my $rs = schema('netdisco')->resultset('DeviceSkip');
+  my %actionset = ();
+
+  foreach my $d (@devices) {
+    my @badactions = _get_denied_actions($d);
+    $actionset{$d->ip} = \@badactions if scalar @badactions;
+  }
+
+  schema('netdisco')->txn_do(sub {
+    $rs->search({ backend => $fqdn })->delete;
+    $rs->populate([
+      map {{
+        backend => $fqdn,
+        device  => $_,
+        actionset => $actionset{$_},
+      }} keys %actionset
+    ]);
+  });
 }
 
 sub jq_lock {
   my $job = shift;
-  my $fqdn = hostfqdn || 'localhost';
+  my $fqdn ||= (hostfqdn || 'localhost');
   my $happy = false;
 
+  # need to handle device discovered since backend daemon started
+  # and the skiplist was primed. these should be checked against
+  # the various acls and have device_skip entry added if needed,
+  # and return false if it should have been skipped.
+  my @badactions = _get_denied_actions($job->device);
+  if (scalar @badactions) {
+    schema('netdisco')->resultset('DeviceSkip')->find_or_create({
+      backend => $fqdn, device => $job->device,
+    },{ key => 'device_skip_pkey' })->add_to_actionset(@badactions);
+
+    return false if scalar grep {$_ eq $job->action} @badactions;
+  }
+
   # lock db row and update to show job has been picked
   try {
     schema('netdisco')->txn_do(sub {
@@ -136,11 +188,23 @@ sub jq_lock {
 
 sub jq_defer {
   my $job = shift;
+  my $fqdn ||= (hostfqdn || 'localhost');
   my $happy = false;
 
+  # note this taints all actions on the device. for example if both
+  # macsuck and arpnip are allowed, but macsuck fails 10 times, then
+  # arpnip (and every other action) will be prevented on the device.
+
+  # seeing as defer is only triggered by an SNMP connect failure, this
+  # behaviour seems reasonable, to me (or desirable, perhaps).
+
   try {
-    # lock db row and update to show job is available
     schema('netdisco')->txn_do(sub {
+      schema('netdisco')->resultset('DeviceSkip')->find_or_create({
+        backend => $fqdn, device => $job->device,
+      },{ key => 'device_skip_pkey' })->increment_deferrals;
+
+      # lock db row and update to show job is available
       schema('netdisco')->resultset('Admin')
         ->find($job->job, {for => 'update'})
         ->update({ status => 'queued', started => undef });
@@ -178,6 +242,21 @@ sub jq_complete {
   return $happy;
 }
 
+sub jq_log {
+  return schema('netdisco')->resultset('Admin')->search({}, {
+    order_by => { -desc => [qw/entered device action/] },
+    rows => 50,
+  })->with_times->hri->all;
+}
+
+sub jq_userlog {
+  my $user = shift;
+  return schema('netdisco')->resultset('Admin')->search({
+    username => $user,
+    finished => { '>' => \"(now() - interval '5 seconds')" },
+  })->with_times->all;
+}
+
 sub jq_insert {
   my $jobs = shift;
   $jobs = [$jobs] if ref [] ne ref $jobs;

lib/App/Netdisco/Util/Device.pm

@@ -11,9 +11,9 @@ our @EXPORT_OK = qw/
   delete_device
   renumber_device
   match_devicetype
-  is_discoverable
-  is_arpnipable
-  is_macsuckable
+  is_discoverable is_discoverable_now
+  is_arpnipable   is_arpnipable_now
+  is_macsuckable  is_macsuckable_now
 /;
 our %EXPORT_TAGS = (all => \@EXPORT_OK);
 
@@ -144,6 +144,8 @@ sub match_devicetype {
                         @{setting($setting_name) || []});
 }
 
+sub _bail_msg { debug $_[0]; return 0; }
+
 =head2 is_discoverable( $ip, $device_type? )
 
 Given an IP address, returns C<true> if Netdisco on this host is permitted by
@@ -159,8 +161,6 @@ Returns false if the host is not permitted to discover the target device.
 
 =cut
 
-sub _bail_msg { debug $_[0]; return 0; }
-
 sub is_discoverable {
   my ($ip, $remote_type) = @_;
   my $device = get_device($ip) or return 0;
@@ -175,16 +175,32 @@ sub is_discoverable {
   return _bail_msg("is_discoverable: device failed to match discover_only")
     unless check_acl_only($device, 'discover_only');
 
-  # cannot check last_discover for as yet undiscovered devices :-)
-  return 1 if not $device->in_storage;
+  return 1;
+}
 
-  if ($device->since_last_discover and setting('discover_min_age')
-      and $device->since_last_discover < setting('discover_min_age')) {
+=head2 is_discoverable_now( $ip, $device_type? )
 
-      return _bail_msg("is_discoverable: time since last discover less than discover_min_age");
+Same as C<is_discoverable>, but also checks the last_discover field if the
+device is in storage, and returns false if that host has been too recently
+discovered.
+
+Returns false if the host is not permitted to discover the target device.
+
+=cut
+
+sub is_discoverable_now {
+  my ($ip, $remote_type) = @_;
+  my $device = get_device($ip) or return 0;
+
+  if ($device->in_storage) {
+    if ($device->since_last_discover and setting('discover_min_age')
+        and $device->since_last_discover < setting('discover_min_age')) {
+
+        return _bail_msg("is_discoverable: time since last discover less than discover_min_age");
+    }
   }
 
-  return 1;
+  return is_discoverable(@_);
 }
 
 =head2 is_arpnipable( $ip )
@@ -209,6 +225,23 @@ sub is_arpnipable {
   return _bail_msg("is_arpnipable: device failed to match arpnip_only")
     unless check_acl_only($device, 'arpnip_only');
 
+  return 1;
+}
+
+=head2 is_arpnipable_now( $ip )
+
+Same as C<is_arpnipable>, but also checks the last_arpnip field if the
+device is in storage, and returns false if that host has been too recently
+arpnipped.
+
+Returns false if the host is not permitted to arpnip the target device.
+
+=cut
+
+sub is_arpnipable_now {
+  my $ip = shift;
+  my $device = get_device($ip) or return 0;
+
   return _bail_msg("is_arpnipable: cannot arpnip an undiscovered device")
     if not $device->in_storage;
 
@@ -218,7 +251,7 @@ sub is_arpnipable {
       return _bail_msg("is_arpnipable: time since last arpnip less than arpnip_min_age");
   }
 
-  return 1;
+  return is_arpnipable(@_);
 }
 
 =head2 is_macsuckable( $ip )
@@ -243,6 +276,23 @@ sub is_macsuckable {
   return _bail_msg("is_macsuckable: device failed to match macsuck_only")
     unless check_acl_only($device, 'macsuck_only');
 
+  return 1;
+}
+
+=head2 is_macsuckable_now( $ip )
+
+Same as C<is_macsuckable>, but also checks the last_macsuck field if the
+device is in storage, and returns false if that host has been too recently
+macsucked.
+
+Returns false if the host is not permitted to macsuck the target device.
+
+=cut
+
+sub is_macsuckable_now {
+  my $ip = shift;
+  my $device = get_device($ip) or return 0;
+
   return _bail_msg("is_macsuckable: cannot macsuck an undiscovered device")
     if not $device->in_storage;
 
@@ -252,7 +302,7 @@ sub is_macsuckable {
       return _bail_msg("is_macsuckable: time since last macsuck less than macsuck_min_age");
   }
 
-  return 1;
+  return is_macsuckable(@_);
 }
 
 1;

lib/App/Netdisco/Web/Plugin/AdminTask/TimedOutDevices.pm

@@ -0,0 +1,28 @@
+package App::Netdisco::Web::Plugin::AdminTask::TimedOutDevices;
+
+use Dancer ':syntax';
+use Dancer::Plugin::Ajax;
+use Dancer::Plugin::DBIC;
+use Dancer::Plugin::Auth::Extensible;
+
+use App::Netdisco::Web::Plugin;
+
+register_admin_task({
+  tag => 'timedoutdevices',
+  label => 'SNMP Connect Failures',
+});
+
+ajax '/ajax/content/admin/timedoutdevices' => require_role admin => sub {
+    my @results = schema('netdisco')->resultset('DeviceSkip')->search({
+      deferrals => { '>' => 0 }
+    },{ order_by =>
+      [{ -desc => 'deferrals' }, { -asc => [qw/device backend/] }]
+    })->hri->all;
+
+    content_type('text/html');
+    template 'ajax/admintask/timedoutdevices.tt', {
+      results => \@results,
+    }, { layout => undef };
+};
+
+true;

share/config.yml

@@ -71,6 +71,7 @@ web_plugins:
   - AdminTask::SlowDevices
   - AdminTask::UndiscoveredNeighbors
   - AdminTask::OrphanedDevices
+  - AdminTask::TimedOutDevices
   - AdminTask::UserLog
   - AdminTask::Users
   - Search::Device
@@ -205,6 +206,7 @@ workers:
   tasks: 'AUTO * 2'
   sleep_time: 1
   min_runtime: 0
+  max_deferrals: 10
   queue: PostgreSQL
 
 dns:

share/schema_versions/App-Netdisco-DB-41-42-PostgreSQL.sql

@@ -0,0 +1,11 @@
+BEGIN;
+
+CREATE TABLE "device_skip" (
+  "backend" text NOT NULL,
+  "device" inet NOT NULL,
+  "actionset" text[] DEFAULT '{}',
+  "deferrals" integer DEFAULT 0,
+  PRIMARY KEY ("backend", "device")
+);
+
+COMMIT;

share/views/ajax/admintask/timedoutdevices.tt

@@ -0,0 +1,31 @@
+[% IF NOT results.size %]
+<div class="span4 alert alert-info">No significant events to report.</div>
+[% ELSE %]
+<table id="data-table" class="table table-striped table-bordered" width="100%" cellspacing="0">
+  <thead>
+    <tr>
+      <th class="nd_center-cell">Poller Host</th>
+      <th class="nd_center-cell">Device</th>
+      <th class="nd_center-cell">Failed Connections</th>
+    </tr>
+  </thead>
+  </tbody>
+    [% FOREACH row IN results %]
+    <tr>
+      <td class="nd_center-cell">[% row.backend | html_entity %]</td>
+      <td class="nd_center-cell"><a class="nd_linkcell"
+        href="[% uri_for('/device') %]?q=[% row.device | uri %]">[% row.device | html_entity %]</a></td>
+      <td class="nd_center-cell">[% row.deferrals | html_entity %]</td>
+    </tr>
+    [% END %]
+  </tbody>
+</table>
+[% END %]
+
+<script>
+$(document).ready(function() {
+  $('#data-table').dataTable({
+[% INCLUDE 'ajax/datatabledefaults.tt' -%]
+  } );
+} );
+</script>