Bug fixes, and AuthN delegation.

Squashed commit of the following:

commit 25bc026dc5e0177cd3aa81c11cdace091eb68f36
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Jun 17 08:16:56 2013 +0100

    bump version for new release

commit d4042f6e8db42c7a85df4dcf9690fec72ad2db69
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 18:27:52 2013 +0100

    Job Queue page play/pause/refresh controls

commit b6c9152516d7800409b7a73c5d0cdce6dd405492
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 17:50:06 2013 +0100

    limit size of job queue table

commit ac9e5feb8b774071fcf4423dd862dced74dee9e6
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 17:47:55 2013 +0100

    update bugs link

commit 9c0fb0e9aedc6297f4462c3cf88343f6d0df40b6
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 17:41:29 2013 +0100

    update MANIFEST

commit 7aaa2fff91ed2b1839bdbb79081d90ad3e144f47
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 17:40:35 2013 +0100

    Fix Plack middleware config for Expiry

commit 313e2cf014cf0da7cf85074e390ad394b28bf42d
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 17:23:00 2013 +0100

    Support for delegated authentication with REMOTE_USER and X-REMOTE_USER

commit 85e21f2bf296c4a5ca6b5afb5091694e56e3031f
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 14:24:08 2013 +0100

    Add tooltip showing the job queue item logged status message

commit 9b14f53ebed51eb46ea278807cfe8a2fbd28743c
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 13:43:26 2013 +0100

    Increase default frequency of job queue polling to 2 seconds

commit 6ba46818d8ab2100c652c8eb8e98bc6f5a54e273
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 12:57:43 2013 +0100

    workaround for https://github.com/PerlDancer/Dancer/issues/935

commit c7a2d8a9d45716959bedbbb8db4cdd82a5950642
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Jun 16 11:54:18 2013 +0100

    Fix hyperlinks when running behind reverse proxy on custom path

commit 0620efa404bc25cb0a9ada5aa6f1b092d5c4d482
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Jun 15 18:31:19 2013 +0100

    update deploy docs

commit 857b1c7aa0fe832f8948349eda5211eb38ba3099
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Jun 15 18:16:50 2013 +0100

    add note about compiler dependency

commit 02a2ad6b2c52db9fbc1e24bc8888f658dc7084ad
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Jun 15 17:44:29 2013 +0100

    sort vlans, macs, ips in device port view

commit 097bad77310728a98b261a2cfca4de7ab50be94b
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Jun 15 16:32:20 2013 +0100

    hint when calling web in fg without starman

commit 6425d89ddb2b56129c610482134482d8f9455d40
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Jun 15 15:53:26 2013 +0100

    macwalk and arpwalk refactored

commit d527b9d05addc82fb38c84f6fea1aa5818fc68d5
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Thu Jun 13 22:27:34 2013 +0100

    implement is_macsuckable and is_arpnipable

commit 7af10ed313e25f5d99a22b53ba438225c2259069
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Thu Jun 13 22:17:39 2013 +0100

    version bump

commit 8ace3bf8fa48cf3e14bdf86fad5a4862aad50a4b
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Thu Jun 13 22:14:05 2013 +0100

    tidy up user menu

commit e6eef605c248471dbfe7ec62cd04d73d653523ca
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Thu Jun 13 22:02:52 2013 +0100

    Add discoverall, macwalk, arpwalk items to the Admin Tasks menu

commit 2631fabd1eccd8a3971e4762eebe57f406623bee
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Thu Jun 13 21:21:50 2013 +0100

    remove length() which only became sane in 5.12

commit a7b7169070a58685cacde26a3b6d462e74be9928
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Thu Jun 13 19:07:56 2013 +0100

    Use DBIx::Class new collapsed query support when we can

commit 77cddab8ba7033ccb1ecae257bafa4eef8f99f47
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Wed Jun 12 17:26:47 2013 +0100

    Database config simplified to only four essential settings

commit 6ed0802bf2ab0fd898ce6945451b8ca6566ae551
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Wed Jun 12 13:03:20 2013 +0100

    Ask to set up guest user for Admin/Port Control rights in deploy script

Netdisco/Changes

@@ -1,3 +1,27 @@
+2.009000_001 - 
+
+  [NEW FEATURES]
+
+  * Support for delegated authentication with REMOTE_USER and X-REMOTE_USER
+  * Ask to set up guest user for Admin/Port Control rights in deploy script
+  * Job Queue page play/pause/refresh controls
+
+  [ENHANCEMENTS]
+
+  * Database config simplified to only four essential settings
+  * Use DBIx::Class new collapsed query support when we can
+  * Add discoverall, macwalk, arpwalk items to the Admin Tasks menu
+  * Increase default frequency of job queue polling to 2 seconds
+  * Add tooltip showing the job queue item logged status message
+
+  [BUG FIXES]
+
+  * Macwalk and Arpwalk job defer fixed
+  * Sort VLANs, MACs, IPs properly in Device Port view
+  * Fix hyperlinks when running behind reverse proxy on custom path
+  * Add workaround for https://github.com/PerlDancer/Dancer/issues/935
+  * Fix Plack middleware config for Expiry
+
 2.008002 - 2013-06-11
 
   [ENHANCEMENTS]

Netdisco/MANIFEST

@@ -33,6 +33,7 @@ lib/App/Netdisco/Daemon/Worker/Interactive/PortActions.pm
 lib/App/Netdisco/Daemon/Worker/Manager.pm
 lib/App/Netdisco/Daemon/Worker/Poller.pm
 lib/App/Netdisco/Daemon/Worker/Poller/Arpnip.pm
+lib/App/Netdisco/Daemon/Worker/Poller/Common.pm
 lib/App/Netdisco/Daemon/Worker/Poller/Device.pm
 lib/App/Netdisco/Daemon/Worker/Poller/Macsuck.pm
 lib/App/Netdisco/Daemon/Worker/Scheduler.pm

Netdisco/META.yml

@@ -60,4 +60,4 @@ resources:
   homepage: http://netdisco.org/
   license: http://opensource.org/licenses/bsd-license.php
   repository: git://git.code.sf.net/p/netdisco/netdisco-ng
-version: 2.008002
+version: 2.010000

Netdisco/bin/netdisco-deploy

@@ -96,6 +96,36 @@ $bool = $term->ask_yn(
 );
 deploy_db() if $bool;
 
+my $users = schema('netdisco')->resultset('User');
+if ($users->count == 0 and setting('no_auth')) {
+    say '';
+    $bool = $term->ask_yn(
+      prompt => 'Would you like the default web user to have Admin rights (discover, etc)?',
+      default => 'n',
+    );
+
+    if ($bool) {
+        $users->create({
+          username => 'guest',
+          admin => 'true',
+          port_control => 'true',
+        });
+    }
+    else {
+        say '';
+        $bool = $term->ask_yn(
+          prompt => 'Would you like the default web user to have Port Control rights?',
+          default => 'n',
+        );
+        if ($bool) {
+            $users->create({
+              username => 'guest',
+              port_control => 'true',
+            });
+        }
+    }
+}
+
 say '';
 $bool = $term->ask_yn(
   prompt => 'Download and update vendor MAC prefixes (OUI data)?', default => 'n',

Netdisco/bin/netdisco-do

@@ -53,6 +53,9 @@ $CONFIG->{log} = ($debug ? 'debug' : 'info');
 # reconfigure logging to force console output
 Dancer::Logger->init('console', $CONFIG);
 
+# for the in-memory local job queue
+schema('daemon')->deploy;
+
 # get requested action
 my $action = shift @ARGV;
 
@@ -79,18 +82,6 @@ if (not $worker->can( $action )) {
   exit (1);
 }
 
-# static configuration for the in-memory local job queue
-setting('plugins')->{DBIC}->{daemon} = {
-    dsn => 'dbi:SQLite:dbname=:memory:',
-    options => {
-        AutoCommit => 1,
-        RaiseError => 1,
-        sqlite_use_immediate_transaction => 1,
-    },
-    schema_class => 'App::Netdisco::Daemon::DB',
-};
-schema('daemon')->deploy;
-
 # what job are we asked to do?
 my $job = schema('daemon')->resultset('Admin')->new_result({
   job => 0,

Netdisco/bin/netdisco-web-fg

@@ -14,16 +14,20 @@ BEGIN {
     dir($FindBin::RealBin, 'lib')->stringify;
 }
 
+if ($ENV{_} and $ENV{_} =~ m/netdisco-web-fg$/) {
+    die "You probably want: '~/bin/localenv starman $0 --workers=1 --disable-keepalive'\n";
+}
+
 use App::Netdisco;
 use Dancer;
 debug sprintf "App::Netdisco %s", ($App::Netdisco::VERSION || 'HEAD');
 
 my $home = ($ENV{NETDISCO_HOME} || $ENV{HOME});
-set('session_dir', dir($home, 'netdisco-web-sessions'));
+set(session_dir => dir($home, 'netdisco-web-sessions')->stringify);
 
 set plack_middlewares => [
   [ Expires => (
-      content_type => [qw{ application/javascript text/css }, qr{image}],
+      content_type => [qr{^application/javascript}, qr{^text/css}, qr{image}, qr{font}],
       expires => 'access plus 1 day',
   )],
   [ Static => (
@@ -31,19 +35,17 @@ set plack_middlewares => [
       root => $ENV{DANCER_PUBLIC},
       pass_through => 1,
   )],
+  # install Dancer::Debug for this...
+  #[ Debug => (
+  #    panels => [qw/Dancer::Settings Parameters Dancer::Version DBITrace/],
+  #)],
 ];
 
 use App::Netdisco::Web;
 use Plack::Builder;
 
-my $app = sub {
-  my $env     = shift;
-  my $request = Dancer::Request->new(env => $env);
-  Dancer->dance($request);
-};
-
 my $path = (setting('path') || '/');
-builder { mount $path => $app };
+builder { mount $path => dance };
 
 =head1 NAME
 

Netdisco/lib/App/Netdisco.pm

@@ -7,10 +7,10 @@ use 5.010_000;
 use File::ShareDir 'dist_dir';
 use Path::Class;
 
-our $VERSION = '2.008002';
+our $VERSION = '2.010000';
 
 BEGIN {
-  if (not length ($ENV{DANCER_APPDIR} || '')
+  if (not ($ENV{DANCER_APPDIR} || '')
       or not -f file($ENV{DANCER_APPDIR}, 'config.yml')) {
 
       my $auto = dir(dist_dir('App-Netdisco'))->absolute;
@@ -24,6 +24,7 @@ BEGIN {
         ? $test_envdir : $auto->subdir('environments')->stringify);
 
       $ENV{DANCER_ENVIRONMENT} ||= 'deployment';
+      $ENV{PLACK_ENV} ||= $ENV{DANCER_ENVIRONMENT};
 
       $ENV{DANCER_PUBLIC} ||= $auto->subdir('public')->stringify;
       $ENV{DANCER_VIEWS}  ||= $auto->subdir('views')->stringify;
@@ -39,6 +40,41 @@ BEGIN {
   }
 }
 
+# set up database schema config from simple config vars
+use Dancer ':script';
+
+if (ref {} eq ref setting('database')) {
+    my $name = (setting('database')->{name} || 'netdisco');
+    my $host = (setting('database')->{host} || 'localhost');
+    my $user = (setting('database')->{user});
+    my $pass = (setting('database')->{pass});
+
+    # set up the netdisco schema now we have access to the config
+    # but only if it doesn't exist from an earlier config style
+    setting('plugins')->{DBIC}->{netdisco} ||= {
+        dsn => (sprintf 'dbi:Pg:dbname=%s;host=%s', $name, $host),
+        user => $user,
+        pass => $pass,
+        options => {
+            AutoCommit => 1,
+            RaiseError => 1,
+        },
+        schema_class => 'App::Netdisco::DB',
+    };
+
+}
+
+# static configuration for the in-memory local job queue
+setting('plugins')->{DBIC}->{daemon} = {
+    dsn => 'dbi:SQLite:dbname=:memory:',
+    options => {
+        AutoCommit => 1,
+        RaiseError => 1,
+        sqlite_use_immediate_transaction => 1,
+    },
+    schema_class => 'App::Netdisco::Daemon::DB',
+};
+
 =head1 NAME
 
 App::Netdisco - An open source web-based network management tool.
@@ -68,7 +104,11 @@ commands will test for the existence of them on your system:
  perl -MDBD::Pg\ 999
  perl -MSNMP\ 999
 
-With those two installed, we can proceed...
+You'll also need a compiler for some of the other Perl dependencies. For
+example on Ubuntu/Debian, install the C<build-essential> package. On
+Fedora/Red-Hat, install C<make>, C<automake>, and C<gcc>.
+
+With those installed, we can proceed...
 
 Create a user on your system called C<netdisco> if one does not already exist.
 We'll install Netdisco and its dependencies into this user's home area, which
@@ -124,7 +164,7 @@ template from this distribution:
  chmod +w ~/environments/deployment.yml
 
 Edit the file and change the database connection parameters to match those for
-your local system (that is, the C<dsn>, C<user> and C<pass>).
+your local system (that is, the C<name>, C<host>, C<user> and C<pass>).
 
 In the same file uncomment and edit the C<domain_suffix> setting to be
 appropriate for your local site. Optionally, set the C<no_auth> value to true
@@ -185,17 +225,6 @@ The main black navigation bar has a search box which is smart enough to work
 out what you're looking for in most cases. For example device names, node IP
 or MAC addreses, VLAN numbers, and so on.
 
-=head2 User Rights
-
-When user authentication is disabled (C<no_auth: true>) the default username
-is "guest", which has no special privilege. To grant port and device control
-rights to this user, create a row in the C<users> table of the Netdisco
-database with a username of C<guest> and appropriate flags set to true:
-
- netdisco=> insert into users (username) values ('guest');
- netdisco=> update users set port_control = true where username = 'guest';
- netdisco=> update users set admin = true where username = 'guest';
-
 =head2 Command-Line Device and Port Actions
 
 To run a device (discover, etc) or port control job from the command-line, use

Netdisco/lib/App/Netdisco/Core/Discover.pm

@@ -192,7 +192,7 @@ sub store_interfaces {
   foreach my $entry (keys %$interfaces) {
       my $port = $interfaces->{$entry};
 
-      if (not length $port) {
+      if (not $port) {
           debug sprintf ' [%s] interfaces - ignoring %s (no port mapping)',
             $device->ip, $port;
           next;
@@ -288,7 +288,7 @@ sub store_wireless {
       (my $iid = $entry) =~ s/\.\d+$//;
       my $port = $interfaces->{$iid};
 
-      if (not length $port) {
+      if (not $port) {
           debug sprintf ' [%s] wireless - ignoring %s (no port mapping)',
             $device->ip, $port;
           next;
@@ -316,7 +316,7 @@ sub store_wireless {
   foreach my $entry (keys %$channel) {
       my $port = $interfaces->{$entry};
 
-      if (not length $port) {
+      if (not $port) {
           debug sprintf ' [%s] wireless - ignoring %s (no port mapping)',
             $device->ip, $port;
           next;
@@ -613,7 +613,7 @@ sub store_neighbors {
       my $remote_type = $c_platform->{$entry};
       my $remote_id   = $c_id->{$entry};
 
-      next unless length $remote_ip;
+      next unless $remote_ip;
 
       # a bunch of heuristics to search known devices if we don't have a
       # useable remote IP...

Netdisco/lib/App/Netdisco/DB/ExplicitLocking.pm

@@ -39,13 +39,13 @@ sub txn_do_locked {
   }
 
   $schema->throw_exception('missing Table name to txn_do_locked()')
-    unless length $table;
+    unless $table;
 
   $table = [$table] if ref '' eq ref $table;
   my $table_fmt = join ', ', ('%s' x scalar @$table);
   my $sql = sprintf $sql_fmt, $table_fmt;
 
-  if (ref '' eq ref $mode and length $mode) {
+  if (ref '' eq ref $mode and $mode) {
       scalar grep {$_ eq $mode} values %lock_modes
         or $schema->throw_exception('bad LOCK_MODE to txn_do_locked()');
   }

Netdisco/lib/App/Netdisco/DB/ResultSet/Device.pm

@@ -196,7 +196,7 @@ sub search_by_field {
 
     # this is a bit of an inelegant trick to catch junk data entry,
     # whilst avoiding returning *all* entries in the table
-    if (length $p->{ip} and 'NetAddr::IP::Lite' ne ref $p->{ip}) {
+    if ($p->{ip} and 'NetAddr::IP::Lite' ne ref $p->{ip}) {
       $p->{ip} = ( NetAddr::IP::Lite->new($p->{ip})
         || NetAddr::IP::Lite->new('255.255.255.255') );
     }

Netdisco/lib/App/Netdisco/Daemon/Queue.pm

@@ -8,17 +8,6 @@ our @EXPORT = ();
 our @EXPORT_OK = qw/ add_jobs capacity_for take_jobs reset_jobs /;
 our %EXPORT_TAGS = ( all => \@EXPORT_OK );
 
-# static configuration for the in-memory local job queue
-setting('plugins')->{DBIC}->{daemon} = {
-    dsn => 'dbi:SQLite:dbname=:memory:',
-    options => {
-        AutoCommit => 1,
-        RaiseError => 1,
-        sqlite_use_immediate_transaction => 1,
-    },
-    schema_class => 'App::Netdisco::Daemon::DB',
-};
-
 schema('daemon')->deploy;
 my $queue = schema('daemon')->resultset('Admin');
 

Netdisco/lib/App/Netdisco/Daemon/Worker/Common.pm

@@ -50,7 +50,7 @@ sub worker_body {
       }
 
       debug "$type ($wid): sleeping now...";
-      sleep( setting('workers')->{sleep_time} || 5 );
+      sleep(1);
   }
 }
 

Netdisco/lib/App/Netdisco/Daemon/Worker/Interactive/PortActions.pm

@@ -21,7 +21,7 @@ sub set_portcontrol {
 
   my $reconfig_check = port_reconfig_check($port);
   return job_error("Cannot alter port: $reconfig_check")
-    if length $reconfig_check;
+    if $reconfig_check;
 
   return _set_port_generic($job, 'up_admin');
 }
@@ -35,11 +35,11 @@ sub set_vlan {
 
   my $port_reconfig_check = port_reconfig_check($port);
   return job_error("Cannot alter port: $port_reconfig_check")
-    if length $port_reconfig_check;
+    if $port_reconfig_check;
 
   my $vlan_reconfig_check = vlan_reconfig_check($port);
   return job_error("Cannot alter vlan: $vlan_reconfig_check")
-    if length $vlan_reconfig_check;
+    if $vlan_reconfig_check;
 
   return _set_port_generic($job, 'vlan');
 }
@@ -96,7 +96,7 @@ sub set_power {
 
   my $reconfig_check = port_reconfig_check($port);
   return job_error("Cannot alter port: $reconfig_check")
-    if length $reconfig_check;
+    if $reconfig_check;
 
 
   my $ip = $job->device;

Netdisco/lib/App/Netdisco/Daemon/Worker/Manager.pm

@@ -84,7 +84,7 @@ sub worker_body {
       # TODO also check for stale jobs in Netdisco DB
 
       debug "mgr ($wid): sleeping now...";
-      sleep( setting('workers')->{sleep_time} || 5 );
+      sleep( setting('workers')->{sleep_time} || 2 );
   }
 }
 

Netdisco/lib/App/Netdisco/Daemon/Worker/Poller/Arpnip.pm

@@ -1,81 +1,16 @@
 package App::Netdisco::Daemon::Worker::Poller::Arpnip;
 
-use Dancer qw/:moose :syntax :script/;
-use Dancer::Plugin::DBIC 'schema';
-
-use App::Netdisco::Util::SNMP 'snmp_connect';
-use App::Netdisco::Util::Device 'get_device';
 use App::Netdisco::Core::Arpnip 'do_arpnip';
-use App::Netdisco::Daemon::Util ':all';
-
-use NetAddr::IP::Lite ':lower';
 
 use Role::Tiny;
 use namespace::clean;
 
-# queue an arpnip job for all devices known to Netdisco
+with 'App::Netdisco::Daemon::Worker::Poller::Common';
-sub arpwalk {
-  my ($self, $job) = @_;
 
-  my $devices = schema('netdisco')->resultset('Device')->get_column('ip');
+sub arpnip_action { \&do_arpnip }
-  my $jobqueue = schema('netdisco')->resultset('Admin');
+sub arpnip_layer { 3 }
 
-  if ($job->subaction and $job->subaction eq 'after-discoverall') {
+sub arpwalk { (shift)->_walk_body('arpnip', @_) }
-      # make sure there are no incomplete discover jobs queued
+sub arpnip  { (shift)->_single_body('arpnip', @_) }
-      my $discover = $jobqueue->search(
-        { action => 'discover', status => { -like => 'queued%' } }
-      )->count;
-
-      return job_defer("Deferred arpwalk due to pending discover jobs")
-        if $discover;
-  }
-
-  schema('netdisco')->txn_do(sub {
-    # clean up user submitted jobs older than 1min,
-    # assuming skew between schedulers' clocks is not greater than 1min
-    $jobqueue->search({
-        action => 'arpnip',
-        status => 'queued',
-        entered => { '<' => \"(now() - interval '1 minute')" },
-    })->delete;
-
-    # is scuppered by any user job submitted in last 1min (bad), or
-    # any similar job from another scheduler (good)
-    $jobqueue->populate([
-      map {{
-          device => $_,
-          action => 'arpnip',
-          status => 'queued',
-      }} ($devices->all)
-    ]);
-  });
-
-  return job_done("Queued arpnip job for all devices");
-}
-
-sub arpnip {
-  my ($self, $job) = @_;
-
-  my $host = NetAddr::IP::Lite->new($job->device);
-  my $device = get_device($host->addr);
-
-  if ($device->in_storage
-      and $device->vendor and $device->vendor eq 'netdisco') {
-      return job_done("Skipped arpnip for pseudo-device $host");
-  }
-
-  my $snmp = snmp_connect($device);
-  if (!defined $snmp) {
-      return job_error("arpnip failed: could not SNMP connect to $host");
-  }
-
-  unless ($snmp->has_layer(3)) {
-      return job_done("Skipped arpnip for device $host without OSI layer 3 capability");
-  }
-
-  do_arpnip($device, $snmp);
-
-  return job_done("Ended arpnip for ". $host->addr);
-}
 
 1;

Netdisco/lib/App/Netdisco/Daemon/Worker/Poller/Common.pm

@@ -0,0 +1,86 @@
+package App::Netdisco::Daemon::Worker::Poller::Common;
+
+use Dancer qw/:moose :syntax :script/;
+use Dancer::Plugin::DBIC 'schema';
+
+use App::Netdisco::Util::SNMP 'snmp_connect';
+use App::Netdisco::Util::Device 'get_device';
+use App::Netdisco::Daemon::Util ':all';
+
+use NetAddr::IP::Lite ':lower';
+
+use Role::Tiny;
+use namespace::clean;
+
+# queue a job for all devices known to Netdisco
+sub _walk_body {
+  my ($self, $job_type, $job) = @_;
+
+  my $devices = schema('netdisco')->resultset('Device')->get_column('ip');
+  my $jobqueue = schema('netdisco')->resultset('Admin');
+
+  if ($job->subaction and $job->subaction eq 'after-discoverall') {
+      # make sure there are no incomplete discover jobs queued
+      my $discover = $jobqueue->search(
+        { action => 'discover', status => { -like => 'queued%' } }
+      )->count;
+
+      return job_defer("Deferred $job_type due to pending discover jobs")
+        if $discover;
+  }
+
+  schema('netdisco')->txn_do(sub {
+    # clean up user submitted jobs older than 1min,
+    # assuming skew between schedulers' clocks is not greater than 1min
+    $jobqueue->search({
+        action => $job_type,
+        status => 'queued',
+        entered => { '<' => \"(now() - interval '1 minute')" },
+    })->delete;
+
+    # is scuppered by any user job submitted in last 1min (bad), or
+    # any similar job from another scheduler (good)
+    $jobqueue->populate([
+      map {{
+          device => $_,
+          action => $job_type,
+          status => 'queued',
+      }} ($devices->all)
+    ]);
+  });
+
+  return job_done("Queued $job_type job for all devices");
+}
+
+sub _single_body {
+  my ($self, $job_type, $job) = @_;
+
+  my $action_method = $job_type .'_action';
+  my $job_action = $self->$action_method;
+
+  my $layer_method = $job_type .'_layer';
+  my $job_layer = $self->$layer_method;
+
+  my $host = NetAddr::IP::Lite->new($job->device);
+  my $device = get_device($host->addr);
+
+  if ($device->in_storage
+      and $device->vendor and $device->vendor eq 'netdisco') {
+      return job_done("Skipped $job_type for pseudo-device $host");
+  }
+
+  my $snmp = snmp_connect($device);
+  if (!defined $snmp) {
+      return job_error("$job_type failed: could not SNMP connect to $host");
+  }
+
+  unless ($snmp->has_layer( $job_layer )) {
+      return job_done("Skipped $job_type for device $host without OSI layer $job_layer capability");
+  }
+
+  $job_action->($device, $snmp);
+
+  return job_done("Ended $job_type for ". $host->addr);
+}
+
+1;

Netdisco/lib/App/Netdisco/Daemon/Worker/Poller/Macsuck.pm

@@ -1,71 +1,16 @@
 package App::Netdisco::Daemon::Worker::Poller::Macsuck;
 
-use Dancer qw/:moose :syntax :script/;
+use App::Netdisco::Core::Macsuck 'do_macsuck';
-use Dancer::Plugin::DBIC 'schema';
-
-use App::Netdisco::Util::SNMP 'snmp_connect';
-use App::Netdisco::Util::Device 'get_device';
-use App::Netdisco::Core::Macsuck ':all';
-use App::Netdisco::Daemon::Util ':all';
-
-use NetAddr::IP::Lite ':lower';
 
 use Role::Tiny;
 use namespace::clean;
 
-# queue a macsuck job for all devices known to Netdisco
+with 'App::Netdisco::Daemon::Worker::Poller::Common';
-sub macwalk {
-  my ($self, $job) = @_;
 
-  my $devices = schema('netdisco')->resultset('Device')->get_column('ip');
+sub macsuck_action { \&do_macsuck }
-  my $jobqueue = schema('netdisco')->resultset('Admin');
+sub macsuck_layer { 2 }
 
-  schema('netdisco')->txn_do(sub {
+sub macwalk { (shift)->_walk_body('macsuck', @_) }
-    # clean up user submitted jobs older than 1min,
+sub macsuck { (shift)->_single_body('macsuck', @_) }
-    # assuming skew between schedulers' clocks is not greater than 1min
-    $jobqueue->search({
-        action => 'macsuck',
-        status => 'queued',
-        entered => { '<' => \"(now() - interval '1 minute')" },
-    })->delete;
-
-    # is scuppered by any user job submitted in last 1min (bad), or
-    # any similar job from another scheduler (good)
-    $jobqueue->populate([
-      map {{
-          device => $_,
-          action => 'macsuck',
-          status => 'queued',
-      }} ($devices->all)
-    ]);
-  });
-
-  return job_done("Queued macsuck job for all devices");
-}
-
-sub macsuck {
-  my ($self, $job) = @_;
-
-  my $host = NetAddr::IP::Lite->new($job->device);
-  my $device = get_device($host->addr);
-
-  if ($device->in_storage
-      and $device->vendor and $device->vendor eq 'netdisco') {
-      return job_done("Skipped macsuck for pseudo-device $host");
-  }
-
-  my $snmp = snmp_connect($device);
-  if (!defined $snmp) {
-      return job_error("macsuck failed: could not SNMP connect to $host");
-  }
-
-  unless ($snmp->has_layer(2)) {
-      return job_done("Skipped macsuck for device $host without OSI layer 2 capability");
-  }
-
-  do_macsuck($device, $snmp);
-
-  return job_done("Ended macsuck for ". $host->addr);
-}
 
 1;

Netdisco/lib/App/Netdisco/Manual/Configuration.pod

@@ -54,7 +54,7 @@ colon character)
 
 If you followed the installation instructions, then you should have set the
 database connection parameters to match those of your local system. That is,
-the C<dsn> (DB name, host, port), C<user> and C<pass>.
+the database C<name>, C<host>, C<user> and C<pass>.
 
 =head2 General Settings
 
@@ -95,14 +95,26 @@ database:
  netdisco=> update users set port_control = true where username = 'guest';
  netdisco=> update users set admin = true where username = 'guest';
 
-=head3 C<port>
+=head3 C<trust_remote_user>
 
-Value: Number. Default: C<5000>.
+Value: Boolean. Default: C<false>.
 
-Port which the web server listens on. Netdisco comes with a good pre-forking
+Enable this if Netdisco is running within another web server such as Apache,
-web server, so you can change this to C<80> if you want to use it directly.
+and you want that server to handle user authentication. Normally the
-However the default is designed to work well with servers such as Apache in
+authenticated username will automatically be set in the C<REMOTE_USER> HTTP
-reverse-proxy mode.
+Header. See L<Dancer::Deployment/Running from Apache> for further details.
+
+=head3 C<trust_x_remote_user>
+
+Value: Boolean. Default: C<false>.
+
+Enable this if you proxy requests to Netdisco via another web server such as
+Apache, and you want that server to handle user authentication. You need to
+configure the authorized username to be passed in the C<X-REMOTE_USER> HTTP
+Header. For example with Apache:
+
+ RequestHeader unset X-REMOTE_USER
+ RequestHeader set X-REMOTE_USER "%{REMOTE_USER}e" env=REMOTE_USER
 
 =head3 C<path>
 
@@ -112,14 +124,6 @@ Mount point for the Netdisco web frontend. This is usually the root of the web
 server. Set this to the path under which all pages live, e.g. C</netdisco2>.
 As an alternative you can use the C<--path> option to C<netdisco-web>.
 
-=head3 C<behind_proxy>
-
-Value: Boolean. Default: C<false>.
-
-A hint to the Netdisco web frontend that it's running behind a reverse proxy.
-In that case, Netdisco will pay attention to the C<X_FORWARDED_PROTOCOL>,
-C<X_FORWARDED_HOST>, etc settings in HTML headers.
-
 =head3 C<web_plugins>
 
 Value: List of Modules. Default: List of bundled L<App::Netdisco::Web::Plugin> names.
@@ -348,7 +352,7 @@ Value: Settings Tree. Default:
  workers:
    interactives: 2
    pollers: 2
-   sleep_time: 5
+   sleep_time: 2
 
 Control the activity of the backend daemon with this configuration setting.
 

Netdisco/lib/App/Netdisco/Manual/Deployment.pod

@@ -14,52 +14,72 @@ Obviously, you'll need to substitute this wherever you see "C<~>" in the
 installation instructions. The Netdisco application will use this setting
 itself to locate files and configuration.
 
+=head1 Pass Options to the Web Frontend Daemon
+
+Simply add any options after the "C<start>" command. See other sections of
+this document for some examples.
+
 =head1 Non-root Hosting
 
 Netdisco will assume its web site is hosted at the apex of your server - that
 is, the document root. To relocate the web application, pass the C<--path>
 parameter to the web startup script:
 
- ~/bin/netdisco-web --path /netdisco2
+ ~/bin/netdisco-web start --path=/netdisco2
 
 Alternatively, can set the C<path> configuration option in your
 C<deployment.yml> file:
 
  path: '/netdisco2'
 
+=head1 Listening Port for the Web Frontend
+
+Pass the C<--port> parameter to any of the web scripts. For example:
+
+ ~/bin/netdisco-web start --port=8080
+
+=head1 Listening Address for the Web Frontend
+
+Pass the C<--listen> parameter to any of the web scripts. Note that you always
+must include the port number. For example:
+
+ ~/bin/netdisco-web start --listen=127.0.0.1:8080
+
 =head1 Behind a Proxy
 
 By default the web application daemon starts listening on port 5000 and goes 
 into the background. This is ideal for hosting behind a web proxy (e.g. Apache
 with C<mod_proxy>).
 
-After enabling the C<proxy> and C<proxy_http> modules in Apache, a suitable
+After enabling the C<headers>, C<proxy> and C<proxy_http> modules in Apache, a
-configuration would be:
+suitable configuration would be:
 
+ ProxyPreserveHost On
  ProxyPass / http://localhost:5000/
  ProxyPassReverse / http://localhost:5000/
  
+ ProxyRequests Off
  <Proxy *>
    Order allow,deny
    Allow from all
  </Proxy>
 
-You also need to set the following configuration in your C<deployment.yml>
-file:
-
- behind_proxy: 1
-
 To combine this with Non-root Hosting as above, simply change the paths
-referenced in the configuration like so (and use Non-root Hosting as above):
+referenced in the configuration, and set C<path> in your C<deployment.yml> as
+discussed above. Note there is no trailing slash in the Apache config:
 
- ProxyPass /netdisco2 http://localhost:5000/
+ ProxyPass /netdisco2 http://localhost:5000/netdisco2
- ProxyPassReverse /netdisco2 http://localhost:5000/
+ ProxyPassReverse /netdisco2 http://localhost:5000/netdisco2
+
+To delegate user authentication to Apache, use the C<trust_remote_user> or
+C<trust_x_remote_user> settings. See L<App::Netdisco::Manual::Configuration>
+for more details.
 
 =head1 SQL and HTTP Trace
 
 For SQL debugging try the following commands:
 
- DBIC_TRACE_PROFILE=console DBIC_TRACE=1 ~/bin/localenv starman --workers=1 ~/bin/netdisco-web-fg
+ DBIC_TRACE_PROFILE=console DBIC_TRACE=1 ~/bin/localenv starman ~/bin/netdisco-web-fg --workers=1 --disable-keepalive
  DBIC_TRACE_PROFILE=console DBIC_TRACE=1 ~/bin/localenv ~/bin/netdisco-daemon-fg
 
 =head1 Further Reading...

Netdisco/lib/App/Netdisco/Manual/ReleaseNotes.pod

@@ -8,6 +8,23 @@ This document will list only the most significant changes with each release of
 Netdisco. You are B<STRONGLY> recommended to read this document each time you
 install and upgrade.
 
+=head1 2.010000
+
+=head2 General Changes
+
+You can now simplify database configuration to just the following, instead of
+the more verbose C<plugins/DBIC> setting which was there before:
+
+ database:
+   name: 'netdisco'
+   host: 'localhost'
+   user: 'someuser'
+   pass: 'somepass'
+
+Also, the C<REMOTE_USER> and C<X-REMOTE_USER> environment variables are now
+supported for delegating authentication to another web server. See the
+Deployment and Configuration documentation for further details.
+
 =head1 2.008000
 
 =head2 Heath Advice
@@ -16,7 +33,7 @@ This release contains the first version of our new poller, which handles
 device and node discovery. Please make sure to backup any existing Netdisco
 database before trying it out.
 
-=head2 Other Changes
+=head2 General Changes
 
 You can remove any settings from C<~/environments/deployment.yml> which you
 didn't edit or add to the file yourself. All defaults are now properly
@@ -35,7 +52,7 @@ Please B<rename or copy> your environment file:
 
  mv ~/environments/development.yml ~/environments/deployment.yml
 
-=head2 Other Changes
+=head2 General Changes
 
 The installation is now relocateable outside of a user's home directory by
 setting the C<NETDISCO_HOME> environment variable. This defaults to your own

Netdisco/lib/App/Netdisco/Util/Device.pm

@@ -10,6 +10,8 @@ our @EXPORT = ();
 our @EXPORT_OK = qw/
   get_device
   is_discoverable
+  is_arpnipable
+  is_macsuckable
 /;
 our %EXPORT_TAGS = (all => \@EXPORT_OK);
 
@@ -54,7 +56,7 @@ sub get_device {
     ->find_or_new({ip => $ip});
 }
 
-=head2 is_discoverable( $ip )
+=head2 is_discoverable( $ip, $device_type? )
 
 Given an IP address, returns C<true> if Netdisco on this host is permitted by
 the local configuration to discover the device.
@@ -62,6 +64,9 @@ the local configuration to discover the device.
 The configuration items C<discover_no> and C<discover_only> are checked
 against the given IP.
 
+If C<$device_type> is also given, then C<discover_no_type> will also be
+checked.
+
 Returns false if the host is not permitted to discover the target device.
 
 =cut
@@ -99,4 +104,82 @@ sub is_discoverable {
   return 1;
 }
 
+=head2 is_arpnipable( $ip )
+
+Given an IP address, returns C<true> if Netdisco on this host is permitted by
+the local configuration to arpnip the device.
+
+The configuration items C<arpnip_no> and C<arpnip_only> are checked
+against the given IP.
+
+Returns false if the host is not permitted to arpnip the target device.
+
+=cut
+
+sub is_arpnipable {
+  my $ip = shift;
+  my $device = get_device($ip) or return 0;
+
+  my $addr = NetAddr::IP::Lite->new($device->ip);
+  my $arpnip_no   = setting('arpnip_no') || [];
+  my $arpnip_only = setting('arpnip_only') || [];
+
+  if (scalar @$arpnip_no) {
+      foreach my $item (@$arpnip_no) {
+          my $ip = NetAddr::IP::Lite->new($item) or return 0;
+          return 0 if $ip->contains($addr);
+      }
+  }
+
+  if (scalar @$arpnip_only) {
+      my $okay = 0;
+      foreach my $item (@$arpnip_only) {
+          my $ip = NetAddr::IP::Lite->new($item) or return 0;
+          ++$okay if $ip->contains($addr);
+      }
+      return 0 if not $okay;
+  }
+
+  return 1;
+}
+
+=head2 is_macsuckable( $ip )
+
+Given an IP address, returns C<true> if Netdisco on this host is permitted by
+the local configuration to macsuck the device.
+
+The configuration items C<macsuck_no> and C<macsuck_only> are checked
+against the given IP.
+
+Returns false if the host is not permitted to macsuck the target device.
+
+=cut
+
+sub is_macsuckable {
+  my $ip = shift;
+  my $device = get_device($ip) or return 0;
+
+  my $addr = NetAddr::IP::Lite->new($device->ip);
+  my $macsuck_no   = setting('macsuck_no') || [];
+  my $macsuck_only = setting('macsuck_only') || [];
+
+  if (scalar @$macsuck_no) {
+      foreach my $item (@$macsuck_no) {
+          my $ip = NetAddr::IP::Lite->new($item) or return 0;
+          return 0 if $ip->contains($addr);
+      }
+  }
+
+  if (scalar @$macsuck_only) {
+      my $okay = 0;
+      foreach my $item (@$macsuck_only) {
+          my $ip = NetAddr::IP::Lite->new($item) or return 0;
+          ++$okay if $ip->contains($addr);
+      }
+      return 0 if not $okay;
+  }
+
+  return 1;
+}
+
 1;

Netdisco/lib/App/Netdisco/Util/SNMP.pm

@@ -93,13 +93,13 @@ sub _snmp_connect_generic {
 
   my $info = undef;
   VERSION: foreach my $ver (@versions) {
-      next unless length $ver;
+      next unless $ver;
 
       CLASS: foreach my $class (@classes) {
-          next unless length $class;
+          next unless $class;
 
           COMMUNITY: foreach my $comm (@communities) {
-              next unless length $comm;
+              next unless $comm;
 
               $info = _try_connect($ver, $class, $comm, \%snmp_args)
                 and last VERSION;

Netdisco/lib/App/Netdisco/Web.pm

@@ -43,6 +43,9 @@ if (setting('extra_web_plugins') and ref [] eq ref setting('extra_web_plugins'))
     _load_web_plugins( setting('extra_web_plugins') );
 }
 
+# workaround for https://github.com/PerlDancer/Dancer/issues/935
+hook after_error_render => sub { setting('layout' => 'main') };
+
 hook 'before_template' => sub {
     my $tokens = shift;
 
@@ -51,7 +54,7 @@ hook 'before_template' => sub {
         if request->base->path ne '/';
 
     # allow portable dynamic content
-    $tokens->{uri_for} = sub { uri_for(@_)->path_query() };
+    $tokens->{uri_for} = sub { uri_for(@_)->path_query };
 
     # allow very long lists of ports
     $Template::Directive::WHILE_MAX = 10_000;

Netdisco/lib/App/Netdisco/Web/AdminTask.pm

@@ -74,7 +74,7 @@ foreach my $jobtype (keys %jobs_all, keys %jobs) {
           if exists $jobs{$jobtype} and not param('device');
 
         add_job($jobtype, param('device'));
-        redirect uri_for('/admin/jobqueue')->path_query;
+        redirect uri_for('/admin/jobqueue')->as_string;
     };
 }
 
@@ -82,7 +82,7 @@ get '/admin/*' => sub {
     my ($tag) = splat;
 
     if (! eval { var('user')->admin }) {
-        return redirect uri_for('/')->path_query;
+        return redirect uri_for('/')->as_string;
     }
 
     # trick the ajax into working as if this were a tabbed page

Netdisco/lib/App/Netdisco/Web/AuthN.pm

@@ -7,10 +7,17 @@ use Digest::MD5 ();
 
 hook 'before' => sub {
     if (! session('user') && request->path ne uri_for('/login')->path) {
-        if (setting('no_auth')) {
+        if (setting('trust_x_remote_user') and scalar request->header('X-REMOTE_USER')) {
+            session(user => scalar request->header('X-REMOTE_USER'));
+        }
+        elsif (setting('trust_remote_user') and scalar request->header('REMOTE_USER')) {
+            session(user => scalar request->header('REMOTE_USER'));
+        }
+        elsif (setting('no_auth')) {
             session(user => 'guest');
         }
         else {
+            # user has no AuthN - force to handler for '/'
             request->path_info('/');
         }
     }
@@ -27,23 +34,24 @@ hook 'before' => sub {
 
 post '/login' => sub {
     if (param('username') and param('password')) {
-        my $user = schema('netdisco')->resultset('User')->find(param('username'));
+        my $user = schema('netdisco')->resultset('User')
+                                     ->find(param('username'));
 
         if ($user) {
             my $sum = Digest::MD5::md5_hex(param('password'));
             if (($sum and $user->password) and ($sum eq $user->password)) {
                 session(user => $user->username);
-                return redirect uri_for('/inventory')->path_query;
+                return redirect uri_for('/inventory')->as_string;
             }
         }
     }
 
-    redirect uri_for('/', {failed => 1})->path_query;
+    redirect uri_for('/', {failed => 1})->as_string;
 };
 
 get '/logout' => sub {
     session->destroy;
-    redirect uri_for('/', {logout => 1})->path_query;
+    redirect uri_for('/', {logout => 1})->as_string;
 };
 
 true;

Netdisco/lib/App/Netdisco/Web/Device.pm

@@ -116,7 +116,7 @@ get '/device' => sub {
     });
 
     if (!defined $dev) {
-        return redirect uri_for('/', {nosuchdevice => 1})->path_query;
+        return redirect uri_for('/', {nosuchdevice => 1})->as_string();
     }
 
     params->{'tab'} ||= 'details';

Netdisco/lib/App/Netdisco/Web/Plugin.pm

@@ -22,7 +22,7 @@ config->{engines}->{template_toolkit}->{INCLUDE_PATH} ||= [ setting('views') ];
 register 'register_template_path' => sub {
   my ($self, $path) = plugin_args(@_);
 
-  if (!length $path) {
+  if (!$path) {
       return error "bad template path to register_template_paths";
   }
 
@@ -34,11 +34,11 @@ register 'register_template_path' => sub {
 sub _register_include {
   my ($type, $plugin) = @_;
 
-  if (!length $type) {
+  if (!$type) {
       return error "bad type to _register_include";
   }
 
-  if (!length $plugin) {
+  if (!$plugin) {
       return error "bad plugin name to register_$type";
   }
 
@@ -60,7 +60,7 @@ register 'register_device_port_column' => sub {
   $config->{default} ||= '';
   $config->{position} ||= 'right';
 
-  if (!length $config->{name} or !length $config->{label}) {
+  if (!$config->{name} or !$config->{label}) {
       return error "bad config to register_device_port_column";
   }
 
@@ -77,9 +77,9 @@ register 'register_device_port_column' => sub {
 register 'register_navbar_item' => sub {
   my ($self, $config) = plugin_args(@_);
 
-  if (!length $config->{tag}
+  if (!$config->{tag}
-      or !length $config->{path}
+      or !$config->{path}
-      or !length $config->{label}) {
+      or !$config->{label}) {
 
       return error "bad config to register_navbar_item";
   }
@@ -97,8 +97,8 @@ register 'register_navbar_item' => sub {
 register 'register_admin_task' => sub {
   my ($self, $config) = plugin_args(@_);
 
-  if (!length $config->{tag}
+  if (!$config->{tag}
-      or !length $config->{label}) {
+      or !$config->{label}) {
 
       return error "bad config to register_admin_task";
   }
@@ -110,8 +110,8 @@ sub _register_tab {
   my ($nav, $config) = @_;
   my $stash = setting("_${nav}_tabs");
 
-  if (!length $config->{tag}
+  if (!$config->{tag}
-      or !length $config->{label}) {
+      or !$config->{label}) {
 
       return error "bad config to register_${nav}_item";
   }
@@ -140,9 +140,9 @@ register 'register_report' => sub {
   my ($self, $config) = plugin_args(@_);
   my @categories = @{ setting('_report_order') };
 
-  if (!length $config->{category}
+  if (!$config->{category}
-      or !length $config->{tag}
+      or !$config->{tag}
-      or !length $config->{label}
+      or !$config->{label}
       or 0 == scalar grep {$config->{category} eq $_} @categories) {
 
       return error "bad config to register_report";

Netdisco/lib/App/Netdisco/Web/Plugin/AdminTask/JobQueue.pm

@@ -13,7 +13,7 @@ register_admin_task({
 
 ajax '/ajax/control/admin/jobqueue/del' => sub {
     send_error('Forbidden', 403) unless var('user')->admin;
-    send_error('Missing job', 400) unless length param('job');
+    send_error('Missing job', 400) unless param('job');
 
     schema('netdisco')->txn_do(sub {
       my $device = schema('netdisco')->resultset('Admin')
@@ -26,7 +26,10 @@ ajax '/ajax/content/admin/jobqueue' => sub {
 
     my $set = schema('netdisco')->resultset('Admin')
       ->with_times
-      ->search({}, {order_by => { -desc => [qw/entered device action/] }});
+      ->search({}, {
+        order_by => { -desc => [qw/entered device action/] },
+        rows => 200,
+      });
 
     content_type('text/html');
     template 'ajax/admintask/jobqueue.tt', {

Netdisco/lib/App/Netdisco/Web/Plugin/AdminTask/PseudoDevice.pm

@@ -15,14 +15,14 @@ register_admin_task({
 sub _sanity_ok {
     return 0 unless var('user') and var('user')->admin;
 
-    return 0 unless length param('dns')
+    return 0 unless param('dns')
       and param('dns') =~ m/^[[:print:]]+$/
       and param('dns') !~ m/[[:space:]]/;
 
     my $ip = NetAddr::IP::Lite->new(param('ip'));
     return 0 unless ($ip and$ip->addr ne '0.0.0.0');
 
-    return 0 unless length param('ports')
+    return 0 unless param('ports')
       and param('ports') =~ m/^[[:digit:]]+$/;
 
     return 1;

Netdisco/lib/App/Netdisco/Web/Plugin/AdminTask/Topology.pm

@@ -21,8 +21,8 @@ sub _sanity_ok {
     my $dev2 = NetAddr::IP::Lite->new(param('dev2'));
     return 0 unless ($dev2 and $dev2->addr ne '0.0.0.0');
 
-    return 0 unless length param('port1');
+    return 0 unless param('port1');
-    return 0 unless length param('port2');
+    return 0 unless param('port2');
 
     return 1;
 }

Netdisco/lib/App/Netdisco/Web/Plugin/Device/Ports.pm

@@ -57,8 +57,15 @@ ajax '/ajax/content/device/ports' => sub {
     # get number of vlans on the port to control whether to list them or not
     $set = $set->with_vlan_count if param('c_vmember');
 
+    # run single collapsed query for all relations, but only if we're not
+    # also fetching archived data (tests show it's better this way)
+    $set = $set->search_rs({}, { prefetch => [{ port_vlans_tagged => 'vlan'}] })
+      if param('c_vmember') and not (param('c_nodes') and param('n_archived'));
+
     # what kind of nodes are we interested in?
     my $nodes_name = (param('n_archived') ? 'nodes' : 'active_nodes');
+    $set = $set->search_rs({}, { order_by => ["${nodes_name}.mac", "ips.ip"] })
+      if param('c_nodes');
     $nodes_name .= '_with_age' if param('c_nodes') and param('n_age');
 
     # retrieve active/all connected nodes, if asked for

Netdisco/lib/App/Netdisco/Web/Plugin/Search/Node.pm

@@ -60,6 +60,7 @@ ajax '/ajax/content/search/node' => sub {
               ->search_by_dns({dns => $node, @active});
         }
         return unless $set and $set->count;
+        $set = $set->search_rs({}, { order_by => 'me.mac' });
 
         template 'ajax/search/node_by_ip.tt', {
           macs => $set,

Netdisco/lib/App/Netdisco/Web/Search.pm

@@ -65,7 +65,7 @@ get '/search' => sub {
 
     if (not param('tab')) {
         if (not $q) {
-            return redirect uri_for('/')->path_query;
+            return redirect uri_for('/')->as_string;
         }
 
         # pick most likely tab for initial results
@@ -82,7 +82,7 @@ get '/search' => sub {
                       tab => 'details',
                       q => ($nd->first->dns || $nd->first->ip),
                       f => '',
-                    })->path_query;
+                    })->as_string;
                 }
 
                 # multiple devices

Netdisco/lib/App/Netdisco/Web/TypeAhead.pm

@@ -35,7 +35,7 @@ ajax '/ajax/data/deviceip/typeahead' => sub {
 ajax '/ajax/data/port/typeahead' => sub {
     my $dev  = param('dev1')  || param('dev2');
     my $port = param('port1') || param('port2');
-    send_error('Missing device', 400) unless length $dev;
+    send_error('Missing device', 400) unless $dev;
 
     my $device = schema('netdisco')->resultset('Device')
       ->find({ip => $dev});
@@ -43,7 +43,7 @@ ajax '/ajax/data/port/typeahead' => sub {
 
     my $set = $device->ports({},{order_by => 'port'});
     $set = $set->search({port => { -ilike => "\%$port\%" }})
-      if length $port;
+      if $port;
 
     my $results = [ sort { &App::Netdisco::Util::Web::sort_port($a->port, $b->port) } $set->all ];
 

Netdisco/share/config.yml

@@ -19,7 +19,6 @@ logger_format: '[%P] %L @%D> %m'
 
 domain_suffix: ''
 no_auth: false
-port: 5000
 path: '/'
 behind_proxy: false
 web_plugins:
@@ -88,7 +87,7 @@ no_port_control: false
 workers:
   interactives: 2
   pollers: 2
-  sleep_time: 5
+  sleep_time: 2
 
 #housekeeping:
 #  discoverall:

Netdisco/share/environments/deployment.yml

@@ -8,17 +8,11 @@
 # ESSENTIAL SETTINGS
 # ------------------
 
-plugins:
+database:
-  DBIC:
+  name: 'netdisco'
-    # alter dsn/user/pass for your local Netdisco DB
+  host: 'localhost'
-    netdisco:
+  user: 'changeme'
-      schema_class: 'App::Netdisco::DB'
+  pass: 'changeme'
-      dsn: 'dbi:Pg:dbname=netdisco;host=localhost'
-      user: 'changeme'
-      pass: 'changeme'
-      options:
-        RaiseError: 1
-        AutoCommit: 1
 
 # --------------------
 # RECOMMENDED SETTINGS

Netdisco/share/public/css/netdisco.css

@@ -35,6 +35,41 @@ body {
   overflow-x: hidden;
 }
 
+/* fake looks for form submit buttons embedded in bootstrap dropdowns */
+.nd_btn-link {
+  display: block;
+  padding: 2px 20px;
+  clear: both;
+  font-weight: normal;
+  line-height: 18px;
+  color: #333333;
+  white-space: nowrap;
+  text-decoration: none;
+  margin-top: 0px !important;
+  width: 100%;
+  text-align: left;
+  margin-left: -1px;
+}
+
+.nd_btn-link:hover, .nd_btn-link:focus {
+  text-decoration: none;
+  color: #ffffff;
+  background-color: #0081c2;
+  background-repeat: repeat-x;
+  text-shadow: none;
+}
+
+/* to be added to qtip-bootstrap class */
+.nd_qtip-unconstrained {
+  min-width: none;
+  max-width: none;
+}
+
+.qtip-content {
+  padding-bottom: 0px;
+  line-height: 8px;
+}
+
 /* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ */
 /* styles to adjust the hero box used for homepage + login */
 
@@ -140,6 +175,11 @@ td > form.nd_inline-form {
   color: #6D5720;
 }
 
+/* for the job control admin page play/pause links */
+#nd_countdown-refresh:hover, #nd_countdown-control:hover {
+  text-decoration: none;
+}
+
 /* when there's only one tab (report, task etc) change the text color */
 .nd_single-tab {
   color: rgb(187,112,0) !important;

Netdisco/share/public/javascripts/netdisco.js

@@ -173,7 +173,7 @@ $(document).ready(function() {
   // activate typeahead on the main search box, for device names only
   $('#nq').typeahead({
     source: function (query, process) {
-      return $.get('/ajax/data/devicename/typeahead', { query: query }, function (data) {
+      return $.get( uri_base + '/ajax/data/devicename/typeahead', { query: query }, function (data) {
         return process(data);
       });
     }

Netdisco/share/views/admintask.tt

@@ -29,7 +29,12 @@
       <li class="active"><a id="[% task.tag %]_link" class="nd_single-tab"
         href="#[% task.tag %]_pane">[% task.label %]</a></li>
       [% IF task.tag == 'jobqueue' %]
-      <span id="nd_device-name"></span>
+      <span id="nd_device-name">
+        <a id="nd_countdown-refresh" href="#"><i class="text-success icon-refresh"></i></a>
+        <a id="nd_countdown-control" href="#">
+          <i id="nd_countdown-control-icon" class="text-error icon-pause"></i></a>
+        <span id="nd_countdown"></span>
+      </span>
       [% END %]
     </ul>
     <div class="tab-content">

Netdisco/share/views/ajax/admintask/jobqueue.tt

@@ -16,9 +16,10 @@
   </tbody>
     [% WHILE (row = results.next) %]
     <tr
-      [% ' class="success"' IF row.status == 'done' %]
+      [% ' class="nd_jobqueueitem success"' IF row.status == 'done' %]
-      [% ' class="error"'   IF row.status == 'error' %]
+      [% ' class="nd_jobqueueitem error"'   IF row.status == 'error' %]
-      [% ' class="info"'    IF row.status.search('^queued-') %]
+      [% ' class="nd_jobqueueitem info"'    IF row.status.search('^queued-') %]
+      data-content="<pre>[% row.log | html_entity %]</pre>"
     >
       <td class="nd_center-cell">[% row.entered_stamp | html_entity %]</td>
       <td class="nd_center-cell">

Netdisco/share/views/ajax/device/ports.tt

@@ -142,9 +142,11 @@
       <td>
         [% IF row.tagged_vlans_count %]
         [% SET output = '' %]
-        [% FOREACH vlan IN row.tagged_vlans %]
+        [% SET vlanlist = [] %]
+        [% FOREACH vlan IN row.tagged_vlans %][% vlanlist.push(vlan.vlan) %][% END %]
+        [% FOREACH vlan IN vlanlist.nsort %]
           [% SET output = output _
-            '<a href="' _ uri_for('/search') _ '?tab=vlan&q=' _ vlan.vlan _ '">' _ vlan.vlan _ '</a>' %]
+            '<a href="' _ uri_for('/search') _ '?tab=vlan&q=' _ vlan _ '">' _ vlan _ '</a>' %]
           [% SET output = output _ ', ' IF NOT loop.last %]
         [% END %]
         [% IF row.tagged_vlans_count > 10 %] [%# TODO make this a settable variable %]

Netdisco/share/views/index.tt

@@ -23,7 +23,7 @@
       <div class="alert fade in">
         <a class="close" data-dismiss="alert">×</a>
         Sorry, page not found.
-          <a href="http://sourceforge.net/tracker/?group_id=80033&atid=558508" target="_blank">Report a Bug?</a>
+          <a href="http://sourceforge.net/p/netdisco/bugs/" target="_blank">Report a Bug?</a>
       </div>
       [% END %]
       <div class="hero-unit">

Netdisco/share/views/js/admintask.js

@@ -11,12 +11,14 @@
   function inner_view_processing(tab) {
 
     // reload this table every 5 seconds
-    if (tab == 'jobqueue') {
+    if (tab == 'jobqueue'
-        $('#nd_device-name').text('5');
+        && $('#nd_countdown-control-icon').hasClass('icon-pause')) {
-        nd_timers.push(setTimeout(function() { $('#nd_device-name').text('4') }, 1000 ));
+
-        nd_timers.push(setTimeout(function() { $('#nd_device-name').text('3') }, 2000 ));
+        $('#nd_countdown').text('5');
-        nd_timers.push(setTimeout(function() { $('#nd_device-name').text('2') }, 3000 ));
+        nd_timers.push(setTimeout(function() { $('#nd_countdown').text('4') }, 1000 ));
-        nd_timers.push(setTimeout(function() { $('#nd_device-name').text('1') }, 4000 ));
+        nd_timers.push(setTimeout(function() { $('#nd_countdown').text('3') }, 2000 ));
+        nd_timers.push(setTimeout(function() { $('#nd_countdown').text('2') }, 3000 ));
+        nd_timers.push(setTimeout(function() { $('#nd_countdown').text('1') }, 4000 ));
         nd_timers.push(setTimeout(function() {
           // clear any running timers
           for (var i = 0; i < nd_timers.length; i++) {
@@ -29,7 +31,7 @@
 
     // activate typeahead on the topo boxes
     $('.nd_topo_dev').autocomplete({
-      source: '/ajax/data/deviceip/typeahead'
+      source: uri_base + '/ajax/data/deviceip/typeahead'
       ,delay: 150
       ,minLength: 0
     });
@@ -38,7 +40,7 @@
     $('.nd_topo_port.nd_topo_dev1').autocomplete({
       source: function (request, response)  {
         var query = $('.nd_topo_dev1').serialize();
-        return $.get('/ajax/data/port/typeahead', query, function (data) {
+        return $.get( uri_base + '/ajax/data/port/typeahead', query, function (data) {
           return response(data);
         });
       }
@@ -49,7 +51,7 @@
     $('.nd_topo_port.nd_topo_dev2').autocomplete({
       source: function (request, response)  {
         var query = $('.nd_topo_dev2').serialize();
-        return $.get('/ajax/data/port/typeahead', query, function (data) {
+        return $.get( uri_base + '/ajax/data/port/typeahead', query, function (data) {
           return response(data);
         });
       }
@@ -77,6 +79,31 @@
       $(this).siblings('.nd_topo_port').autocomplete('search');
     });
 
+    // job control refresh icon should reload the page
+    $('#nd_countdown-refresh').click(function() {
+      event.preventDefault();
+      for (var i = 0; i < nd_timers.length; i++) {
+          clearTimeout(nd_timers[i]);
+      }
+      $('#' + tab + '_form').trigger('submit');
+    });
+
+    // job control pause/play icon switcheroo
+    $('#nd_countdown-control').click(function() {
+      event.preventDefault();
+      var icon = $('#nd_countdown-control-icon');
+      icon.toggleClass('icon-pause icon-play text-error text-success');
+
+      if (icon.hasClass('icon-play')) {
+        for (var i = 0; i < nd_timers.length; i++) {
+            clearTimeout(nd_timers[i]);
+        }
+        $('#nd_countdown').text('0');
+      }
+      else {
+        $('#' + tab + '_form').trigger('submit');
+      }
+    });
 
     // activity for admin task tables
     // dynamically bind to all forms in the table
@@ -101,15 +128,38 @@
             '<div class="span2 alert">Request submitted...</div>'
           );
         }
-        ,success: function(content) {
+        ,success: function() {
           $('#' + tab + '_form').trigger('submit');
         }
+        // skip any error reporting for now
+        // TODO: fix sanity_ok in Netdisco Web
         ,error: function() {
-          $(target).html(
+          $('#' + tab + '_form').trigger('submit');
-            '<div class="span5 alert alert-error">' +
-            'Request failed! Please contact your site administrator.</div>'
-          );
         }
       });
     });
+
+    // bind qtip2 to show the event log output
+    $(target).on('mouseover', '.nd_jobqueueitem', function(event) {
+      $(this).qtip({
+        overwrite: false,
+        content: {
+          attr: 'data-content'
+        },
+        show: {
+          event: event.type,
+          ready: true,
+          delay: 100
+        },
+        position: {
+          my: 'top center',
+          at: 'bottom center',
+          target: false
+        },
+        style: {
+          classes: 'qtip-cluetip qtip-rounded nd_qtip-unconstrained'
+        }
+      });
+    });
+
   });

Netdisco/share/views/layouts/main.tt

@@ -48,14 +48,6 @@
 </head>
 
 <body>
-[%
-  user_dd = [
-    { "title" = "Settings", "link" = "/settings" },
-    { "title" = "Help", "link" = "/help" },
-    { "title" = "Log Out", "link" = "/logout" }
-  ]
-%]
-
 <div class="navbar navbar-inverse navbar-fixed-top">
   <div class="navbar-inner">
     <div class="container">
@@ -99,6 +91,22 @@
         <li class="dropdown[% ' active' IF vars.nav == 'admin' %]">
           <a href="#" class="dropdown-toggle" data-toggle="dropdown">Admin Tasks <b class="caret"></b></a>
           <ul class="dropdown-menu">
+          <li>
+            <form method="post" class="nd_inline-form" action="[% uri_for('/admin/discoverall') %]">
+              <button type="submit" class="btn btn-link nd_btn-link">Discover All</button>
+            </form>
+          </li>
+          <li>
+            <form method="post" class="nd_inline-form" action="[% uri_for('/admin/arpwalk') %]">
+              <button type="submit" class="btn btn-link nd_btn-link">Arpnip All</button>
+            </form>
+          </li>
+          <li>
+            <form method="post" class="nd_inline-form" action="[% uri_for('/admin/macwalk') %]">
+              <button type="submit" class="btn btn-link nd_btn-link">Macsuck All</button>
+            </form>
+          </li>
+          [% '<li class="divider"></li>' IF settings._admin_tasks.keys.size %]
           [% FOREACH ai IN settings._admin_tasks.keys.sort %]
             <li><a href="[% uri_for('/admin/' _ ai) %]">[% settings._admin_tasks.$ai.label | html_entity %]</a></li>
           [% END %]
@@ -117,12 +125,14 @@
           [% ELSE %]
             <i class="icon-user"></i>
           [% END %]
+          [% IF NOT settings.no_auth %]
             [% session.user | html_entity %] <b class="caret"></b></a>
-          <ul class="dropdown-menu">
+            <ul class="dropdown-menu">
-            [% FOREACH item IN user_dd %]
+              <li><a href="[% uri_for('/logout') %]">Log Out</a></li>
-            <li><a href="[% uri_for(item.link) %]">[% item.title | html_entity %]</a></li>
+            </ul>
-            [% END %]
+          [% ELSE %]
-          </ul>
+            [% session.user | html_entity %]</a>
+          [% END %]
         </li> <!-- /dropdown -->
       </ul>
       [% END %]

Netdisco/share/views/plugin/observium/observium.js

@@ -1,6 +1,6 @@
 $(document).ready(function() {
 
-  // bind qtip2 even to all future .observium buttons
+  // bind qtip2 event to all future .nd_observium buttons
   $('#ports_pane').on('mouseover', '.nd_observium', function(event) {
     $(this).qtip({
       overwrite: false,

TODO

@@ -10,15 +10,11 @@ DAEMON
 BACKEND
 =======
 
-* db connect params as separate settings
-* new DBIC multi-prefetch
 * PING from workers
-* ask to set up guest user in deploy script
 
 CORE
 ====
 
-* NetAddr::MAC ?
 * VRF support
 * import legacy config file