Files
netdisco/lib/App/Netdisco/Configuration.pm
Oliver Gorwits d44a8f56ea Integrate netdisco-sshcollector into Worker::Plugin architecture (#489) (#535)
* update changes and SNMP::Info dep

* Integrate netdisco-sshcollector into Worker::Plugin architecture (#489)

* Initial integration of sshcollector into Worker::Plugin architecture

 * add NodesBySSH.pm
 * update Build.PL and config.yml to integrate the new module

* Further integration of sshcollector into Worker::Plugin architecture

 * added App::Netdisco::Transport::CLI loosely based on ::SNMP counterpart
 * switched to the more prevalent two-space tabs style
 * removed various TBD items, some new ones

* Further steps to integration of sshcollector into Worker::Plugin architecture

 * cleaned up code
 * added various error handling
 * warning for bin/netdisco-sshcollector deprecation
 * device_auth allows passing master_opts to Net::OpenSSH
 * netdisco-do -D also toggles Net::OpenSSH debug

* Merged NodesBySSH.pm into Nodes.pm

 * see https://github.com/netdisco/netdisco/pull/489#pullrequestreview-205603516

* Further integration of sshcollector into Worker::Plugin architecture

 * add snmp_arpnip_also option to sshcollector device_auth
 * cleanup code

* Remove big TBD: comment from CLI.pm as doc is updated now

* add transport/cli.pm to manifest

* revert some changes to allow simpler merging

* silent exit legacy script unless explicitly requested

* move ssh code into Transport, part one

* rewrite the CLI transport to provide an API

* merge in og-get_external_credentials

Squashed commit of the following:

commit 3fe8f383a7
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 11 17:07:42 2019 +0000

    add debug lines and tested

commit 3249739e42
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 11 16:54:11 2019 +0000

    change config name to get_credentials

commit e78558397a
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 11 16:51:11 2019 +0000

    separate out generic device auth to DeviceAuth module

commit 249f05165f
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Wed Mar 6 18:43:31 2019 +0000

    release 2.040007

commit e3af64df77
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Wed Mar 6 18:42:47 2019 +0000

    #521-redux fix wifi date search

commit 48857ae300
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 4 12:03:31 2019 +0000

    release 2.040006

commit e09dab5362
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 4 11:39:12 2019 +0000

    #527 update List::MoreUtils version requirement

commit 6e7de3fff3
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Mon Mar 4 09:59:41 2019 +0000

    release 2.040005

commit 0c98318a45
Author: Oliver Gorwits <oliver@spike.local>
Date:   Mon Mar 4 09:57:18 2019 +0000

    #526 fix discover syntax bug

commit e9efc45182
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 14:56:48 2019 +0000

    release 2.040004

commit 6cdfd80d10
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 14:34:00 2019 +0000

    allow undiscovered neighbors report to use discover_{waps,phones} setting

commit ac381e0802
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 14:13:20 2019 +0000

    #506 was a red herring

commit b83e614c85
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 13:00:36 2019 +0000

    make discover_{phones,waps} work with LLDP capabilities as well

commit 189d234b55
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 12:47:38 2019 +0000

    check discover_no_type and friends earlier on in neighbors list build

commit 9c956466f3
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 12:32:07 2019 +0000

    also update default config for new discover_phones and discover_waps settings

commit 09d29954d2
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 12:26:50 2019 +0000

    #512 fix regression in phone/wap discovery exclusion

commit 2bae91f1b6
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 12:01:34 2019 +0000

    rename match_devicetype() to match_to_setting()

commit 57cb6ddb70
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sun Mar 3 09:19:39 2019 +0000

    fix for over-eager fix to #506

commit ef560fb59a
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 22:41:40 2019 +0000

    #506 relax device renumber so it works for an alias

commit 7a8bcb094e
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 22:23:39 2019 +0000

    #521 Search Node Date Range not working

commit a643820a62
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 21:54:27 2019 +0000

    #428 Port-Channels not showing in netmap

commit 5ba5bcd295
Merge: e7aacddb a1f95028
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 20:04:11 2019 +0000

    Merge branch 'master' of github.com:netdisco/netdisco

commit e7aacddbc6
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 20:01:05 2019 +0000

    #498 Map with VLAN filter omits unconnected devices

commit a1f95028ca
Author: nick n <39005454+inphobia@users.noreply.github.com>
Date:   Sat Mar 2 19:54:22 2019 +0100

    catch up with changes

    noticed that rc-sshcollector-core received updates to changes, add them here as well.

    didn't mention #499 & #522

commit ce1b847cea
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 18:47:44 2019 +0000

    fix bug showing no nodes when only one matches in netmap

commit 78e30a7926
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 16:28:15 2019 +0000

    #500 filtering in device/ports on native vlan duplicates entries

commit 9952f0c6c7
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 15:02:12 2019 +0000

    #499 netdisco-do renumber reports wrong ip (inphobia)

commit ca3fd8f466
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 15:00:18 2019 +0000

    #505 device renumber should update device port properties and device skips

commit 1265bc8470
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 14:52:21 2019 +0000

    #520 catch slave ports defined without a master

commit d4c7579c10
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 14:47:49 2019 +0000

    #522 TypeAhead.pm can reference empty data (inphobia)

commit 77decc23b7
Author: Oliver Gorwits <oliver@cpan.org>
Date:   Sat Mar 2 14:45:37 2019 +0000

    #514 inconsistent results in ip inventory (inphobia)

commit 3f211650b8
Author: nick n <39005454+inphobia@users.noreply.github.com>
Date:   Fri Mar 1 12:34:42 2019 +0100

    last pieces for db schema upgrade

    last piece of #510

* import legacy sshcollector config

* add default use_legacy_sshcollector config

* remove unneeded deps

* various fixes and now tested

* enable sshcollector platform tests
2019-03-12 18:50:24 +00:00

208 lines
6.9 KiB
Perl
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

package App::Netdisco::Configuration;
use App::Netdisco::Environment;
use App::Netdisco::Util::DeviceAuth ();
use Dancer ':script';
use Path::Class 'dir';
use Net::Domain 'hostdomain';
BEGIN {
if (setting('include_paths') and ref [] eq ref setting('include_paths')) {
# stuff useful locations into @INC
push @{setting('include_paths')},
dir(($ENV{NETDISCO_HOME} || $ENV{HOME}), 'nd-site-local', 'lib')->stringify
if (setting('site_local_files'));
unshift @INC, @{setting('include_paths')};
}
}
# set up database schema config from simple config vars
if (ref {} eq ref setting('database')) {
# override from env for docker
setting('database')->{name} =
($ENV{NETDISCO_DB_NAME} || $ENV{NETDISCO_DBNAME} || setting('database')->{name});
setting('database')->{host} =
($ENV{NETDISCO_DB_HOST} || setting('database')->{host});
setting('database')->{host} .= (';'. $ENV{NETDISCO_DB_PORT})
if (setting('database')->{host} and $ENV{NETDISCO_DB_PORT});
setting('database')->{user} =
($ENV{NETDISCO_DB_USER} || setting('database')->{user});
setting('database')->{pass} =
($ENV{NETDISCO_DB_PASS} || setting('database')->{pass});
my $name = setting('database')->{name};
my $host = setting('database')->{host};
my $user = setting('database')->{user};
my $pass = setting('database')->{pass};
my $dsn = "dbi:Pg:dbname=${name}";
$dsn .= ";host=${host}" if $host;
# set up the netdisco schema now we have access to the config
# but only if it doesn't exist from an earlier config style
setting('plugins')->{DBIC}->{netdisco} ||= {
dsn => $dsn,
user => $user,
password => $pass,
options => {
AutoCommit => 1,
RaiseError => 1,
auto_savepoint => 1,
pg_enable_utf8 => 1,
},
schema_class => 'App::Netdisco::DB',
};
foreach my $c (@{setting('external_databases')}) {
my $schema = delete $c->{tag} or next;
next if $schema eq 'netdisco';
setting('plugins')->{DBIC}->{$schema} = $c;
setting('plugins')->{DBIC}->{$schema}->{schema_class}
||= 'App::Netdisco::GenericDB';
}
}
# always set this
$ENV{DBIC_TRACE_PROFILE} = 'console';
# override from env for docker
config->{'community'} = ($ENV{NETDISCO_RO_COMMUNITY} ?
[split ',', $ENV{NETDISCO_RO_COMMUNITY}] : config->{'community'});
config->{'community_rw'} = ($ENV{NETDISCO_RW_COMMUNITY} ?
[split ',', $ENV{NETDISCO_RW_COMMUNITY}] : config->{'community_rw'});
# if snmp_auth and device_auth not set, add defaults to community{_rw}
if ((setting('snmp_auth') and 0 == scalar @{ setting('snmp_auth') })
and (setting('device_auth') and 0 == scalar @{ setting('device_auth') })) {
config->{'community'} = [ @{setting('community')}, 'public' ];
config->{'community_rw'} = [ @{setting('community_rw')}, 'private' ];
}
# fix up device_auth (or create it from old snmp_auth and community settings)
# also imports legacy sshcollcetor config
config->{'device_auth'}
= [ App::Netdisco::Util::DeviceAuth::fixup_device_auth() ];
# defaults for workers
setting('workers')->{queue} ||= 'PostgreSQL';
if ($ENV{ND2_SINGLE_WORKER}) {
setting('workers')->{tasks} = 1;
delete config->{'schedule'};
}
# force skipped DNS resolution, if unset
setting('dns')->{hosts_file} ||= '/etc/hosts';
setting('dns')->{no} ||= ['fe80::/64','169.254.0.0/16'];
# set max outstanding requests for AnyEvent::DNS
$ENV{'PERL_ANYEVENT_MAX_OUTSTANDING_DNS'}
= setting('dns')->{max_outstanding} || 50;
$ENV{'PERL_ANYEVENT_HOSTS'} = setting('dns')->{hosts_file};
# load /etc/hosts
setting('dns')->{'ETCHOSTS'} = {};
{
# AE::DNS::EtcHosts only works for A/AAAA/SRV, but we want PTR.
# this loads+parses /etc/hosts file using AE. dirty hack.
use AnyEvent::Socket 'format_address';
use AnyEvent::DNS::EtcHosts;
AnyEvent::DNS::EtcHosts::_load_hosts_unless(sub{},AE::cv);
no AnyEvent::DNS::EtcHosts; # unimport
setting('dns')->{'ETCHOSTS'}->{$_} =
[ map { [ $_ ? (format_address $_->[0]) : '' ] }
@{ $AnyEvent::DNS::EtcHosts::HOSTS{ $_ } } ]
for keys %AnyEvent::DNS::EtcHosts::HOSTS;
}
# override from env for docker
if ($ENV{NETDISCO_DOMAIN}) {
if ($ENV{NETDISCO_DOMAIN} eq 'discover') {
delete $ENV{NETDISCO_DOMAIN};
if (! setting('domain_suffix')) {
info 'resolving domain name...';
config->{'domain_suffix'} = hostdomain;
}
}
else {
config->{'domain_suffix'} = $ENV{NETDISCO_DOMAIN};
}
}
# support unordered dictionary as if it were a single item list
if (ref {} eq ref setting('device_identity')) {
config->{'device_identity'} = [ setting('device_identity') ];
}
else { config->{'device_identity'} ||= [] }
# copy devices_no and devices_only into others
foreach my $name (qw/devices_no devices_only
discover_no macsuck_no arpnip_no nbtstat_no
discover_only macsuck_only arpnip_only nbtstat_only/) {
config->{$name} ||= [];
config->{$name} = [setting($name)] if ref [] ne ref setting($name);
}
foreach my $name (qw/discover_no macsuck_no arpnip_no nbtstat_no/) {
push @{setting($name)}, @{ setting('devices_no') };
}
foreach my $name (qw/discover_only macsuck_only arpnip_only nbtstat_only/) {
push @{setting($name)}, @{ setting('devices_only') };
}
# legacy config item names
config->{'devport_vlan_limit'} =
config->{'deviceport_vlan_membership_threshold'}
if setting('deviceport_vlan_membership_threshold')
and not setting('devport_vlan_limit');
delete config->{'deviceport_vlan_membership_threshold'};
config->{'schedule'} = config->{'housekeeping'}
if setting('housekeeping') and not setting('schedule');
delete config->{'housekeeping'};
# used to have separate types of worker
if (exists setting('workers')->{interactives}
or exists setting('workers')->{pollers}) {
setting('workers')->{tasks} ||=
(setting('workers')->{pollers} || 0)
+ (setting('workers')->{interactives} || 0);
delete setting('workers')->{pollers};
delete setting('workers')->{interactives};
}
# moved the timeout setting
setting('workers')->{'timeout'} = setting('timeout')
if defined setting('timeout')
and !defined setting('workers')->{'timeout'};
# 0 for workers max_deferrals and retry_after is like disabling
# but we need to fake it with special values
setting('workers')->{'max_deferrals'} ||= (2**30);
setting('workers')->{'retry_after'} ||= '100 years';
# schedule expire used to be called expiry
setting('schedule')->{expire} ||= setting('schedule')->{expiry}
if setting('schedule') and exists setting('schedule')->{expiry};
delete config->{'schedule'}->{'expiry'} if setting('schedule');
# upgrade reports config from hash to list
if (setting('reports') and ref {} eq ref setting('reports')) {
config->{'reports'} = [ map {{
tag => $_,
%{ setting('reports')->{$_} }
}} keys %{ setting('reports') } ];
}
# add system_reports onto reports
config->{'reports'} = [ @{setting('system_reports')}, @{setting('reports')} ];
true;