135/netdisco
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

makerancidconf overhaul

Browse Source 
* Use the Fullname in top bar of Web interface and not the login

* Implement the following parameters:
 - add a rancid_conf parameter to retrieve the existing type for RANCID devices
 - add the default_group parameter to add devices in this group if they doesn't match the other parameters
 - add the hability to work with the RANCID version in https://github.com/earendilfr/rancid with subdirectory for the configuration files

* Fix following issues:
- Problem during the check of ACL when we have a negate character. The begate character is deleted during the first check
- The renumber function doesn't want to start

* Revert "Fix following issues:"

This reverts commit a8d9e8200a925b1a4d24374827d3fd5146f85b40.

* Revert "Implement the following parameters:"

This reverts commit aa29ca924cd92943490e4a2420a3972054e81e41.

* Add the following parameters to the MakeRancidConf:

 + Change the default delimiter character from ':' to ';' that is the default RANCID delimiter since
   RANCID 3.x
 + Add an option to define the rancid config directory.
   This directory will be used to extract the vendor defined in the RANCID configuration to avoid to extract
   unexisting vendor devices in RANCID configuration (by example, Ubiquiti AirOS devices).
 + Add an option to define a default_group.
   If a device doesn't match the defined group, it will add to this group
 + Add an option to defined a manually excluded ACL
   To permit to not export devices from Netdisco to RANCID based on an ACL.

* New version of the pull request to merge the makerancid improvment.
I have fixe some errors that have impact the revert for the PR #408

I have manually check on my instance and didn't see any warning on the result of the command netdisco-do makerancidconf

* Fix an issue if the by_ip group not exist in the RANCiD part of configuration

* Print the entry sorted in the router.db file

* testing if the sql in #436 can be integrated

* a bit of documentation cleanup

* a few more doc fixes

* minor docs changes

* rename rancid_home to rancid_cvsroot, which is how the setting is defined in rancid. this should make things clearer

* a bit more in-depth examples. allow for more device types (allow - in their name). use strict & warnings

* just to many caps
This commit is contained in:
nick n
2019-01-14 01:56:32 +01:00
committed by GitHub
parent 562d525b78
commit 0205a11f01
1 changed files with 139 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

189
lib/App/Netdisco/Worker/Plugin/MakeRancidConf.pm
View File

@@ -1,5 +1,8 @@
package App::Netdisco::Worker::Plugin::MakeRancidConf;
use strict;
use warnings;
use Dancer ':syntax';
use Dancer::Plugin::DBIC;
@@ -8,7 +11,7 @@ use aliased 'App::Netdisco::Worker::Status';
use Path::Class;
use List::Util qw/pairkeys pairfirst/;
use File::Slurper 'write_text';
use File::Slurper qw/read_lines write_text/;
use App::Netdisco::Util::Permission 'check_acl_no';
register_worker({ phase => 'main' }, sub {
@@ -16,14 +19,31 @@ register_worker({ phase => 'main' }, sub {
my $config = setting('rancid') || {};
my $domain_suffix = setting('domain_suffix') || '';
my $delimiter = $config->{delimiter} || ':';
my $delimiter = $config->{delimiter} || ';';
my $down_age = $config->{down_age} || '1 day';
my $default_group = $config->{default_group} || 'default';
my $rancidhome = $config->{rancid_home}
my $rancidconf = $config->{rancid_conf} || '/etc/rancid';
my $rancidcvsroot = $config->{rancid_cvsroot}
|| dir($ENV{NETDISCO_HOME}, 'rancid')->stringify;
mkdir $rancidhome if ! -d $rancidhome;
return Status->error("cannot create or see rancid home: $rancidhome")
if ! -d $rancidhome;
mkdir $rancidcvsroot if ! -d $rancidcvsroot;
return Status->error("cannot create or access rancid cvsroot: $rancidcvsroot")
if ! -d $rancidcvsroot;
my $allowed_types = {};
foreach my $type (qw/base conf/) {
my $type_file = file($rancidconf, "rancid.types.$type")->stringify;
debug sprintf("trying rancid configuration file %s\n", $type_file);
next unless -f $type_file;
my @lines = read_lines($type_file);
foreach my $line (@lines) {
next if $line =~ m/^(?:\#|\$)/;
$allowed_types->{$1} += 1 if $line =~ m/^([a-z0-9_\-]+);login;.*$/;
}
}
return Status->error("You didn't have any device types configured in your rancid installation.")
if ! scalar keys %$allowed_types;
my $devices = schema('netdisco')->resultset('Device')->search(undef, {
'+columns' => { old =>
@@ -32,24 +52,37 @@ register_worker({ phase => 'main' }, sub {
$config->{groups} ||= { default => 'any' };
$config->{vendormap} ||= {};
$config->{excluded} ||= {};
$config->{by_ip} ||= {};
$config->{by_hostname} ||= {};
my $routerdb = {};
while (my $d = $devices->next) {
my $name =
check_acl_no($d, $config->{by_ip}) ? $d->ip : ($d->dns || $d->name);
$name =~ s/$domain_suffix$//
if check_acl_no($d, $config->{by_hostname});
if (check_acl_no($d, $config->{excluded})) {
debug " skipping $d: device excluded of export";
next
}
my $name = check_acl_no($d, $config->{by_ip}) ? $d->ip : ($d->dns || $d->name);
$name =~ s/$domain_suffix$// if check_acl_no($d, $config->{by_hostname});
my ($group) =
pairkeys pairfirst { check_acl_no($d, $b) } %{ $config->{groups} };
(pairkeys pairfirst { check_acl_no($d, $b) } %{ $config->{groups} }) || $default_group;
my ($vendor) =
(pairkeys pairfirst { check_acl_no($d, $b) } %{ $config->{vendormap} })
|| $d->vendor;
if ($vendor =~ m/(?:enterprises\.|netdisco)/) {
if (not ($name and $vendor)) {
debug " skipping $d: the name or vendor is not defined";
next
} elsif ($vendor =~ m/(?:enterprises\.|netdisco)/) {
debug " skipping $d with unresolved vendor: $vendor";
next;
} elsif (scalar keys %$allowed_types and !exists($allowed_types->{$vendor})) {
debug " skipping $d: $vendor doesn't exist in rancid's vendor list";
next;
}
push @{$routerdb->{$group}},
@@ -58,12 +91,14 @@ register_worker({ phase => 'main' }, sub {
}
foreach my $group (keys %$routerdb) {
mkdir dir($rancidhome, $group)->stringify;
my $content = join "\n", @{$routerdb->{$group}};
write_text(file($rancidhome, $group, 'router.db')->stringify, "${content}\n");
mkdir dir($rancidcvsroot, $group)->stringify;
my $content = "#\n# Router list file for rancid group $group.\n";
$content .= "# Generate automatically by App::Netdisco::Worker::Plugin::MakeRancidConf\n#\n";
$content .= join "\n", sort @{$routerdb->{$group}};
write_text(file($rancidcvsroot, $group, 'router.db')->stringify, "${content}\n");
}
return Status->done('Wrote RANCID configuration.');
return Status->done('Wrote rancid configuration.');
});
true;
@@ -72,17 +107,20 @@ true;
=head1 NAME
MakeRancidConf - Generate RANCID Configuration
MakeRancidConf - Generate rancid Configuration
=head1 INTRODUCTION
This worker will generate a RANCID configuration for all devices in Netdisco.
This worker will generate a rancid configuration for all devices in Netdisco.
Optionally you can provide configuration to control the output, however the
defaults are sane, and will create one RANCID group called "C<default>" which
contains all devices. Those devices not discovered successfully within the
past day will be marked as "down" for RANCID to skip. Configuration is saved
to the "rancid" subdirectory of Netdisco's home folder.
defaults are sane for rancid versions 3.x and will create one rancid group
called C<default> which contains all devices. Those devices not discovered
successfully within the past day will be marked as C<down> for rancid to skip.
Configuration is saved to the F<~/rancid> subdirectory of Netdisco's home folder.
Note that this only generates the router.db files, you will still need to
configure rancid's F<.cloginrc> and schedule C<rancid-run> to run.
You could run this worker at 09:05 each day using the following configuration:
@@ -90,95 +128,142 @@ You could run this worker at 09:05 each day using the following configuration:
makerancidconf:
when: '5 9 * * *'
Since MakeRancidConf is a worker module it can also be run via C<netdisco-do>:
netdisco-do makerancidconf
=head1 CONFIGURATION
Here is a complete example of the configuration, which must be called
"C<rancid>". All keys are optional:
C<rancid>. All keys are optional:
rancid:
rancid_home: "$ENV{NETDISCO_HOME}/rancid" # default
rancid_cvsroot: '$ENV{NETDISCO_HOME}/rancid' # default
rancid_conf: '/etc/rancid' # default
down_age: '1 day' # default
delimiter: ':' # default
delimiter: ';' # default
default_group: 'default' # default
excluded:
excludegroup1: 'host_group1_acl'
excludegroup2: 'host_group2_acl'
groups:
groupname1: 'host_group1_acl'
groupname2: 'host_group2_acl'
groupname1: 'host_group3_acl'
groupname2: 'host_group4_acl'
vendormap:
vname1: 'host_group3_acl'
vname2: 'host_group4_acl'
by_ip: 'host_group5_acl'
by_hostname: 'host_group6_acl'
vname1: 'host_group5_acl'
vname2: 'host_group6_acl'
by_ip: 'host_group7_acl'
by_hostname: 'host_group8_acl'
Note that the default home for writing files is not "C</var/lib/rancid>" so
you may wish to set this (especially if migrating from the old
Note that the default directory for writing files is not F</var/lib/rancid> so
you may wish to set this in C<rancid_cvsroot>, (especially if migrating from the old
C<netdisco-rancid-export> script).
Any values above that are a Host Group ACL will take either a single item or
list of Network Identifiers or Device Properties. See the L<ACL
Any values above that are a host group ACL will take either a single item or
a list of network identifiers or device properties. See the L<ACL
documentation|https://github.com/netdisco/netdisco/wiki/Configuration#access-control-lists>
wiki page for full details. We advise you to use the "C<host_groups>" setting
wiki page for full details. We advise you to use the C<host_groups> setting
and then refer to named entries in that, for example:
host_groups:
coredevices: '192.0.2.0/24'
edgedevices: '172.16.0.0/16'
grp-nxos: 'os:nx-os'
rancid:
groups:
core_devices: 'group:coredevices'
edge_devices: 'group:edgedevices'
vendormap:
cisco-nx: 'group:grp-nxos'
by_ip: 'any'
=head2 C<rancid_home>
Do not forget that rancid also needs configuring when adding a new group,
such as scheduling the group to run, adding it to F<rancid.conf>, setting up the
email config and creating the repository with C<rancid-cvs>.
The location to write RANCID Group configuration files into. A subdirectory
for each Group will be created.
=head2 C<rancid_conf>
The location where the rancid configuration (F<rancid.types.base> and
F<rancid.types.conf>) is installed. It will be used to check the existance
of device types before exporting the devices to the rancid configuration. if no match
is found the device will not be added to rancid.
=head2 C<rancid_cvsroot>
The location to write rancid group configuration files (F<router.db>) into. A
subdirectory for each group will be created.
=head2 C<down_age>
This should be the same or greater than the interval between regular discover
jobs on your network. Devices which have not been discovered within this time
will be marked as "C<down>" to RANCID.
will be marked as C<down> to rancid.
The format is any time interval known and understood by PostgreSQL, such as at
L<https://www.postgresql.org/docs/8.4/static/functions-datetime.html>.
L<https://www.postgresql.org/docs/10/static/functions-datetime.html>.
=head2 C<delimiter>
Set this to the delimiter character if needed to be different from the
default.
Set this to the delimiter character for your F<router.db> entries if needed to
be different from the default, the default is C<;>.
=head2 C<default_group>
Put devices into this group if they do not match any other groups defined.
=head2 C<excluded>
This dictionary defines a list of devices that you do not wish to export to
rancid configuration.
The value should be a L<Netdisco ACL|https://github.com/netdisco/netdisco/wiki/Configuration#access-control-lists>
to select devices in the Netdisco database.
=head2 C<groups>
This dictionary maps RANCID Group names with configuration which will match
This dictionary maps rancid group names with configuration which will match
devices in the Netdisco database.
The left hand side (key) should be the RANCID group name, the right hand side
The left hand side (key) should be the rancid group name, the right hand side
(value) should be a L<Netdisco
ACL|https://github.com/netdisco/netdisco/wiki/Configuration#access-control-lists>
to select devices in the Netdisco database.
=head2 C<vendormap>
If the device Vendor in Netdisco is not the same as the RANCID vendor script,
configure a mapping here.
If the device vendor in Netdisco is not the same as the rancid vendor script or
device type, configure a mapping here.
The left hand side (key) should be the RANCID vendor, the right hand side
The left hand side (key) should be the rancid device type, the right hand side
(value) should be a L<Netdisco
ACL|https://github.com/netdisco/netdisco/wiki/Configuration#access-control-lists>
to select devices in the Netdisco database.
Note that vendors might have a large array of operating systems which require
different rancid modules. Mapping operating systems to rancid device types is
a good solution to use the correct device type. Example:
host_groups:
grp-ciscosb: 'os:ros'
rancid:
vendormap:
cisco-sb: 'group:grp-ciscosb'
=head2 C<by_ip>
L<Netdisco
ACL|https://github.com/netdisco/netdisco/wiki/Configuration#access-control-lists>
to select devices which will be written to the RANCID config as an IP address,
instead of the DNS FQDN or SNMP host name.
to select devices which will be written to the rancid config as an IP address,
instead of the DNS FQDN or SNMP hostname.
=head2 C<by_hostname>
L<Netdisco
ACL|https://github.com/netdisco/netdisco/wiki/Configuration#access-control-lists>
to select devices which will have the unqualified host name written to the
RANCID config. This is done simply by stripping the C<domain_suffix>
to select devices which will have the unqualified hostname written to the
rancid config. This is done simply by stripping the C<domain_suffix>
configuration setting from the device FQDN.
=head1 SEE ALSO