OMD-Projekte Ansätze beim Monitoring der IT von Landeshauptstädten, Automobilherstellern und Discountern

Transkript

1 OMD-Projekte Ansätze beim Monitoring der IT von Landeshauptstädten, Automobilherstellern und Discountern Gerhard Laußer ConSol*

2 Generierung von Konfigdateien Build-Server site1 cmdb coshsh.cfg coshsh site2.csv site3 xy-db classes templates 2

3 Was geht rein Was kommt raus host.cfg cmdb sog. datasource os_linux_default.cfg os_linux_fs.cfg app_db2_default.cfg hosts applications contacts contactgroups app_db2_tbs.cfg 1 Unterverzeichnis pro Host coshsh 3

4 Coshsh-Klassen, z.b. os_linux.py class Linux(OperatingSystem): template_rules = [ TemplateRule(needsattr=None, template="os_linux_default"), TemplateRule(needsattr=None, template="os_linux_logs"), TemplateRule(needsattr="virtual", isattr="ps", template="os_linux_ilo"), TemplateRule(needsattr=None, template="os_linux_fs"), TemplateRule(needsattr="cluster", isattr="veritas", template="os_linux_cluster_veritas"), ] monitoring_details = [ MonitoringDetail({"monitoring_type" : "FILESYSTEM", 'path' : '/', 'warning' : 15, 'critical' : 5, 'units' : '%', 'optional' : False }), ] def new (cls, params={}): if compare_attr("version", params, ".*embedded.*"): cls = EmbeddedLinux elif compare_attr("virtual", params, "po"): # ping only cls = EmbeddedLinux #elif compare_attr("virtual", params, "VS"): # cls = XENGuest return object. new (cls) 4

5 Coshsh-Templates, z.b. os_linux_default.tpl {{ application service("os_linux_default_check_ntp") }} use os_linux_default host_name {{ application.host_name }} check_interval 121 retry_interval 10 check_command check_by_ssh!60!$user10$/local/lib/nagios/plugins/check_ntp_health } {{ application service("os_linux_default_check_crond") }} use os_linux_default host_name {{ application.host_name }} {% if application.version.startswith('sles') %} check_command check_by_ssh!60!$user10$/lib/nagios/plugins/check_procs -w :10 -c 1: -C cron {% else %} check_command check_by_ssh!60!$user10$/lib/nagios/plugins/check_procs -w :10 -c 1: -C crond {% endif %} } {% if application.host.virtual == "PS" %} {% if application.host.hardware.startswith("hp") %} {{ application service("os_linux_default_check_hardware") }} use os_linux_default host_name {{ application.host_name }} check_interval 5040 check_command check_by_ssh!60!$user10$/local/lib/nagios/plugins/check_hpasm } {% elif application.version is re_match('sles.*11') %} {{ application service("os_linux_default_check_hardware") }} use os_linux_default host_name {{ application.host_name }} check_interval 5040 check_command check_by_ssh!60!$user10$/local/lib/nagios/plugins/check_ipmitool } 5

6 Coshsh-Templates, z.b. os_linux_fs.tpl {% for fs in application.filesystems %} {{ application service('os_linux_fs_check_disk_' + fs.path) }} use os_linux_fs host_name {{ application.host_name }} check_interval 60 {% if fs.units == '%' %} check_command check_by_ssh!60!$user10$/lib/nagios/plugins/check_disk --warning {{ fs.warning }}% --critical {{ fs.critical }}% --path {{ fs.path }} {% else %} check_command check_by_ssh!60!$user10$/lib/nagios/plugins/check_disk --warning {{ fs.warning }} --critical {{ fs.critical }} --units {{ fs.units }} --path {{ fs.path }} {% endif %} } {% endfor %} {% if 'vrtsvxvm' in application.host.software %} {{ application service('os_linux_fs_check_vxvm') }} use os_linux_fs host_name {{ application.host_name }} check_interval 15 check_command check_by_ssh!60!$user10$/local/lib/nagios/plugins/check_vxvm } {{ application service('os_linux_fs_check_san_mirror') }} use os_linux_fs host_name {{ application.host_name }} check_interval 15 check_command check_by_ssh!400!$user10$/local/lib/nagios/plugins/check_san_mirror _CRITICAL_SEVERITY CRITICAL } {% endif %} 6

7 Coshsh-Templates, z.b. host.tpl define host { {% if host.templates %} use {{ host.templates }} {% endif %} host_name {{ host.host_name }} address {{ host.address }} alias {{ host.host_name }} {% if host.hostgroups %} hostgroups {{ host.hostgroups }} {% endif %} {% if host.contact_groups %} contact_groups {{ host.contact_groups }} {% endif %} {% if host.check_period %} check_period {{ host.check_period }} {% endif %} {% if host.notification_period %} notification_period {{ host.notification_period }} {% endif %} check_command check_host_alive _SSH_PORT {{ host.ports[-1] }} } 7

8 Wie kommt eins zum anderen? coshsh.cfg [datasource_mycmdb] type = mycmdb sid = cmdb username = nagios password = nagios Wo stehen die Rohdaten? [datasource_solarisextras] type = csv dir = /opt/nagios/coshsh/sites/mycomp/data [site_os_solaris] templates_dir = /opt/nagios/coshsh/sites/mycomp/templates classes_dir = /opt/nagios/coshsh/sites/mycomp/classes objects_dir = /opt/nagios/siteconfigs/os_solaris datasources = MYCMDB,SOLARISEXTRAS filter = MYCMDB(os_solaris) Aus den Hosts und Applications welcher Datasources setzt sich eine Site zusammen? Gibt es Site-spezifische Classes und Templates? Wohin mit den fertigen Konfigfiles? 8

9 Generieren einer Konfiguration (z.b. für die Site os_solaris) coshsh-generate --site os_solaris --config coshsh.cfg :36:46,093 INFO - site os_solaris init :36:46,093 INFO - site os_solaris objects_dir /opt/nagios/siteconfigs/os_solaris :36:46,094 INFO - site os_solaris classes_dir /opt/nagios/coshsh/sites/mycomp/classes :36:46,094 INFO - site os_solaris templates_dir /opt/nagios/coshsh/sites/mycomp/templates :36:46,205 INFO - site os_solaris remove dynamic_dir /opt/nagios/siteconfigs/os_solaris/dynamic/contacts :36:46,216 INFO - site os_solaris remove dynamic_dir /opt/nagios/siteconfigs/os_solaris/dynamic/hosts :36:46,266 INFO - site os_solaris remove dynamic_dir /opt/nagios/siteconfigs/os_solaris/dynamic/contactgroups :36:46,266 INFO - site os_solaris remove dynamic_dir /opt/nagios/siteconfigs/os_solaris/dynamic/hostgroups :36:46,278 INFO - site os_solaris dynamic_dir /opt/nagios/siteconfigs/os_solaris/dynamic :37:20,853 INFO - site os_solaris read from datasource mycompcmdb 210 hosts, 210 applications, 100 details, 251 contacts, 0 dependencies, 0 business processes :37:20,856 INFO - read hosts from /opt/nagios/coshsh/sites/mycomp/data/solarisextras_hosts.csv :37:20,857 INFO - read applications from /opt/nagios/coshsh/sites/mycomp/data/solarisextras_applications.csv :37:20,858 INFO - read appdetails from /opt/nagios/coshsh/sites/mycomp/data/solarisextras_applicationdetails.csv :37:20,858 INFO - read contactgroups from /opt/nagios/coshsh/sites/mycomp/data/solarisextras_contactgroups.csv :37:20,858 INFO - read contacts from /opt/nagios/coshsh/sites/mycomp/data/solarisextras_contacts.csv :37:20,859 INFO - site os_solaris read from datasource solarisextras 0 hosts, 4 applications, 0 details, 0 contacts, 0 dependencies, 0 business processes :37:20,904 INFO - load template host :37:20,985 INFO - load template os_solaris_default :37:20,990 INFO - load template os_solaris_logs :37:21,015 INFO - load template os_solaris_fs :37:21,019 INFO - load template os_solaris_cluster_veritas :37:21,022 INFO - load template os_solaris_admin :37:21,062 INFO - load template app_storage_sstcam_logs :37:21,122 INFO - load template contactgroup :37:21,134 INFO - load template contact :37:21,158 INFO - load template hostgroup (210, 654) :37:21,325 INFO - number of files before: 210 hosts, 654 applications :37:21,325 INFO - number of files after: 210 hosts, 654 applications git add git commit commit-comment hostfiles,654 appfiles Created commit b91a7b6: hostfiles,654 appfiles 2 files changed, 2 insertions(+), 2 deletions(-) 9

10 Wie gelangt eine generierte Konfiguration in die Produktion? Build-Server Prod-Server siteconfigs mycomp /omd/sites/os_solaris/ etc/nagios/conf.d/ mycomp static pull os_solaris os_solaris static static dynamic coshsh dynamic pull

11 Nach dem Build.Durchstarten von Nagios auf dem Prod-Server sudo omd su os_solaris OMD[os_solaris]:~$ omd reload nagios ~/etc/nagios/conf.d/mycomp/os_solaris/static ~ Already up-to-date. ~ ~/etc/nagios/conf.d/mycomp/os_solaris/dynamic ~ remote: Counting objects: 15, done. remote: Compressing objects: 100% (7/7), done. remote: Total 8 (delta 5), reused 0 (delta 0) Unpacking objects: 100% (8/8), done. From ssh://nagios@nagiosbuild/opt/nagios/siteconfigs/os_solaris/dynamic 52b6a64..b91a7b6 master -> origin/master Updating 52b6a64..b91a7b6 Fast forward hostgroups/filer_nvfi012/hostgroup.cfg 2 +- hosts/ssap018.mycomp.de/host.cfg files changed, 2 insertions(+), 2 deletions(-) ~ Running configuration check... done. Reloading nagios configuration (PID: 14725)... OK OMD[os_solaris]:~$ OMD init-hooks nagios-reload-pre macht git pull oder ggf. git clone 11

12 Monitoring mit check_by_ssh und ControlMaster ssh N -ocontrolmaster=yes ssh N -ocontrolmaster=yes ssh N -ocontrolmaster=yes Teurer Verbindungsaufbau (einmalig) check_by_ssh -ocontrolmaster=no check_by_ssh -ocontrolmaster=no check_by_ssh -ocontrolmaster=no check_by_ssh -ocontrolmaster=no Host A Host B Host A Host B 12

13 Monitoring mit check_by_ssh und ControlMaster, Beispiel Linux-OS 13

14 Zurück zum Build-Server es gibt noch eine Site rollout In rollout sind sämtliche Hosts drin. Jeder hat 1 Service. Das dazugehörige Plugin loggt sich am Host ein und: Ermittelt das genaue Betriebssystem, z.b. i86pc_i386_sunos_5.10 Rsync d den Verzeichnisbaum mit Plugins (für dieses Betriebssystem) zum Host. Ermittelt installierte Pakete, Mounts, /proc/modules, custom configs mit Filesystemschwellwerten der Serverbetreiber, Schreibt dies Informationen in eine MongoDB Das Ganze 2x täglich Die Konfiguration von rollout wird stündlich gebaut. Neuankömmlinge in der CMDB (=pending um 5 nach) werden sofort scheduled. 14

15 Zurück zum Build-Server es gibt noch eine Site rollout In rollout sind sämtliche Hosts drin. Jeder hat 1 Service. Das dazugehörige Plugin loggt sich am Host ein und: Ermittelt das genaue Betriebssystem, z.b. i86pc_i386_sunos_5.10 Rsync d den Verzeichnisbaum mit Plugins (für dieses Betriebssystem) zum Host. Ermittelt installierte Pakete, Mounts, /proc/modules, custom configs mit Filesystemschwellwerten der Serverbetreiber, Schreibt dies Informationen in eine MongoDB Das Ganze 2x täglich Die Konfiguration von rollout wird stündlich gebaut. Neuankömmlinge in der CMDB (=pending um 5 nach) werden sofort scheduled. Clients sind up to date. Der Monitoring-Betrieb hat die Hoheit. 15

16 Wer braucht diese Informationen? massgeschneidert! cmdb Adapter cmdb.py info = db.find({host_name: $hostname}) host.software = info[ software ] host.cluster = info[ cluster ][ type ]. coshsh os_linux_fs.tpl {% if 'vrtsvxvm' in application.host.software %} {{ application service('os_linux_fs_check_vxvm') }} use os_linux_fs host_name {{ application.host_name }} check_interval 15 check_command }. os_linux.py check_by_ssh!\ 60!$USER10$/local/lib/nagios/plugins/check_vxvm class Linux(OperatingSystem): template_rules = [ TemplateRule(needsattr=None, template="os_linux_default"), TemplateRule(needsattr="cluster", isattr="veritas", template="os_linux_cluster_veritas"),. 16

17 Wozu braucht man diese Informationen noch? Die rollout -Site sammelt auch die fstabs ein, insb. die NFS-Mounts. Beim Generieren landen alle Hosts, die etwas vom Filer XY mounten, in der Hostgroup filer_xy. Bei Filerchanges bekommt man auf einen Blick die Hosts, die davon betroffen sind. Der Serverbetrieb hinterlegt Dateien der Form: /opt/applxy 10% 5% /u0/dbfs 1G 200M Beim Generieren werden die entpr. check_disk-services angelegt. 17

18 PAUSE, FRAGEN, DEMO,...etc 18