Datensammeln für Service Requests Ernst Leber

Transkript

1 Datensammeln für Service Requests Ernst Leber

2 Im Überblick Technologie-orientiert Branchen-unabhängig Hauptsitz Ratingen 240 Beschäftigte Ausbildungsbetrieb Inhabergeführte Aktiengesellschaft Gründungsjahr 1994 Niederlassung Frankfurt am Main Zertifizierter Partner von Oracle, Microsoft und SAP 2

3 Agenda Motivation Wie sage ich. Tools Sammeln der Daten Links und MOS Notes 3

4 Wer kennt das nicht: SR erstellt neue (?) Daten werden vom Support angefordert Rückfragen Nein, eine Downtime um den Fehler nachzustellen, ist nicht möglich! 4

5 Problem Der Support Engineer kennt Ihr System nicht! Was ist passiert => Das Problem ORA-????? Wo => Ihr System: Datenbank, Betriebssystem, Hardware Auswirkung Loss of Service Minor Severe Workaround Sporadisch / Reproduzierbar Was wurde geändert 5

6 Problem Unterschiedliche Datenquellen Unterschiedliche Darstellung Aufbereiten der Daten dauert Viel Zeit vergeht SR Erstellung (Sie) Analyse der Daten (Support) Server Betriebssystem Datenbank Version Patch Level Daten zum Problem Log-Files Trace-Files Eigene Daten oder Beobachtungen 6

7 Tools Normierung mit Tools: Automatic Diagnostic Repository Command Interpreter adrci Trace File Analyzer tfactl opatch awr / ash sqlplus MOS-Scripte 7

8 Wir haben ein Problem ORA ORA ORA ORA-????? RAC Startet nicht Node Eviction Diskgroups korrupt Applikationsfehler 8

9 alert-log ORA-7445 Beispiele :57: :00 Errors in file../diag/rdbms/../../incident/incdir_ /.._j015_63472_i trc: ORA-00039: Fehler bei periodischer Aktion ORA-00602: Exception bei interner Programmierung ORA-07445: Exception aufgetreten: CORE Dump [kghfrempty_sh_all_depth()+69] [SIGSEGV] [ADDR:0x0] [PC:0xD129195] [SI_KERNEL(general_protection)] [] ORA-07445: Exception aufgetreten: CORE Dump [kghfnd_in_free_lists()+713] [SIGSEGV] [ADDR:0xFFFFFFFFFFFFFFF0] [PC:0xD1320C9] [Address not mapped to object] [] Dumping diagnostic data in directory=[cdmp_ ], requested by (instance=1, osid=63472 (J015)), summary=[incident= ]. 9

10 alert-log ORA Beispiele :57: :00 Errors in file../diag/rdbms/../../trace/xx_j015_63472.trc (incident= ): ORA-00600: Interner Fehlercode, Argumente: [dbgtb:iter_max_recs], [3897], [3896], [0x7F12F6B180A8], [0x7F12F6B18000], [], [], [], [], [], [], [] ORA-07445: Exception aufgetreten: CORE Dump [kghfnd_in_free_lists()+713] [SIGSEGV] [ADDR:0xFFFFFFFFFFFFFFF0] [PC:0xD1320C9] [Address not mapped to object] [] Incident details in:../../incident/incdir_ /xx_j015_63472_i trc Use ADRCI or Support Workbench to package the incident. See Note at My Oracle Support for error and packaging details. 10

11 Tools adrci, opatch und SQL adrci (tfactl) Packagefile aus Incident erzeugen opatch Auslesen der installierten Oracle Versionen und Patches SQL Invalid Objects DB Registry Duplicate Dictionary Objects 11

12 Tools adrci, opatch und SQL Was Tool User Befehl Ergebnis Incidents adrci oracle / grid show incidents Incident home, Incident Liste inc#,. Package Incident adrci oracle / set home <Incident home> pckg# grid ips create package incident <inc#> adrci oracle / ips generate package <pckg#> in /tmp Dateinamen zum Upload grid Oracle Versionen opatch oracle / grid opatch lsinventory -detail > /tmp/opatch.lst Dateinamen zum Upload Datenbank Informationen sqlplus oracle spool /tmp/invalidobj_registry.log zum Upload hcheck.sql hcheck.sql Tracefile zum Upload Service Request erstellen Browser SR Nummer 12

13 adrci: Daten Sammeln zum Incident adrci> show incidents ADR Home = /u01/app/oracle/diag/rdbms/<sid1>/<unique name1>: ************************************************************************* 0 rows fetched ADR Home = /u01/app/oracle/diag/rdbms/<sid2>/<unique name2>: ************************************************************************* INCIDENT_ID PROBLEM_KEY CREATE_TIME ORA 600 [15212] :37: : ORA 600 [kxspoac: zero bct] :15: : ORA 600 [kxspoac: zero bct] :15: : ORA 600 [kxspoac: zero bct] :45: : ORA 600 [kxspoac: zero bct] :46: : ORA 600 [kxspoac: zero bct] :48: :00 ADR Home = /u01/app/oracle/diag/rdbms/<sid3>/<unique name3>: ************************************************************************* 0 rows fetched 13

14 adrci: Daten Sammeln zum Incident adrci> set home diag/rdbms/<sid2>/<unique name2> adrci> ips create package incident Created package 10 based on incident id , correlation level typical adrci> ips generate package 10 in /tmp Generated package 10 in file /tmp/ora600kxs_ _com_1.zip, mode complete adrci> ips show package adrci> ips show files package 10 14

15 adrci: Was haben wir gesammelt?? adrci> ips show package 5 DETAILS FOR PACKAGE 5: PACKAGE_ID 5 PACKAGE_NAME ORA600dbg_ PACKAGE_DESCRIPTION DRIVING_PROBLEM 38 DRIVING_PROBLEM_KEY ORA 600 [dbgtb:iter_max_recs] DRIVING_INCIDENT DRIVING_INCIDENT_TIME :57: :00 STATUS Finalized (2) CORRELATION_LEVEL Basic (1) PROBLEMS 1 main problems, 4 correlated problems INCIDENTS 1 main incidents, 5 correlated incidents INCLUDED_FILES 51 SEQUENCES Last 0, last full 0, last base 0 UNPACKED FALSE CREATE_TIME :26: :00 UPDATE_TIME :26: :00 BEGIN_TIME N/A END_TIME N/A FLAGS 0 15

16 adrci: Was haben wir gesammelt?? adrci> ips show files package 5 FILE_ID 1 FILE_LOCATION <ADR_HOME>/incident/incdir_ FILE_NAME <sid>_m003_60010_i _a.trm LAST_SEQUENCE 0 EXCLUDE Included FILE_ID 137 FILE_LOCATION <ADR_HOME>/incpkg/pkg_5/seq_1/opatch FILE_NAME opatch.xml LAST_SEQUENCE 0 EXCLUDE Included FILE_ID 138 FILE_LOCATION <ADR_HOME>/incpkg/pkg_5/seq_1/opatch FILE_NAME opatch.log LAST_SEQUENCE 0 EXCLUDE Included adrci> 16

17 opatch: Versionen auslesen opatch lsinventory -detail > /tmp/opatch.lst Oracle Interim Patch Installer version Copyright (c) 2018, Oracle Corporation. All rights reserved Installed Top-level Products (1): Oracle Database 18c There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home OPatch succeeded. 17

18 sqlplus : Registry und Invalid Objects auslesen: spool /tmp/invalidobj_registry.log set pages 80 set linesize 1000 column "OWNER.OBJECT" format a62 SELECT SUBSTR(comp_id,1,15) comp_id, status, SUBSTR(version,1,10) version,substr(comp_name,1,30) comp_name FROM dba_registry ORDER BY 1; SELECT status, object_id, object_type, owner '.' object_name OWNER.OBJECT FROM dba_objects WHERE status!= 'VALID' ORDER BY 4,2; select object_name, object_type,owner from dba_objects where object_name object_type in (select object_name object_type from dba_objects where owner = 'SYS') and owner = 'SYSTEM'; 18

19 sqlplus: Zusätzliche Daten / MOS Script Run the hcheck script as described in "hcheck.sql" script to check for known problems in Oracle8i, Oracle9i, Oracle10g and Oracle 11g [ID ] Da wird ein Helper Package hout fehlen: Script to Install the "hout" Helper Package ("hout.sql") (Doc ID Found 0 potential problem(s) and 2 warning(s) Contact Oracle Support with the output and trace file to check if the above needs attention or not PL/SQL procedure successfully completed. Statement processed. Complete output is in trace file: /u01/app/oracle/diag/rdbms/<sid2>/<unique name2>/trace/<sid2>_ora_ _hcheck.trc 19

20 Zusammenfassung Create SR mit gesammelten Daten Incidentfile aus adrci oder tfactl Versionen in Opatch.lst Invalid Objects und DB Registry in invalidobj_registry.log Allgemeine DB Daten als Output von hcheck.sql Optional: Eigene Daten Optional: AWR oder ASH Report 20

21 tfactl: Daten Sammeln zum Incident Sammeln mit TFA DiagCollect 4 Stunden vor und nach dem Incident: tfactl diagcollect -from "MAR/23/ :00:00" -to "MAR/24/ :00:00" Sammeln für Service Requests (SRDC: Service Request Diagnostic Collection): tfactl diagcollect -srdc <srdc_type> [ sr <sr_number>] tfactl diagcollect srdc ORA sr

22 tfactl: Daten Sammeln zum Incident tfactl ips show incidents Multiple ADR basepaths were found, please select one... ( ) option[0] /u01/app/grid ( ) option[1] /u01/app/oracle Pls select an ADR basepath [0..1]?1 /u01/app/oracle was selected INCIDENT_ID PROBLEM_KEY CREATE_TIME ORA 7445 [kghfnd_in_free_lists] :56: : ORA 7445 [kghfrempty_sh_all_depth] :56: : ORA 7445 [kghfrempty_sh_all_depth] :57: : ORA 600 [dbgtb:iter_max_recs] :57: :00 22

23 tfactl: Daten Sammeln zum Incident export LANG=en_US.UTF8 tfactl diagcollect -srdc ora600 Enter the time of the ORA [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] : Enter the Database Name [<RETURN>=ALL] : 1. Mar/12/ :57:14 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [dbgtb:iter_ 2. Mar/12/ :55:51 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [17147], [0x 3. Mar/12/ :55:51 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [dbgtb:iter_ 4. Mar/12/ :55:36 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [17147], [0x 5. Feb/27/ :11:35 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [qcspqb: nxt 6. Feb/27/ :07:16 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [qcspqb: nxt 7. Feb/27/ :06:53 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [qcspqb: nxt 8. Feb/27/ :06:14 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [qcspqb: nxt 9. Feb/27/ :06:11 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [qcspqb: nxt 10. Feb/27/ :51:13 : [<host>] ORA-00600: Interner Fehlercode, Argumente: [qcspqb: nx Please choose the event : 1-10 [1] 10 Selected value is : 10 ( Feb/27/ :51:13 ) Scripts to be run by this srdc: ipspack rdahcve1210 rdahcve1120 rdahcve1110 Components included in this srdc: OS CRS DATABASE NOCHMOS Collecting data for local node(s) 23

24 tfactl: Daten Sammeln zum Incident export LANG=en_US.UTF8 tfactl diagcollect -ips -adrbasepath /u01/app/oracle -incident Collecting data for incident Trying ADR basepath /u01/app/oracle Submitting request to generate package for ADR homepath../diag/rdbms/<sid>/<unique_name> Master package completed for ADR homepath../diag/rdbms/<sid>/<unique_name> Created package 5 based on incident id , correlation level basic Collection Id : oda-xx Detailed Logging at : /../diagcollect_ _<host>.log 2018/03/24 08:26:44 CET : NOTE : 2018/03/24 08:27:01 CET : Completed Local Collection Collection Summary Host Status Size Time oda-xx Completed 19MB 17s ' ' Logs are being collected to: /u01/../<host>.tfa_sat_mar_24_08_26_23_cet_2018.zip 24

25 tfactl: Daten Sammeln zum Incident export LANG=en_US.UTF8 tfactl diagcollect -from " :00:00" -to " :00:00 Collecting data for all nodes Trying to use ADR homepath diag/rdbms/prod/prod... Submitting request to generate package for ADR homepath /u01/app/oracle/diag/rdbms/prod/prod Created package 2 based on time range :00: :00 to :00: :00, correlation level basic 2018/03/22 08:57:43 CET : Collection Name : tfa_thu_mar_22_08_57_38_cet_2018.zip 2018/03/22 08:57:44 CET : Collecting diagnostics from hosts : [prod] 2018/03/22 08:57:56 CET : Collecting ADR incident files /03/22 08:58:11 CET : Completed collection of additional diagnostic information /03/22 08:58:11 CET : Completed Local Collection Collection Summary Host Status Size Time prod Completed 3.1MB 27s ' ' /u01/app/oracle/tfa/repository/collection_thu_mar_22_08_57_38_cet_2018_node_all/ prod.tfa_thu_mar_22_08_57_38_cet_2018.zip 25

26 Links SRDC Data Collection for MOS Doc ID TFA Collector - TFA with Database Support Tools Bundle (Doc ID ) Autonomous Health Framework User s Guide: ORA-4031 Troubleshooting Tool (Doc ID ) ORA-600/ORA-7445/ORA-700 Error Look-up Tool (Doc ID ) adrci

27 DOAG Datenbank 14. und 15. Mai Düsseldorf DOAG Exaday 18. Juni Frankfurt Ernst Leber Telefon: ernst.leber@mt-ag.com 27