close to live but customized nevertheless buildup of test-environments

Transkript

1 close to live but customized nevertheless buildup of test-environments DOAG Alexander Pilfusek IB-BS

2 software witt Josef Witt GmbH individual software is developed by ourself over 100 software products over 50 software developers (PL/SQL, Java, PowerBuilder, Web,..) data is stored in an Oracle-RAC OLTP database one single schema for all a company-wide data model changes in PL/SQL developer data model user DB changes on data model architects 2

3 software witt seven databases for tests & development one RAC-database for integration all are snaps / clones of live-system at the beginning there was just a single persistent DB and one SQL-file for DDLs problems: were there errors? what kind of errors? live test A who requested what when? development integration test B test C continuous integration test&release management DBA-DB 3

4 TED TEstDatenbases - BuildUp PL/SQL shellscripts TED data model 4

5 TED data model 5

6 Process of a TED-Buildup 6

7 Preparation 7

8 Preparation TED_RESTART_DB stops the database (immediate) starts it in restricted-mode à avoids lock-situations due to long-time sql-statements srvctl stop database d ${dbid_lower} >> $protokoll_datei sleep 20 srvctl start database d ${dbid_lower} o restrict >> $protokoll_datei checks the status via srvctl as well: erg=$(srvctl status database -d ${dbid_lower}) if (! test "$erg" = "Database is running.") then in case of RAC it will be checked, if all nodes are up 8

9 Preparation TED_TTSEXPORTVORBEREITEN individualisation.. exempted from a buildup can be: tables table statistics schemas implementation of exceptions with transportable tablespace objects in a TTS must be self contained, which means no dependencies in and out of the TTS are allowed TTS Data test-db test-db cloning 9

10 Preparation TED_TTSEXPORTVORBEREITEN individualisation.. creates the TED-snapshot-tables to preserve data integrity exports the table-statistics, the STATTAB is on TTS dbms_stats.export_table_stats( ownname => rec.tableowner, tabname => rec.tablename, partname => null, stattab => TED_STATSTABLE, cascade => true, statown => TED_STATSTABLEOWNER ); drops constraints from the tables to remain on test-database runs the self-contained-check: sys.dbms_tts.transport_set_check( TTS_NAME, TRUE ); moves the tables to remain into the transportable tablespace moves the tables, which should not remain any longer, from the TTS into another tablespace 10

11 Preparation TED_TTSEXPORTIEREN individualisation.. exports TTS via expdp: expdp dumpfile=${dumpdatei} \ directory=daten_dir \ transport_tablespaces=tts \ logfile=${se_log} << EOT $WWDBA_PASS EOT Only the tablespace s metadata are exported. The data persists in the datafile and is not touched. The TTS-datafile remains in ASM of the test-database. 11

12 Preparation TED_SONDER_IMPEXP_EXP individualisation.. saves objects, which cannot be transported via TTS: schemas sequences special tables (e. g. ID-table, TED-snapshot-tables ) exports schemas via DBMS_DATAPUMP, the names of the schemas to backup are kept in the TED-tables reads metadata for sequences out of the data dictionary und writes it into a SQLspool-file, which will be executed later; the names of the sequenes are also kept in the TED-tables exports special tables via expdp and parameterfile 12

13 Preparation TED_RMANDATAFILEBACKUP backs up the TTS-Datafiles via RMAN run { backup as copy datafile '$transdatafile TAG TTSDFSicherung FORMAT '$db/$(basename $transdatafile).$(date +%y%m%d_%h%m%s).bak'; backup current controlfile TAG TTSDFSicherungCTL FORMAT '$db/rman_d%d_ctlfile_t%t_s%s_p%p'; } deletes old backups, only three will be kept find. \! -newer $oldest_ctlfile_piece \ -mtime +$keep_backups_daysmin \ -exec rm {} \; 13

14 Preparation TED_STOP stops the database srvctl stop database -d $dbid_lower -o immediate >> $protokoll_datei 2>&1 unmonuts the ASM-diskgroups diskgroups="data redogb for diskgroup in $diskgroups; do asmcmd umount -f ${diskgroup}${dbid} done in case of a RAC, the diskgroups have to be unmounted on all nodes! nodelist=$(olsnodes) 14

15 Cloning / Snapping 15

16 Cloning / Snapping TED_ERZEUGE_SNAP data- and redo-diskgroups are LUNs in our SAN environment makes a binary copy of all disks maintaining the DATA and REDO diskgroups in our case we use storage capabilities and scripting framework emcadm +DATA +DATA +REDO +REDO +FRA 16

17 Cloning / Snapping TED_START mounts the diskgroups in case of multiple test-databases on a single host you need to rename the diskgroups disk header (diskgroup and disk name) must be unique in ASM when disk is mounted you must ensure there is one (and onlyone!) copy of each disk clone of other test systems have to wait until the rename is completed renames datafiles because there are new diskgroup names 17

18 Cloning / Snapping TED_START mounts database and turns off archive and flashback logs because database is in a crash consistent state you have to do an instance recovery (open and cleanly shutdown the database) renames database via nid (database has to be mounted) and opens the database with resetlogs nid target=$sys_pass DBNAME=$dbid_lower << XXX >> $protokoll_datei Y XXX sqlplus '/ as sysdba' << XXX >> $protokoll_datei 2>&1 startup mount; alter system enable restricted session; alter database open resetlogs; shutdown immediate; exit; XXX 18

19 Cloning / Snapping TED_PATCH individualisation.. purpose: testing new PSUs usually deactivated oracle home is patched once, because the test-databases run on a dedicated server with an own oracle home execution of post-installation-tasks sqlplus / as sysdba << EOT >> psu apply exit; EOT 19

20 Cloning / Snapping TED_SETUP_AUDIT individualisation.. activates / changes a witt-specific audit environment e. g. applicable for data masking 20

21 Postprocessing 21

22 Postprocessing TED_INSTALL_FRAMEWORKS individualisation.. installs frameworks, which aren t available on live-database (yet) calls install-scripts either self-written or provided by a manufacturer # PL/JSON installieren echo `date +%H:%M:%S`" Start Install PLJSON" >> $protokoll_datei /pljson/install.sh pljson/json_v1_0_4" pljson Js0n >> $protokoll_datei 22

23 Postprocessing TED_SONDER_IMPEXP_IMP individualisation.. executes the outputs of job TED_SONDER_IMPEXP_EXP dumps (impdp) SQL-spool-files (e. g. sequences, user) objects of old test-database are now in place (again) dumpfile TED_SONDER_IMPEXP_IMP TED_SONDER_IMPEXP_EXP sqlfile test-db 23

24 Postprocessing BUILD_TED_OBJEKT committal of PL/SQL-objects, which relate to the TED-buildup next jobs, espcially TED_DDL, are implemented in PL/SQL because of this extra-committal of these packages tests in single TED-buildups are possible alternative: committal to live-database, but this would affect all buildups 24

25 Postprocessing TED_DDL individualisation.. applies changes on data model in the test-database core of TED reads DDL-statements from TED-tables and executes these per execute immediate writes feedback into the TED-tables DDL-statements are set up over a PL/SQL-interface interface is called by modelling-tool before grants are executed, there is checking whether they are still necessary 25

26 Postprocessing TED_DDL_FEHLERVERARBEITUNG analyses errors (ORAs) of DDL-statements from job TED_DDL rates errors: good respectively expectable, these statements will be deactivated, e.g. ORA and a DROP or SET UNUSED rec.oracode = 'ORA-00904' and regexp_like( rec.stmtbeginn, 'drop', 'i' ) or regexp_like( rec.stmtbeginn, '(.)*set([[:space:]])+unused', 'i' ) ORA and a DROP rec.oracode = 'ORA-00942' and instr( upper( rec.statementbeginn ), 'DROP' ) > 0) bad respectivley not expectable, these statements will be executed next time again any other error J determines the deletion of whole requests e.g. remaining DDL-statements to execute are comments 26

27 Postprocessing TED_DDL_FEHLERVERARBEITUNG regexp_like( stmt, '(.)*comment([[:space:]])+on(.)*', 'i' ) or regexp_like( stmt, '(.)*create([[:space:]])+or([[:space:]])+replace([[:space:]])+public([[:spac e:]])+synonym(.)*', 'i' ) or regexp_like( stmt, '(.)*pkg_userrechte(.)*', 'i' ) or regexp_like( stmt, '(.)*enable([[:space:]])+novalidate([[:space:]])+constraint(.)*', 'i' ) or regexp_like( stmt, '(.)*modify([[:space:]])+constraint(.)*', 'i' ) or regexp_like( stmt, '(.)*alter([[:space:]])+index(.)*noparallel(.)*', 'i' ) or regexp_like( lower( stmt ), '(.)*revoke([[:space:]])+all(.)*', 'i' ) ) 27

28 Postprocessing BUILD_TEDAUFBAU individualisation.. SVN check-in of the sources PL/SQL, views, etc. BEEPER self-made web-application define status of source test, integration, live BUILD_ TEDAUFBAU determination of versions to build delivery to the database 28

29 Postprocessing TED_TTSIMPORTVORBEREITEN individualisation.. tables, which should be imported, are existing yet renames existing tables: tables constraints trigger saves and drops remaining constraints, which points in/out of the TTS advantages of renaming versus dropping: live-table is still available à clarifications, comparisons are possible faster J TestTable PK_TestTable TRG_TestTable_IU TED_ TestTable TED_PK_TestTable TED_TRG_TestTable_IU 29

30 Postprocessing TED_TTSIMPORTIEREN individualisation.. imports metadata from created dumpfile impdp \ dumpfile=$dumpdatei \ directory=daten_dir \ TRANSPORT_DATAFILES="'$transdatafile' \ logfile=$se_log << EOT $WWDBA_PASS EOT now exported tables incl. data are available 30

31 Postprocessing TED_TTSIMPORTNACHBEREITEN individualisation.. sets TTS-Tablespace to READ WRITE again imports table-statistics dbms_stats.import_table_stats( ownname => rec.tableowner, tabname => rec.tablename, partname => null, stattab => TED_STATSTABLE, cascade => true, statown => TED_STATSTABLEOWNER, force => true ); recreates constraints, which were saved prior DDL-object DBMS_METADATA.getDDL DDL_ SavTable DDL-object 31

32 Postprocessing TED_SYNC_RECHTE PL/SQL-development needs direct rights, rights granted over roles won t work with stored procedures and Definer s Rights rights are granted through roles via this job rights, which are granted through roles, will be granted directly to the user (only PL/SQL-developers) Tab_1 Tab_1 ROLE_IT ROLE_IT 32

33 Postprocessing TED_JOBCTL individualisation.. activates jobs to be tested jobsystem is also running on test-databases possibility of influencing single jobs sql commands on jobsystem s datamodel 33

34 Postprocessing TED_FREIGABE opens the (restricted) database for all users alter system disable restricted session; takes AWR-snapshot EXEC DBMS_WORKLOAD_REPOSITORY.create_snapshot; advantages of restricted session no locks caused by users buildup is on its own disadvantages: no parallel processing via dbms_parallel_execute possible! 34

35 Postprocessing TED_DDL_FEEDBACK_xxx transfer ofted-feedback into live-database thus the feedback is persistent TED_ FEEDBACK TED_ FEEDBACK TED_ FEEDBACK test-databases TED_ FEEDBACK live-database 35

36 Lessons Learned stable buildupis necessary, better a test-system with minor errors (DDL, etc.) than no test-system a own clone / snap for testing the buildup is helpful speed up open reset logs (TED_START) by dropping some of the online redo logs (first writes on snaps are possibly much slower) posibility to run some scripts as sys (or other user) needed all in all: TED is a permanent growing system 36