DKRZ Datenhierarchie Von der Antragstellung bis zur Archivierung

Größe: px

Ab Seite anzeigen:

Download "DKRZ Datenhierarchie Von der Antragstellung bis zur Archivierung"

Maya Bäcker
vor 8 Jahren
Abrufe

1 DKRZ Datenhierarchie Von der Antragstellung bis zur Archivierung 07-May-2014 Hannes Thiemann Deutsches Klimarechenzentrum (DKRZ) Datenmanagement

2 Antrag Ressourcen Archivierung + Zugriff Support LiveDemo

3 Antragsphase (I) Das DKRZ stellt unentgeltlich Ressourcen (Rechenzeit, Archivkapazität, Fachberatung) für Wissenschaftler/innen zur Verfügung, die auf dem Gebiet der Erdsystemforschung tätig sind, und für ihre Arbeiten auf Höchstleistungs-DV- Ressourcen angewiesen sind. Die Ressourcen werden zu gleichen Teilen auf zwei Kontingente verteilt Freies Kontingent, verwaltet vom WLA Gesellschafter Kontingent

tätig sind, und für ihre Arbeiten auf Höchstleistungs-DV- Ressourcen angewiesen sind.

4 Antragsphase (II) Anträge auf Rechenzeit müssen bis zum 31. Oktober des vorherigen Jahres eingehen. Wenn Sie Ressourcen aus dem Gesellschafter-Kontingent nutzen wollen, melden Sie in Absprache mit Ihrer Institutsleitung bzw. dem/der EDV- Verantwortlichen Ihres Institutes ein HLRE-Projekt an.

Wenn Sie Ressourcen aus dem Gesellschafter-Kontingent nutzen wollen,

5 DKRZ Projects HOME /pf /scratch /work E S G F Research Data Environment ESGF Disseminated results: new questions, new data producing projects. Production Selection and postprocessing Dissemination During Project CERA Dissemination DOI publication CERA MD + GUI..doku Container files..arch..double Standard Archiving At the end of blocking period W D C C Long Term Archive Environment Long-Term Archiving Selection for 10 years storing

Production Selection and postprocessing Dissemination During Project CERA Dissemination DOI publication CERA

6 DKRZ Projects HOME /pf /scratch /work Production

7 Data resources (Disk) Diskspace (GPFS) HOME /pf < 8 GB (per User) for source codes, Backed up regularly SCRATCH /scratch <15 TB (per User) no backup old data automatically removed, granted period is 14 days WORK /work project based, quota applies no backup no data removed by DKRZ

no backup old data automatically removed, granted period is 14 days

8 DKRZ Projects HOME /pf /scratch /work E S G F Research Data Environment ESGF Disseminated results: new questions, new data producing projects. Production Selection and postprocessing Dissemination During Project CERA Dissemination DOI publication CERA MD + GUI..doku Container files..arch..double Standard Archiving At the end of blocking period W D C C Long Term Archive Environment Long-Term Archiving Selection for 10 years storing

9 ..arch Standard Archiving..double

10 Data resources (Tape / HSM system) ARCH DOUBLE /hpss/arch /hpss/double Single copy on tape 2 copies on tapes in different fire compartments Size GB recommended Data removed one year after project has expired One quota for ARCH and DOUBLE

fire compartments Size 20-50 GB recommended Data removed one

11 DKRZ Projects HOME /pf /scratch /work E S G F Research Data Environment ESGF Disseminated results: new questions, new data producing projects. Production Selection and postprocessing Dissemination During Project CERA Dissemination DOI publication CERA MD + GUI..doku Container files..arch..double Standard Archiving At the end of blocking period W D C C Long Term Archive Environment Long-Term Archiving Selection for 10 years storing

12 CERA CERA MD + GUI..doku Container files W D C C Long Term Archive Environment Long-Term Archiving

13 Long-Term Archiving / Doku Service for DKRZ Projects Data moved from /arch to /doku By default to be removed 10 years after project expiration Two copies in different fire compartments Must not be changed Accessible by all registered users of DKRZ and WDCC Persistent CERA-Link Metadata needed (basic set) Project description Contact Filelist

fire compartments Must not be changed Accessible by all registered users of DKRZ and

14 Long-Term Archiving / Doku / Filelist Filelist available at /hpss/arch/<prj>/_project.<prj>.filelist.giga Example -rw-r :1076 Mar 11 08: giga /hpss/arch/bm0146/k204026/ceop2/ceop2_ncep_grid_2013.tar -rw-r :1076 Jan 24 08: giga /hpss/arch/bm0146/k204026/ceop2/ceop2_ncep_molts_raw_2008.tar

giga Example -rw-r----- 426:1076 Mar 11 08:32 188011141120 189 giga

15 DOKU: addressing data Aus /hpss/arch/bm0146/k204026/ceop2/ceop2_ncep_grid_2013.tar wird /hpss/doku/bm0146/dkrz_lta_146_ds00001 /k204026/ceop2/ceop2_ncep_grid_2013.tar User <tape> <arch> <dir> <file> Tape /hpss/arch/bm0146 k204026/ceop2 CEOP2_NCEP_GRID _2013.tar DOKU <tape> <doku> <ds> <dir> <file> Tape /hpss/doku/bm0146 DKRZ_LTA_146_ds k204026/ceop2 CEOP2_NCEP_GR ID_2013.tar

tar User <tape> <arch> <dir> <file> Tape /hpss/arch/bm0146 k204026/ceop2 CEOP2_NCEP_GRID _2013.

16 DOKU: addressing data Metadata =DKRZ_LTA_800_ds00001 CERA download acronym=dkrz_lta_800_ds00001&point erid=

acronym =DKRZ_LTA_800_ds00001 CERA download

17 CERA CERA MD + GUI..doku Container files W D C C Long Term Archive Environment Long-Term Archiving

18 Long-Term Archiving / CERA Two tape copies in different fire compartments Stored within container files Greater safety with regards to tape damage Better utilization of HSM resources Must not be changed, new versions possible Metadata needed complete set Access Embargo possible Access using CERA account (GUI, jblob *, wget) ESGF data available through ESGF + CERA portal (if already archived) Prerequisite for data DataCite publication

versions possible Metadata needed complete set Access Embargo possible Access using CERA account (GUI, jblob

19 Support Main entry for all support enquiries:

20 Important Links Main Entry to CERA List of projects in DOKU Interface to generate metadata Status of CERA services

de/wdcc/doku Interface to generate metadata http://cera-www.

21 Poster Lobster CERA Download B2FIND / EUDAT

22 Live Demo - > Hermann Eintragen von Metadaten: Archivierung der Daten unter 1.) /hpss/doku (zur fachinternen Nutzung) i.d.r. Speicherung von.tar Files 2.) CERA Container (für interdisziplinäre Nachnutzung) i.d.r Zeitreihen Zus. Absprache/Abstimmungen für 2.)

Ähnliche Dokumente

Die Repositorien-Infrastruktur des Deutschen Klimarechenzentrums

Die Repositorien-Infrastruktur des Deutschen Klimarechenzentrums DINI/nestor Workshop "Forschungsdatenrepositorien" 27.11.2017 Hannes Thiemann Deutsches Klimarechenzentrum (DKRZ) DKRZ a brief overview