Differences between revisions 1 and 24 (spanning 23 versions)
⇤ ← Revision 1 as of 2006-10-24 08:22:01 →
Size: 25
Editor: EkkehardWieprecht
Comment:
← Revision 24 as of 2009-07-15 14:32:38 → ⇥
Size: 2734
Editor: localhost
Comment: converted to 1.6 markup
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= Every Morning check = = Every Morning checks =

== PACSSCOS ==

S2K, Event monitor and CDMS sim. diskspaces
{{{
ssh -X oper@pacsscos
df
}}}

== PACS1 - PACS6 (at present only for PACS3 & PACS4 & Pacs6) ==
Check PCSS version.
{{{
ssh -X pcss@pacsx
cat software/pcss/RELEASE
}}}

if not up-to-date:
{{{
cd software
./fetchpcss
}}}

== PACS 5 ==
=== Is TmIngestion still running ? ===
{{{
  ssh -X oper@pacs5
  ps -elf | grep TmIngestion
}}}
if not:
{{{
  psetup
  TmIngestion [start|stop|status|log]
}}}


=== TC History and OOL Ingestion running ? (at present not because of deadlock) ===
{{{
> ssh -X oper@pacs5
> ps -elf | grep thfser
> ps -elf | grep tchi
> ps -elf | grep orf
> ps -elf | grep ool
}}}

=== Check TmSourcePackets arrive and get propagated ===
{{{
  ssh -X oper@pacs5
  psetup
  db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total
  sleep 20
  db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total
}}}
Check for an increasing number of objects.


=== Check disk space and space of the operational Database ===
{{{
ssh -X dbsa@pacs5
df
psetup
dbtool -space -volume -all pacs_fm_ilt_3
}}}

if less than 20% free:
{{{
oscp -i
ls -als /home/db/versant/pacs_fm_ilt_3
(not during tests)
addvol -n extraspace_ -p /home/db/versant/pacs_fm_ilt_3/extraspace_ -s 2047M pacs_fm_ilt_3
}}}


=== Database Backup done ? ===
{{{
ssh -X dbsa@pacs5
ls /home/db/dbBck
}}}


== PACS5 ==

=== Check Disk space and space of the propagated database ===
{{{
ssh -X oper@pacs5
df
psetup
dbtool -space -volume -all pacs_fm_ilt_3_prop
}}}

== Check daily and hourly TmPacketRecorder files ==
{{{
  ssh -X oper@pacs1
  ps -elf | grep PacketRecorder
  E.g for 01-Mar-2007 :
  ls /dbBck/PacketFiles/*20070301*
}}}
restart if necessary:
{{{
  psetup
  PacketRecorder -host pacs-sv -daily /dbBck/PacketFiles/fm%.tm < /dev/null >& /dev/null &
  PacketRecorder -host pacs-sv -hourly /dbBck/PacketFiles/fm_%.tm < /dev/null > & /dev/null &
}}}
   
== Check whether Hardware support Monitor runs ==
{{{
ssh -X oper@pacs1
more /home/jakob/FMMonitor.txt
}}}

If not :
{{{
ssh jakob@pacs1
psetup
cd monitor
./startMonitor >& /dev/null &
ps -elf | grep writeHk
}}}


= Monday Morning checks =

== PACS2 ==

=== QLA running ? ===

= PACS-SV =

== Check disk space ==
{{{
ssh oper@pacs-sv
df -h .
}}}
To recover disk space try to delete "old" TM/TC packet files.
{{{
ssh oper@pacs-sv
cd packetArchive

find . -mtime +10 | xargs rm # to delete files older than 10 days
# or
find . -mtime +7| xargs rm # to delete files older than one week
}}}

Every Morning checks

PACSSCOS

S2K, Event monitor and CDMS sim. diskspaces 

ssh -X oper@pacsscos
df

PACS1 - PACS6 (at present only for PACS3 & PACS4 & Pacs6)

Check PCSS version. 

ssh -X pcss@pacsx
cat software/pcss/RELEASE

if not up-to-date: 

cd software
./fetchpcss

PACS 5

Is TmIngestion still running ?

  ssh -X oper@pacs5
  ps -elf | grep TmIngestion

if not: 

  psetup
  TmIngestion [start|stop|status|log]

TC History and OOL Ingestion running ? (at present not because of deadlock)

> ssh -X oper@pacs5
> ps -elf | grep thfser
> ps -elf | grep tchi
> ps -elf | grep orf
> ps -elf | grep ool

Check TmSourcePackets arrive and get propagated

  ssh -X oper@pacs5
  psetup
  db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total
  sleep 20
  db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total

Check for an increasing number of objects.

Check disk space and space of the operational Database

ssh -X dbsa@pacs5
df
psetup
dbtool -space -volume -all pacs_fm_ilt_3

if less than 20% free: 

oscp -i
ls -als /home/db/versant/pacs_fm_ilt_3
(not during tests)
addvol -n extraspace_ -p /home/db/versant/pacs_fm_ilt_3/extraspace_ -s 2047M pacs_fm_ilt_3

Database Backup done ?

ssh -X dbsa@pacs5
ls /home/db/dbBck

PACS5

Check Disk space and space of the propagated database

ssh -X oper@pacs5
df
psetup
dbtool -space -volume -all pacs_fm_ilt_3_prop

Check daily and hourly TmPacketRecorder files

  ssh -X oper@pacs1
  ps -elf | grep PacketRecorder 
  E.g for 01-Mar-2007 :
  ls /dbBck/PacketFiles/*20070301*

restart if necessary: 

  psetup
  PacketRecorder -host pacs-sv -daily /dbBck/PacketFiles/fm%.tm < /dev/null >& /dev/null  &
  PacketRecorder -host pacs-sv -hourly /dbBck/PacketFiles/fm_%.tm < /dev/null > & /dev/null &

Check whether Hardware support Monitor runs

ssh -X oper@pacs1
more /home/jakob/FMMonitor.txt

If not : 

ssh jakob@pacs1
psetup
cd monitor 
./startMonitor >& /dev/null &
ps -elf | grep writeHk

Monday Morning checks

PACS2

QLA running ?

PACS-SV

Check disk space

ssh oper@pacs-sv
df -h .

To recover disk space try to delete "old" TM/TC packet files. 

ssh oper@pacs-sv
cd packetArchive

find . -mtime +10 | xargs rm        # to delete files older than 10 days
# or
find . -mtime +7| xargs rm          # to delete files older than one week

Herschel: PACS/FM_ILT_Procedures/RoutineChecks (last edited 2009-07-15 14:32:38 by localhost)