|
Size: 1245
Comment:
|
← Revision 24 as of 2009-07-15 14:32:38 ⇥
Size: 2734
Comment: converted to 1.6 markup
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| == Is TmIngestion still running ? == | == PACSSCOS == S2K, Event monitor and CDMS sim. diskspaces {{{ ssh -X oper@pacsscos df }}} == PACS1 - PACS6 (at present only for PACS3 & PACS4 & Pacs6) == Check PCSS version. {{{ ssh -X pcss@pacsx cat software/pcss/RELEASE }}} if not up-to-date: {{{ cd software ./fetchpcss }}} == PACS 5 == === Is TmIngestion still running ? === |
| Line 8: | Line 30: |
If it is not running type {{{ restartTmIngestion |
if not: {{{ psetup TmIngestion [start|stop|status|log] |
| Line 14: | Line 36: |
| == Check TmSourcePackets arrive and get propagated == | === TC History and OOL Ingestion running ? (at present not because of deadlock) === {{{ > ssh -X oper@pacs5 > ps -elf | grep thfser > ps -elf | grep tchi > ps -elf | grep orf > ps -elf | grep ool }}} === Check TmSourcePackets arrive and get propagated === |
| Line 18: | Line 50: |
| db2tty -d pacs_fm_ilt_1 | grep TmsourcePackets ssh -X oper@pacs1 psetup db2tty -d pacs_fm_ilt_1_prop | grep TmsourcePackets |
db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total sleep 20 db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total }}} Check for an increasing number of objects. === Check disk space and space of the operational Database === {{{ ssh -X dbsa@pacs5 df psetup dbtool -space -volume -all pacs_fm_ilt_3 |
| Line 24: | Line 65: |
| == Check disk space and space of the Database == | if less than 20% free: {{{ oscp -i ls -als /home/db/versant/pacs_fm_ilt_3 (not during tests) addvol -n extraspace_ -p /home/db/versant/pacs_fm_ilt_3/extraspace_ -s 2047M pacs_fm_ilt_3 }}} === Database Backup done ? === {{{ ssh -X dbsa@pacs5 ls /home/db/dbBck }}} == PACS5 == === Check Disk space and space of the propagated database === |
| Line 27: | Line 86: |
| df | |
| Line 28: | Line 88: |
| dbtool -space -volume -all pacs_fm_ilt_1 | dbtool -space -volume -all pacs_fm_ilt_3_prop |
| Line 30: | Line 90: |
If necessary expand the size : {{{ addvol -n extraspace -p /data/versant/db/pacs_fm_ilt_1/extraspace -s 2047M pacs_fm_ilt_1 }}} {{{ ssh -X oper@pacs1 psetup dbtool -space -volume -all pacs_fm_ilt_1_prop }}} If necessary expand the size : {{{ addvol -n extraspace -p /data/versant/db/pacs_fm_ilt_1_prop/extraspace -s 2047M pacs_fm_ilt_1_prop }}} |
|
| Line 51: | Line 94: |
| ps -elf | grep TmPacketRecorder | ps -elf | grep PacketRecorder E.g for 01-Mar-2007 : ls /dbBck/PacketFiles/*20070301* }}} restart if necessary: {{{ psetup PacketRecorder -host pacs-sv -daily /dbBck/PacketFiles/fm%.tm < /dev/null >& /dev/null & PacketRecorder -host pacs-sv -hourly /dbBck/PacketFiles/fm_%.tm < /dev/null > & /dev/null & |
| Line 57: | Line 108: |
| cat /home/jakob/TMMonitor.txt | more /home/jakob/FMMonitor.txt |
| Line 62: | Line 113: |
| ssh -X jakob@pacs1 | ssh jakob@pacs1 psetup |
| Line 64: | Line 116: |
| jylaunch writeHkNrt.py | ./startMonitor >& /dev/null & ps -elf | grep writeHk |
| Line 66: | Line 119: |
= Monday Morning checks = == PACS2 == === QLA running ? === = PACS-SV = == Check disk space == {{{ ssh oper@pacs-sv df -h . }}} To recover disk space try to delete "old" TM/TC packet files. {{{ ssh oper@pacs-sv cd packetArchive find . -mtime +10 | xargs rm # to delete files older than 10 days # or find . -mtime +7| xargs rm # to delete files older than one week }}} |
Every Morning checks
PACSSCOS
S2K, Event monitor and CDMS sim. diskspaces
ssh -X oper@pacsscos df
PACS1 - PACS6 (at present only for PACS3 & PACS4 & Pacs6)
Check PCSS version.
ssh -X pcss@pacsx cat software/pcss/RELEASE
if not up-to-date:
cd software ./fetchpcss
PACS 5
Is TmIngestion still running ?
ssh -X oper@pacs5 ps -elf | grep TmIngestion
if not:
psetup TmIngestion [start|stop|status|log]
TC History and OOL Ingestion running ? (at present not because of deadlock)
> ssh -X oper@pacs5 > ps -elf | grep thfser > ps -elf | grep tchi > ps -elf | grep orf > ps -elf | grep ool
Check TmSourcePackets arrive and get propagated
ssh -X oper@pacs5 psetup db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total sleep 20 db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total
Check for an increasing number of objects.
Check disk space and space of the operational Database
ssh -X dbsa@pacs5 df psetup dbtool -space -volume -all pacs_fm_ilt_3
if less than 20% free:
oscp -i ls -als /home/db/versant/pacs_fm_ilt_3 (not during tests) addvol -n extraspace_ -p /home/db/versant/pacs_fm_ilt_3/extraspace_ -s 2047M pacs_fm_ilt_3
Database Backup done ?
ssh -X dbsa@pacs5 ls /home/db/dbBck
PACS5
Check Disk space and space of the propagated database
ssh -X oper@pacs5 df psetup dbtool -space -volume -all pacs_fm_ilt_3_prop
Check daily and hourly TmPacketRecorder files
ssh -X oper@pacs1 ps -elf | grep PacketRecorder E.g for 01-Mar-2007 : ls /dbBck/PacketFiles/*20070301*
restart if necessary:
psetup PacketRecorder -host pacs-sv -daily /dbBck/PacketFiles/fm%.tm < /dev/null >& /dev/null & PacketRecorder -host pacs-sv -hourly /dbBck/PacketFiles/fm_%.tm < /dev/null > & /dev/null &
Check whether Hardware support Monitor runs
ssh -X oper@pacs1 more /home/jakob/FMMonitor.txt
If not :
ssh jakob@pacs1 psetup cd monitor ./startMonitor >& /dev/null & ps -elf | grep writeHk
Monday Morning checks
PACS2
QLA running ?
PACS-SV
Check disk space
ssh oper@pacs-sv df -h .
To recover disk space try to delete "old" TM/TC packet files.
ssh oper@pacs-sv cd packetArchive find . -mtime +10 | xargs rm # to delete files older than 10 days # or find . -mtime +7| xargs rm # to delete files older than one week