= Every Morning checks = == PACSSCOS == S2K, Event monitor and CDMS sim. diskspaces {{{ ssh -X oper@pacsscos df }}} == PACS1 - PACS6 (at present only for PACS3 & PACS4 & Pacs6) == Check PCSS version. {{{ ssh -X pcss@pacsx cat software/pcss/RELEASE }}} if not up-to-date: {{{ cd software ./fetchpcss }}} == PACS 5 == === Is TmIngestion still running ? === {{{ ssh -X oper@pacs5 ps -elf | grep TmIngestion }}} if not: {{{ psetup TmIngestion [start|stop|status|log] }}} === TC History and OOL Ingestion running ? (at present not because of deadlock) === {{{ > ssh -X oper@pacs5 > ps -elf | grep thfser > ps -elf | grep tchi > ps -elf | grep orf > ps -elf | grep ool }}} === Check TmSourcePackets arrive and get propagated === {{{ ssh -X oper@pacs5 psetup db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total sleep 20 db2tty -d pacs_fm_ilt_3 herschel.versant.ccm.TmSourcePacketImpl | grep Total }}} Check for an increasing number of objects. === Check disk space and space of the operational Database === {{{ ssh -X dbsa@pacs5 df psetup dbtool -space -volume -all pacs_fm_ilt_3 }}} if less than 20% free: {{{ oscp -i ls -als /home/db/versant/pacs_fm_ilt_3 (not during tests) addvol -n extraspace_ -p /home/db/versant/pacs_fm_ilt_3/extraspace_ -s 2047M pacs_fm_ilt_3 }}} === Database Backup done ? === {{{ ssh -X dbsa@pacs5 ls /home/db/dbBck }}} == PACS5 == === Check Disk space and space of the propagated database === {{{ ssh -X oper@pacs5 df psetup dbtool -space -volume -all pacs_fm_ilt_3_prop }}} == Check daily and hourly TmPacketRecorder files == {{{ ssh -X oper@pacs1 ps -elf | grep PacketRecorder E.g for 01-Mar-2007 : ls /dbBck/PacketFiles/*20070301* }}} restart if necessary: {{{ psetup PacketRecorder -host pacs-sv -daily /dbBck/PacketFiles/fm%.tm < /dev/null >& /dev/null & PacketRecorder -host pacs-sv -hourly /dbBck/PacketFiles/fm_%.tm < /dev/null > & /dev/null & }}} == Check whether Hardware support Monitor runs == {{{ ssh -X oper@pacs1 more /home/jakob/FMMonitor.txt }}} If not : {{{ ssh jakob@pacs1 psetup cd monitor ./startMonitor >& /dev/null & ps -elf | grep writeHk }}} = Monday Morning checks = == PACS2 == === QLA running ? === = PACS-SV = == Check disk space == {{{ ssh oper@pacs-sv df -h . }}} To recover disk space try to delete "old" TM/TC packet files. {{{ ssh oper@pacs-sv cd packetArchive find . -mtime +10 | xargs rm # to delete files older than 10 days # or find . -mtime +7| xargs rm # to delete files older than one week }}}