Differences between revisions 2 and 48 (spanning 46 versions)

PACS - PCSS

Procedure to develop Pipeline Tasks

Select an assigned IA application from the IA Functionality list at the pacs website BR
Check whether you are assigned to develop this application BR
Contact Juergen (to update the website) and Ekki (to have an informed person) BR
Juergen will flag the application as "under construction" in the IA Functionality list BR
Write the Chapter in the User Manual (against this description you will develop) BR
- Covering API (parameter, keywords, etc.) BR
- Algorithm (references if possible) BR
- Examples how the data is processed (before - after, figures) BR
Contact a "scientific expert" to verify that your application is doing what is expected (if not already done earlier) BR
Select and document Testdata (currently in User manual) BR
Write a TestHarness - please contact Rik, Michael or Ekki if you do not know howBR
Write the application (TestHarness may help) BR
Test the application with the TestHarness BR
Check the application into CVS BR
Inform Juergen/Ekki that its finished BR
Juergen will flag the application as done BR
Tag the application BR

Generate User Manual from XML as PDF

cd vs-pcss-src>/pacs/develop/pcss/
checkout the doc-files from the herschel.pacs.doc-files
cd to <cvs-pcss-src>/pacs/develop/pcss/herschel/pacs/doc-files
make you changes in the *.xml files in help/um/xml
make sure the following env variables are set:
- $HCSS_DIR pointing to your hcss distribution
- $PCSS_OUT pointing to your development out directory (in this example $HOME/out)
make a file jake.root
- ```
      hcss.jake.outdir = $PCSS_OUT
```
run the make_um.local from the doc-files directory
- ```
         help/bin/make_um.local pdf 
```
In the moment you might get an error : xception in thread "main" java.lang.OutOfMemoryError: Java heap space
If you get no error
- check the result in your web browser pointing to $HOME/out/pcss/doc/help/um/html/um.html
if you get an error : exception in thread "main" java.lang.OutOfMemoryError: Java heap space
- you find the $HOME/out/doc/help/um/pdf/um.fo file which will need to be converted (with the help of FOP http://xmlgraphics.apache.org/fop/download.html#binary) to PDF
- ```
           java -Xmx3g herschel.ia.document.tools.Fo \
              -fo ${LOCATION_PDF}/${DOC_NAME}.fo ${LOCATION_PDF}/${DOC_NAME}.pdf 
```
- check the result in your web browser pointing to $HOME/out/pcss/doc/help/um/html/um.html

Attention: for the conversion to work properly you have to install a version of dbdoclet.jar. This software comes with its own version of jfreechart. If the dbdoclet.jar is in the classpath before the hcss external libraries, also hcss will use the jfreechart version of dbdoclet. This leads to problems at least with PlotXY and PacsQla. So make sure dbdoclet.jar is added at the end of your classpath!

Usage of TestExecutionBrowser just to get an overview

In JIDE :

from herschel.access.util import *
t=TestExecutionBrowser()

Size of level 0.5 data structures in PCSS

Spectrometer

Typical Spectrometer observation after decompression :

One second of Spectrometer data (default mode/3 raw channels) :

Basic science data + reduction add on (error, wavelength, ra, dec, 8 masks)

Detectors : 25 * 18

Ramp length : 64

Readout frequency : 256 Hz

NrSubRamps : 8

All data as doubles : 8 Byte

Readouts (Ramps) : 25 * 18 * 256/64 * 8 * 8 = 115.2 KB / second

masks: same as readouts/64: 14.4 KB/second

Ramps DecMec data:

Readout frequency : 256 Hz

All data as int : 4 Byte

DecMec entries : 13

256 * 4 * 13 = 13KB / second

Raw data

Readout frequency : 256 Hz

All data as doubles : 8 Byte

Nr raw channels : 3

Raw data : 256 * 8 * 3 = 6 KB /second

data reduced to Frames:

signals (Frames): 25 * 18 * 4 * 8 = 14.4 KB/second

Wavelength : same as above : 14.4 KB /second

Error : same as above : 14.4 KB / second

Ra: same as above: 14.4 KB/second

Dec: same as above: 14.4 KB/second

masks: 14.4 KB/second/64*9 = 2 KB/second

Frames DecMec data:

Readout frequency : 4 Hz

All data as int : 4 Byte

DecMec entries : 20

4 * 4 * 20 = 0.32 KB / second

==> ~ 223 KB per second

One hour :

==> ~ 803 MB per hour

Observation duration <= 3hour (there may be exceptions, but then we reduce the number of user)

==> ~ 2.4 GB per observation

will produce additionally level 1 and 2 level 2 data cubes

==> ~ 9 GB per observation

if you use red and blue spectrometer array, numbers are:

446 KB per second, 1.6 GB per hour, 4.8 GB per level 0.5 observation and 18 GB per level 2 observation (3 hours)

Photometer

Basic science data + reduction add on (error, noise)

Detectors : 64 * 32 = 2048 blue channel

32 * 16 = 512 red channel

Readout frequency : 40 Hz averaged on board to 10Hz All data as doubles : 8 Byte

Reduced data :

Readouts : 64 * 32 * 10 * 8 = 163 KB / second Noise : same as above : 163 KB / second Error : same as above : 163 KB / second

Product Access Layer

The Product Access Layer (PAL) is now available in PCSS. We encourage everybody to use it. Basic documentation you find under :

ftp://ftp.rssd.esa.int/pub/HERSCHEL/csdt/releases/doc/ia/document/um/html/ch12s02.html

A typical example for Frames is :

storage=ProductStorage()
storage.register(SimplePool.getInstance()) 
reference=storage.save(frames)
print reference.urn

from herschel.pacs.signal import *
reference = storage.load("urn:simple.default:herschel.pacs.signal.Frames:1")

#get back your saved product
frame = reference.getProduct()

To use the GUI Browser to find your Products you may use :

from herschel.ia.pal.browser import ProductBrowser
from herschel.ia.pal.browser import ProductBrowserFrame

p=ProductBrowser(storage)
d=ProductBrowserFrame(p)

To get out the Data you need to move it in the Basket (marking in the GUI). Then you may extract it from the basket :

prod=p.getJideBasket().toArray()[0].getProduct()

Work distribution move to new Calibration Framework

---------------------------------------------------------------------------------------------------------
PacsCal_BoloCSs_CQM_1_0.fits                -> PCalPhotometer_CalSources_QM_v1.fits               -EkW -OK 
PacsCal_BoloCSs_FM_1_0.fits                 -> PCalPhotometer_CalSources_FM_v1.fits               -EkW -OK
---------------------------------------------------------------------------------------------------------
PacsCal_BoloFlatFields_CQM_1_0.fits         -> PCalPhotometer_FlatField_QM_v1.fits                -MW -OK
PacsCal_BoloFlatFields_FM_1_0.fits          -> PCalPhotometer_FlatField_FM_v1.fits                -MW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_BoloMasks_CQM_1_0.fits              -> PCalPhotometer_Masks_QM_v1.fits                    -MW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_ChopJitterThreshold_CQM_1_0.fits    -> PCalCommon_ChopperJitterThreshold_QM_v1.fits       -RH
PacsCal_ChopJitterThreshold_FM_1_0.fits     -> PCalCommon_ChopperJitterThreshold_FM_v1.fits       -RH
PacsCal_ChopJitterThreshold_FM_2_0.fits     -> PCalCommon_ChopperJitterThreshold_FM_v2.fits       -RH

spg/CleanPlateauTask.py  -JS- OK
spg/CleanPlateauRampsTask.py -JS -OK
---------------------------------------------------------------------------------------------------------
PacsCal_ChopperAngle_CQM_1_0.fits           -> PCalCommon_ChopperAngle_QM_v1.fits                 -RH
PacsCal_ChopperAngle_FM_1_0.fits            -> PCalCommon_ChopperAngle_FM_v1.fits                 -RH
PacsCal_ChopperAngle_FM_2_0.fits            -> PCalCommon_ChopperAngle_FM_v2.fits                 -RH

---------------------------------------------------------------------------------------------------------
PacsCal_ChopperAngleRedundant_FM_1_0.fits   -> PCalCommon_ChopperAngleRedundant_FM_v1.fits        -JS -OK
PacsCal_ChopperAngleRedundant_FM_2_0.fits   -> PCalCommon_ChopperAngleRedundant_FM_v2.fits        -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_ChopperSkyAngle_CQM_1_0.fits        -> PCalCommon_ChopperSkyAngle_QM_v1.fits              -RH
PacsCal_ChopperSkyAngle_FM_2_0.fits         -> PCalCommon_ChopperSkyAngle_FM_v2.fits              -RH

spg/ConvertChopper2AngleTask.py -JS -OK
common/chopPos2FpuAnglePol -JS -OK
common/fpuAngle2ChopPosPol -JS- OK
common/chopPos2SkyAnglePol -JS-OK
common/skyAngle2ChopPosPol -JS -OK
common/convertChopperAngleSteps -JS-OK
common/convertChopperReadoutSteps -JS -OK
---------------------------------------------------------------------------------------------------------
PacsCal_DiscardRampHooks_CQM_1_0.fits       -> PCalSpectrometer_DiscardRampHooks_QM_v1.fits       -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_EffectiveCapacitance_CQM_1_0.fits   -> PCalSpectrometer_EffectiveCapacitance_QM_v1.fits   -EkW -OK 
PacsCal_EffectiveCapacitance_FM_1_0.fits    -> PCalSpectrometer_EffectiveCapacitance_FM_v1.fits   -EkW -OK

spg/ConvertSignal2StandardCapTask.py - EkW - OK
spec/GetEffectiveCapacitanceTask.py  - JS- OK
---------------------------------------------------------------------------------------------------------
PacsCal_FilterTransmissionB2_CQM_1_0.fits   -> PCalCommon_FilterTransmissionB2_QM_v1.fits         - 
PacsCal_FilterTransmissionB3_CQM_1_0.fits   -> PCalCommon_FilterTransmissionB3_QM_v1.fits         -
PacsCal_FilterTransmissionBL_CQM_1_0.fits   -> PCalCommon_FilterTransmissionBL_QM_v1.fits         -
PacsCal_FilterTransmissionBS_CQM_1_0.fits   -> PCalCommon_FilterTransmissionBS_QM_v1.fits         -
PacsCal_FilterTransmission_FM_1_0.fits      -> PCalCommon_FilterTransmission_FM_v1.fits           -
PacsCal_FilterTransmission_FM_1_1.fits      -> PCalCommon_FilterTransmission_FM_v2.fits           -
PacsCal_FilterTransmission_FM_1_2.fits      -> PCalCommon_FilterTransmission_FM_v3.fits           -
PacsCal_FilterTransmissionR1_CQM_1_0.fits   -> PCalCommon_FilterTransmissionR1_QM_v1.fits         -
PacsCal_FilterTransmissionR_CQM_1_0.fits    -> PCalCommon_FilterTransmissionR_QM_v1.fits          -

---------------------------------------------------------------------------------------------------------
PacsCal_FilterWheel2Band_CQM_1_0.fits       -> PCalCommon_FilterWheel2Band_QM_v1.fits             -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_GPRHall_CQM_2_0.fits                -> PCalSpectrometer_GprHall_QM_v2.fits                -PR

---------------------------------------------------------------------------------------------------------
PacsCal_GratingJitterThreshold_CQM_1_0.fits -> PCalSpectrometer_GratingJitterThreshold_QM_v1.fits -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_KeyWavelength_CQM_1_0.fits          -> PCalSpectrometer_KeyWavelengths_QM_v1.fits         -MW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_LabelDescription_CQM_1_0.fits       -> PCalCommon_LabelDescription_QM_v1.fits             -RH

---------------------------------------------------------------------------------------------------------
PacsCal_LittrowParameters_CQM_1_0.fits      -> PCalSpectrometer_LittrowParameters_QM_v1.fits      -EkW ->BV
PacsCal_LittrowParameters_CQM_2_0.fits      -> PCalSpectrometer_LittrowParameters_QM_v2.fits      -EkW ->BV
PacsCal_LittrowParameters_FM_1_0.fits       -> PCalSpectrometer_LittrowParameters_FM_v1.fits      -EkW ->BV

spg/WaveCalcTask.py       - EkW - OK
spec/gratPos2Wave.py      - EkW - OK
---------------------------------------------------------------------------------------------------------
PacsCal_OBCPDescription_CQM_1_0.fits        -> PCalCommon_ObcpDescription_QM_v1.fits              -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotAbsorption_FM_1_0.fits          -> PCalPhotometer_Absorption_FM_v1.fits               -EkW ->BM
PacsCal_PhotAbsorption_FM_1_1.fits          -> PCalPhotometer_Absorption_FM_v2.fits               -EkW ->BM

---------------------------------------------------------------------------------------------------------
PacsCal_PhotArrayInstrument_CQM_1_0.fits    -> PCalPhotometer_ArrayInstrument_QM_v1.fits          -EkW -OK
PacsCal_PhotArrayInstrument_FM_1_0.fits     -> PCalPhotometer_ArrayInstrument_FM_v1.fits          -EkW -OK
PacsCal_PhotArrayInstrument_FM_2_0.fits     -> PCalPhotometer_ArrayInstrument_FM_v2.fits          -EkW -OK
PacsCal_PhotArrayInstrument_FM_2_1.fits     -> PCalPhotometer_ArrayInstrument_FM_v3.fits          -EkW -OK
herschel.pacs.spg/PhotassignRaDec.java     - EkW - OK
herschel.pacs.spg/PhotAssignRaDecTask.java - EkW - OK
---------------------------------------------------------------------------------------------------------
PacsCal_PhotBadPixelsMask_FM_1_0.fits       -> PCalPhotometer_BadPixelsMask_FM_v1.fits            -MW  -OK
                                                                              Task: toolbox.spg.PhotFlagBadPixelsTask.py                      -MW  -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotCorrZeroLevel_FM_1_0.fits       -> PCalPhotometer_CorrZeroLevel_FM_v1.fits            -MW  -OK
                                                                            Task: toolbox.spg.PhotCorrZeroLevelTask.py                    -MW  -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotCrosstalkMatrix_FM_1_0.fits     -> PCalPhotometer_CrosstalkMatrix_FM_v1.fits          -MW  -OK
                                                                              Task: herschel.pacs.spg.PhotCorrectCrossTalkTask          -MW  -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotGain_CQM_1_0.fits               -> PCalPhotometer_Gain_QM_v1.fits                     -EkW -OK
PacsCal_PhotGain_FM_1_0.fits                -> PCalPhotometer_Gain_FM_v1.fits                     -EkW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotSatLimits_CQM_1_0.fits          -> PCalPhotometer_SatLimits_QM_v1.fits                -EkW -OK
PacsCal_PhotSatLimits_FM_1_0.fits           -> PCalPhotometer_SatLimits_FM_v1.fits                -EkW -OK

spg/PhotFlagSaturationTask.py
---------------------------------------------------------------------------------------------------------
PacsCal_PhotSortMatrix_CQM_1_0.fits         -> PCalPhotometer_DetectorSortMatrix_QM_v1.fits       -MW  -OK
PacsCal_PhotSortMatrix_FM_0_0.fits          -> PCalPhotometer_DetectorSortMatrix_FM_v1.fits       -MW  -OK 
PacsCal_PhotSortMatrix_FM_1_0.fits          -> PCalPhotometer_DetectorSortMatrix_FM_v2.fits       -MW  -OK
PacsCal_PhotSortMatrix_FM_1_1.fits          -> PCalPhotometer_DetectorSortMatrix_FM_v3.fits       -MW  -OK

                                                                        Task: herschel.pacs.spu.DetectorSorting                                  -MW -OK
---------------------------------------------------------------------------------------------------------
PacsCal_RampModel_CQM_1_0.fits              -> PCalSpectrometer_RampModel_QM_v1.fits              -RH

---------------------------------------------------------------------------------------------------------
PacsCal_RsrfB2_CQM_1_0.fits                 -> PCalSpectrometer_RsrfB2_QM_v1.fits                 -RH
PacsCal_RsrfB2_CQM_2_0.fits                 -> PCalSpectrometer_RsrfB2_QM_v2.fits                 -RH
PacsCal_RsrfR1_CQM_1_0.fits                 -> PCalSpectrometer_RsrfR1_QM_v1.fits                 -RH
PacsCal_RsrfR1_CQM_2_0.fits                 -> PCalSpectrometer_RsrfR1_QM_v2.fits                 -RH

---------------------------------------------------------------------------------------------------------
PacsCal_SatLimits_CQM_1_0.fits              -> PCalSpectrometer_RampSatLimits_QM_v1.fits          -EkW -OK 
PacsCal_SatLimits_FM_1_0.fits               -> PCalSpectrometer_RampSatLimits_FM_v1.fits          -EkW -OK

spg/SpecFlagSaturationRampsTask.py  -JS-OK
---------------------------------------------------------------------------------------------------------
PacsCal_Siam_CQM_1_0.fits                   -> PCalCommon_Siam_QM_v1.fits                         -RH

---------------------------------------------------------------------------------------------------------
PacsCal_SpecBadPixelsMask_FM_1_0.fits       -> PCalSpectrometer_BadPixelMask_FM_v1.fits           -JS -OK

spg/SpecFlagBadPixelsTask.py  -JS -OK
---------------------------------------------------------------------------------------------------------
PacsCal_SpecRedPSF_CQM_1_0.fits             -> PCalSpectrometer_Psf_QM_v1.fits                    -MW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_SpecSatLimits_CQM_1_0.fits          -> PCalSpectrometer_SignalSatLimits_QM_v1.fits        -JS -OK
PacsCal_SpecSatLimits_FM_1_0.fits           -> PCalSpectrometer_SignalSatLimits_FM_v1.fits        -JS -OK

spg/SpecFlagSaturationFramesTask.py -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_SpecSortMatrix_CQM_1_0.fits      -> PCalSpectrometer_DetectorSortMatrix_QM_v1.fits     -MW -OK
PacsCal_SpecSortMatrix_FM_1_0.fits          -> PCalSpectrometer_DetectorSortMatrix_FM_v1.fits     -MW -OK

                                                                         Task: herschel.pacs.spu.DetectorSorting                                  -MW -OK
---------------------------------------------------------------------------------------------------------
PacsCal_SpecVolts_CQM_1_0.fits              -> PCalSpectrometer_Volts_QM_v1.fits                  -JS
PacsCal_SpecVolts_FM_1_0.fits               -> PCalSpectrometer_Volts_FM_v1.fits                  -JS 

spg/SpecConvDigit2VoltsPerSecFramesTask.py -JS -OK
spg/SpecConvDigit2VoltsRampsTask.py -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_SubarrayArray_CQM_1_0.fits          -> PCalPhotometer_SubArrayArray_QM_v1.fits            -EkW -OK
PacsCal_SubarrayArray_FM_1_0.fits           -> PCalPhotometer_SubArrayArray_FM_v1.fits            -EkW -OK
PacsCal_SubarrayArray_FM_2_0.fits           -> PCalPhotometer_SubArrayArray_FM_v2.fits            -EkW -OK
PacsCal_SubarrayArray_FM_2_1.fits           -> PCalPhotometer_SubArrayArray_FM_v3.fits            -EkW -OK

herschel.pacs.spg/PhotassignRaDec.java      - EkW - OK 
herschel.pacs.spg/PhotAssignRaDecTask.java  - EkW - OK
---------------------------------------------------------------------------------------------------------

-  ⇤ ← Revision 2 as of 2006-05-22 12:58:07 → 
  Size: 1085
  Editor: lalinekw2
  Comment:
+   ← Revision 48 as of 2008-10-09 13:16:12 → ⇥
  Size: 20293
  Editor: JuergenSchreiber
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 2:
-== Generate User Manual from XML as HTML ==

 * checkout the doc-files from the herschel.pacs
 * make you changes in the *.xml files in doc-files/help/um/xml
+[[TableOfContents(2)]]

== Procedure to develop Pipeline Tasks ==
 1. Select an assigned IA application from the IA Functionality list at the pacs website [[BR]]
 2. Check whether you are assigned  to develop this  application [[BR]]
 3. Contact Juergen (to update the website) and Ekki (to have an informed person) [[BR]]
 4. Juergen will flag the application as "under construction" in the IA Functionality list [[BR]]
 5. Write the Chapter in the User Manual (against this description you will develop) [[BR]]
     * Covering API (parameter, keywords, etc.) [[BR]]
     * Algorithm (references if possible) [[BR]]
     * Examples how the data is processed (before - after, figures) [[BR]]
 6. Contact a "scientific expert" to verify that your application is doing what is expected (if not already done earlier) [[BR]]
 7. Select and document Testdata (currently in User manual) [[BR]]
 8. Write a TestHarness - please contact Rik, Michael or Ekki if you do not know how[[BR]]
 9. Write the application (TestHarness may help) [[BR]]
 10. Test the application with the TestHarness  [[BR]]
 11. Check the application into CVS  [[BR]]
 12. Inform Juergen/Ekki that its finished  [[BR]]
 13. Juergen will flag the application as done  [[BR]]
 14. Tag the application  [[BR]]


== Generate User Manual from XML as PDF ==
 * cd vs-pcss-src>/pacs/develop/pcss/
 * checkout the doc-files from the herschel.pacs.doc-files
 * cd to <cvs-pcss-src>/pacs/develop/pcss/herschel/pacs/doc-files
 * make you changes in the *.xml files in help/um/xml
-Line 10:
+Line 31:
-     * $PCSS_OUT pointing to your development out directory
+     * $PCSS_OUT pointing to your development out directory (in this example $HOME/out)
 * make a file jake.root 
   {{{
      hcss.jake.outdir = $PCSS_OUT
   }}}
-Line 15:
+Line 40:
- * check the result in your web browser pointing to $HOME/out/pcss/doc/help/um/html/um.html
 * let me know if it worked, then I'll adopt the other scripts and write a nicer procedure!

Now you get a um.fo which will be converted (with the help of FOP http://xmlgraphics.apache.org/fop/download.html#binary) to PDF

{{{
  java -Xmx3g herschel.ia.document.tools.Fo \
     -fo ${LOCATION_PDF}/${DOC_NAME}.fo ${LOCATION_PDF}/${DOC_NAME}.pdf 
}}}
+ * In the moment you might get an error : xception in thread "main" java.lang.OutOfMemoryError: Java heap space
 * If you get no error  
       * check the result in your web browser pointing to $HOME/out/pcss/doc/help/um/html/um.html
 * if you get an error : exception in thread "main" java.lang.OutOfMemoryError: Java heap space
       * you find the $HOME/out/doc/help/um/pdf/um.fo file which will need to be converted (with the help of FOP http://xmlgraphics.apache.org/fop/download.html#binary) to PDF
       *  {{{
           java -Xmx3g herschel.ia.document.tools.Fo \
              -fo ${LOCATION_PDF}/${DOC_NAME}.fo ${LOCATION_PDF}/${DOC_NAME}.pdf 
           }}}
       * check the result in your web browser pointing to $HOME/out/pcss/doc/help/um/html/um.html

'''Attention:''' for the conversion to work properly you have to install a version of dbdoclet.jar. This software comes with its own version of jfreechart. If the dbdoclet.jar is in the classpath before the hcss external libraries, also hcss will use the jfreechart version of dbdoclet. This leads to problems at least with PlotXY and PacsQla.
So make sure dbdoclet.jar is added at the end of your classpath!
-Line 30:
+Line 59:
-t.main(["apid=1152"])
}}}
+}}}

== Size of level 0.5 data structures in PCSS ==
=== Spectrometer ===
Typical Spectrometer observation after decompression :

One second of Spectrometer data (default mode/3 raw channels) :

Basic science data + reduction add on (error, wavelength, ra, dec, 8 masks)

Detectors           : 25 * 18

Ramp length         : 64

Readout frequency   : 256 Hz

NrSubRamps          : 8

All data as doubles : 8 Byte

Readouts (Ramps)   : 25 * 18 * 256/64 * 8 * 8 = 115.2 KB / second

masks: same as readouts/64: 14.4 KB/second

Ramps DecMec data:

Readout frequency : 256 Hz

All data as int   : 4 Byte

DecMec entries    : 13

256 * 4 * 13 = 13KB / second

Raw data

Readout frequency   : 256 Hz

All data as doubles : 8 Byte

Nr raw channels     : 3

Raw data : 256 * 8 * 3 = 6 KB /second

-----------------------------------------------------------------

data reduced to Frames:

signals (Frames): 25 * 18 * 4 * 8 = 14.4 KB/second

Wavelength  : same as above : 14.4 KB /second

Error       : same as above : 14.4 KB / second

Ra:           same as above: 14.4 KB/second

Dec:         same as above: 14.4 KB/second

masks: 14.4 KB/second/64*9 = 2 KB/second

Frames DecMec data:

Readout frequency : 4 Hz

All data as int   : 4 Byte

DecMec entries    : 20

4 * 4 * 20 = 0.32 KB / second
--------------------------------------------------------------------
==> ~ 223 KB per second

--------------------------------------

One hour :

==> ~ 803 MB per hour

Observation duration <= 3hour (there may be exceptions, but then we
reduce the number of user)
----------------------------------------------------------------------------------------------
==> ~ 2.4 GB per observation

will produce additionally level 1 and 2 level 2 data cubes
--------------------------------
==> ~ 9 GB per observation

if you use red and blue spectrometer array, numbers are:

446 KB per second, 1.6 GB per hour, 4.8 GB per level 0.5 observation and 18 GB per level 2 observation (3 hours)


=== Photometer ===
Basic science data + reduction add on (error, noise)
----------------------------------------------------------
Detectors           : 64 * 32 = 2048 blue channel
                      32 * 16 =  512 red  channel
Readout frequency   : 40 Hz averaged on board to 10Hz
All data as doubles : 8 Byte

Reduced data :
------------
Readouts    : 64 * 32 * 10 * 8 = 163 KB / second
Noise       : same as above : 163 KB    / second
Error       : same as above : 163 KB    / second


== Product Access Layer ==
The Product Access Layer (PAL) is now available in PCSS. We encourage everybody to use it.
Basic documentation you find under :

ftp://ftp.rssd.esa.int/pub/HERSCHEL/csdt/releases/doc/ia/document/um/html/ch12s02.html

A typical example for Frames is :
{{{

storage=ProductStorage()
storage.register(SimplePool.getInstance()) 
reference=storage.save(frames)
print reference.urn

from herschel.pacs.signal import *
reference = storage.load("urn:simple.default:herschel.pacs.signal.Frames:1")

#get back your saved product
frame = reference.getProduct()
}}}

To use the GUI Browser to find your Products you may use :
{{{
from herschel.ia.pal.browser import ProductBrowser
from herschel.ia.pal.browser import ProductBrowserFrame

p=ProductBrowser(storage)
d=ProductBrowserFrame(p)
}}}

To get out the Data you need to move it in the Basket (marking in the GUI).
Then you may extract it from the basket :
{{{
prod=p.getJideBasket().toArray()[0].getProduct()
}}}

== Work distribution move to new Calibration Framework ==

{{{
---------------------------------------------------------------------------------------------------------
PacsCal_BoloCSs_CQM_1_0.fits                -> PCalPhotometer_CalSources_QM_v1.fits               -EkW -OK 
PacsCal_BoloCSs_FM_1_0.fits                 -> PCalPhotometer_CalSources_FM_v1.fits               -EkW -OK
---------------------------------------------------------------------------------------------------------
PacsCal_BoloFlatFields_CQM_1_0.fits         -> PCalPhotometer_FlatField_QM_v1.fits                -MW -OK
PacsCal_BoloFlatFields_FM_1_0.fits          -> PCalPhotometer_FlatField_FM_v1.fits                -MW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_BoloMasks_CQM_1_0.fits              -> PCalPhotometer_Masks_QM_v1.fits                    -MW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_ChopJitterThreshold_CQM_1_0.fits    -> PCalCommon_ChopperJitterThreshold_QM_v1.fits       -RH
PacsCal_ChopJitterThreshold_FM_1_0.fits     -> PCalCommon_ChopperJitterThreshold_FM_v1.fits       -RH
PacsCal_ChopJitterThreshold_FM_2_0.fits     -> PCalCommon_ChopperJitterThreshold_FM_v2.fits       -RH

spg/CleanPlateauTask.py  -JS- OK
spg/CleanPlateauRampsTask.py -JS -OK
---------------------------------------------------------------------------------------------------------
PacsCal_ChopperAngle_CQM_1_0.fits           -> PCalCommon_ChopperAngle_QM_v1.fits                 -RH
PacsCal_ChopperAngle_FM_1_0.fits            -> PCalCommon_ChopperAngle_FM_v1.fits                 -RH
PacsCal_ChopperAngle_FM_2_0.fits            -> PCalCommon_ChopperAngle_FM_v2.fits                 -RH

---------------------------------------------------------------------------------------------------------
PacsCal_ChopperAngleRedundant_FM_1_0.fits   -> PCalCommon_ChopperAngleRedundant_FM_v1.fits        -JS -OK
PacsCal_ChopperAngleRedundant_FM_2_0.fits   -> PCalCommon_ChopperAngleRedundant_FM_v2.fits        -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_ChopperSkyAngle_CQM_1_0.fits        -> PCalCommon_ChopperSkyAngle_QM_v1.fits              -RH
PacsCal_ChopperSkyAngle_FM_2_0.fits         -> PCalCommon_ChopperSkyAngle_FM_v2.fits              -RH

spg/ConvertChopper2AngleTask.py -JS -OK
common/chopPos2FpuAnglePol -JS -OK
common/fpuAngle2ChopPosPol -JS- OK
common/chopPos2SkyAnglePol -JS-OK
common/skyAngle2ChopPosPol -JS -OK
common/convertChopperAngleSteps -JS-OK
common/convertChopperReadoutSteps -JS -OK
---------------------------------------------------------------------------------------------------------
PacsCal_DiscardRampHooks_CQM_1_0.fits       -> PCalSpectrometer_DiscardRampHooks_QM_v1.fits       -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_EffectiveCapacitance_CQM_1_0.fits   -> PCalSpectrometer_EffectiveCapacitance_QM_v1.fits   -EkW -OK 
PacsCal_EffectiveCapacitance_FM_1_0.fits    -> PCalSpectrometer_EffectiveCapacitance_FM_v1.fits   -EkW -OK

spg/ConvertSignal2StandardCapTask.py - EkW - OK
spec/GetEffectiveCapacitanceTask.py  - JS- OK
---------------------------------------------------------------------------------------------------------
PacsCal_FilterTransmissionB2_CQM_1_0.fits   -> PCalCommon_FilterTransmissionB2_QM_v1.fits         - 
PacsCal_FilterTransmissionB3_CQM_1_0.fits   -> PCalCommon_FilterTransmissionB3_QM_v1.fits         -
PacsCal_FilterTransmissionBL_CQM_1_0.fits   -> PCalCommon_FilterTransmissionBL_QM_v1.fits         -
PacsCal_FilterTransmissionBS_CQM_1_0.fits   -> PCalCommon_FilterTransmissionBS_QM_v1.fits         -
PacsCal_FilterTransmission_FM_1_0.fits      -> PCalCommon_FilterTransmission_FM_v1.fits           -
PacsCal_FilterTransmission_FM_1_1.fits      -> PCalCommon_FilterTransmission_FM_v2.fits           -
PacsCal_FilterTransmission_FM_1_2.fits      -> PCalCommon_FilterTransmission_FM_v3.fits           -
PacsCal_FilterTransmissionR1_CQM_1_0.fits   -> PCalCommon_FilterTransmissionR1_QM_v1.fits         -
PacsCal_FilterTransmissionR_CQM_1_0.fits    -> PCalCommon_FilterTransmissionR_QM_v1.fits          -

---------------------------------------------------------------------------------------------------------
PacsCal_FilterWheel2Band_CQM_1_0.fits       -> PCalCommon_FilterWheel2Band_QM_v1.fits             -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_GPRHall_CQM_2_0.fits                -> PCalSpectrometer_GprHall_QM_v2.fits                -PR

---------------------------------------------------------------------------------------------------------
PacsCal_GratingJitterThreshold_CQM_1_0.fits -> PCalSpectrometer_GratingJitterThreshold_QM_v1.fits -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_KeyWavelength_CQM_1_0.fits          -> PCalSpectrometer_KeyWavelengths_QM_v1.fits         -MW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_LabelDescription_CQM_1_0.fits       -> PCalCommon_LabelDescription_QM_v1.fits             -RH

---------------------------------------------------------------------------------------------------------
PacsCal_LittrowParameters_CQM_1_0.fits      -> PCalSpectrometer_LittrowParameters_QM_v1.fits      -EkW ->BV
PacsCal_LittrowParameters_CQM_2_0.fits      -> PCalSpectrometer_LittrowParameters_QM_v2.fits      -EkW ->BV
PacsCal_LittrowParameters_FM_1_0.fits       -> PCalSpectrometer_LittrowParameters_FM_v1.fits      -EkW ->BV

spg/WaveCalcTask.py       - EkW - OK
spec/gratPos2Wave.py      - EkW - OK
---------------------------------------------------------------------------------------------------------
PacsCal_OBCPDescription_CQM_1_0.fits        -> PCalCommon_ObcpDescription_QM_v1.fits              -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotAbsorption_FM_1_0.fits          -> PCalPhotometer_Absorption_FM_v1.fits               -EkW ->BM
PacsCal_PhotAbsorption_FM_1_1.fits          -> PCalPhotometer_Absorption_FM_v2.fits               -EkW ->BM

---------------------------------------------------------------------------------------------------------
PacsCal_PhotArrayInstrument_CQM_1_0.fits    -> PCalPhotometer_ArrayInstrument_QM_v1.fits          -EkW -OK
PacsCal_PhotArrayInstrument_FM_1_0.fits     -> PCalPhotometer_ArrayInstrument_FM_v1.fits          -EkW -OK
PacsCal_PhotArrayInstrument_FM_2_0.fits     -> PCalPhotometer_ArrayInstrument_FM_v2.fits          -EkW -OK
PacsCal_PhotArrayInstrument_FM_2_1.fits     -> PCalPhotometer_ArrayInstrument_FM_v3.fits          -EkW -OK
herschel.pacs.spg/PhotassignRaDec.java     - EkW - OK
herschel.pacs.spg/PhotAssignRaDecTask.java - EkW - OK
---------------------------------------------------------------------------------------------------------
PacsCal_PhotBadPixelsMask_FM_1_0.fits       -> PCalPhotometer_BadPixelsMask_FM_v1.fits            -MW  -OK
                                                                              Task: toolbox.spg.PhotFlagBadPixelsTask.py                      -MW  -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotCorrZeroLevel_FM_1_0.fits       -> PCalPhotometer_CorrZeroLevel_FM_v1.fits            -MW  -OK
                                                                            Task: toolbox.spg.PhotCorrZeroLevelTask.py                    -MW  -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotCrosstalkMatrix_FM_1_0.fits     -> PCalPhotometer_CrosstalkMatrix_FM_v1.fits          -MW  -OK
                                                                              Task: herschel.pacs.spg.PhotCorrectCrossTalkTask          -MW  -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotGain_CQM_1_0.fits               -> PCalPhotometer_Gain_QM_v1.fits                     -EkW -OK
PacsCal_PhotGain_FM_1_0.fits                -> PCalPhotometer_Gain_FM_v1.fits                     -EkW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_PhotSatLimits_CQM_1_0.fits          -> PCalPhotometer_SatLimits_QM_v1.fits                -EkW -OK
PacsCal_PhotSatLimits_FM_1_0.fits           -> PCalPhotometer_SatLimits_FM_v1.fits                -EkW -OK

spg/PhotFlagSaturationTask.py
---------------------------------------------------------------------------------------------------------
PacsCal_PhotSortMatrix_CQM_1_0.fits	    -> PCalPhotometer_DetectorSortMatrix_QM_v1.fits       -MW  -OK
PacsCal_PhotSortMatrix_FM_0_0.fits	    -> PCalPhotometer_DetectorSortMatrix_FM_v1.fits       -MW  -OK 
PacsCal_PhotSortMatrix_FM_1_0.fits	    -> PCalPhotometer_DetectorSortMatrix_FM_v2.fits       -MW  -OK
PacsCal_PhotSortMatrix_FM_1_1.fits	    -> PCalPhotometer_DetectorSortMatrix_FM_v3.fits	  -MW  -OK

                                                                        Task: herschel.pacs.spu.DetectorSorting                                  -MW -OK
---------------------------------------------------------------------------------------------------------
PacsCal_RampModel_CQM_1_0.fits              -> PCalSpectrometer_RampModel_QM_v1.fits              -RH

---------------------------------------------------------------------------------------------------------
PacsCal_RsrfB2_CQM_1_0.fits                 -> PCalSpectrometer_RsrfB2_QM_v1.fits                 -RH
PacsCal_RsrfB2_CQM_2_0.fits                 -> PCalSpectrometer_RsrfB2_QM_v2.fits                 -RH
PacsCal_RsrfR1_CQM_1_0.fits                 -> PCalSpectrometer_RsrfR1_QM_v1.fits                 -RH
PacsCal_RsrfR1_CQM_2_0.fits                 -> PCalSpectrometer_RsrfR1_QM_v2.fits                 -RH

---------------------------------------------------------------------------------------------------------
PacsCal_SatLimits_CQM_1_0.fits              -> PCalSpectrometer_RampSatLimits_QM_v1.fits          -EkW -OK 
PacsCal_SatLimits_FM_1_0.fits               -> PCalSpectrometer_RampSatLimits_FM_v1.fits          -EkW -OK

spg/SpecFlagSaturationRampsTask.py  -JS-OK
---------------------------------------------------------------------------------------------------------
PacsCal_Siam_CQM_1_0.fits                   -> PCalCommon_Siam_QM_v1.fits                         -RH

---------------------------------------------------------------------------------------------------------
PacsCal_SpecBadPixelsMask_FM_1_0.fits       -> PCalSpectrometer_BadPixelMask_FM_v1.fits           -JS -OK

spg/SpecFlagBadPixelsTask.py  -JS -OK
---------------------------------------------------------------------------------------------------------
PacsCal_SpecRedPSF_CQM_1_0.fits             -> PCalSpectrometer_Psf_QM_v1.fits                    -MW -OK

---------------------------------------------------------------------------------------------------------
PacsCal_SpecSatLimits_CQM_1_0.fits          -> PCalSpectrometer_SignalSatLimits_QM_v1.fits        -JS -OK
PacsCal_SpecSatLimits_FM_1_0.fits           -> PCalSpectrometer_SignalSatLimits_FM_v1.fits        -JS -OK

spg/SpecFlagSaturationFramesTask.py -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_SpecSortMatrix_CQM_1_0.fits	 -> PCalSpectrometer_DetectorSortMatrix_QM_v1.fits     -MW -OK
PacsCal_SpecSortMatrix_FM_1_0.fits	    -> PCalSpectrometer_DetectorSortMatrix_FM_v1.fits     -MW -OK

                                                                         Task: herschel.pacs.spu.DetectorSorting                                  -MW -OK
---------------------------------------------------------------------------------------------------------
PacsCal_SpecVolts_CQM_1_0.fits	            -> PCalSpectrometer_Volts_QM_v1.fits	          -JS
PacsCal_SpecVolts_FM_1_0.fits               -> PCalSpectrometer_Volts_FM_v1.fits                  -JS 

spg/SpecConvDigit2VoltsPerSecFramesTask.py -JS -OK
spg/SpecConvDigit2VoltsRampsTask.py -JS -OK

---------------------------------------------------------------------------------------------------------
PacsCal_SubarrayArray_CQM_1_0.fits	    -> PCalPhotometer_SubArrayArray_QM_v1.fits            -EkW -OK
PacsCal_SubarrayArray_FM_1_0.fits	    -> PCalPhotometer_SubArrayArray_FM_v1.fits            -EkW -OK
PacsCal_SubarrayArray_FM_2_0.fits	    -> PCalPhotometer_SubArrayArray_FM_v2.fits            -EkW -OK
PacsCal_SubarrayArray_FM_2_1.fits	    -> PCalPhotometer_SubArrayArray_FM_v3.fits            -EkW -OK

herschel.pacs.spg/PhotassignRaDec.java      - EkW - OK 
herschel.pacs.spg/PhotAssignRaDecTask.java  - EkW - OK
---------------------------------------------------------------------------------------------------------
}}}

Herschel: Diff for "PACS/Pcss"

PACS - PCSS

Procedure to develop Pipeline Tasks

Generate User Manual from XML as PDF

Usage of TestExecutionBrowser just to get an overview

Size of level 0.5 data structures in PCSS

Spectrometer

Photometer

Product Access Layer

Work distribution move to new Calibration Framework