EROSITAwiki: eSASSdevel_200929

eSASS developers Zoom meeting

Start of meeting: 2020-09-29 13:30 (1:30pm)

Zoom link: https://zoom.us/j/7909307159?pwd=ZEdGeTJQUzdsRGpkd2Q5aUZXQmZZdz09

Participants: Ian Stewart, Hermann Brunner, Miriam Ramos, Ingo Kreykenbohm, Georg Lamer, Chandreyee Maitra, Andrea Merloni, Alain Gueguen, Mara Salvato, Jeremy Sanders, Michael Freyberg

• Open actions
  • Act20200811/1_GL: investigate whether ersensmap output maps are correct - ongoing
    • algorithm seems correct, but pointing PSF is not implemented (work already started)
    • how well do the ersensmap maps agree with survey/field scan source catalogs?
    • could be put in 947 survey pipeline immediately but pointing pipeline requires update
• New actions
  • Act2020_0929/1_all (Ian, Georg, Jeremy, Antonis, Mirsiam and Hermann will provide recomendations): make sure html task descriptions are current and consistent with user release
• NRTA
  • too many emails / Bamberg email access?
• DAtool
  • (Jul. 28 meeting)
    • Region requests issues
      • owner filtering / evtool: use MPE owner event flag
      • Were all of Georg's but reports fixed?
    • catalogs can't yet be filtered, whole catalog will be returned
      • Should this be implemented? So far only event files are filtered (via evtool).
      • Catalog filtering could be implemented in Python based DAtool 2
    • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
    • move simple copying request to Python / should everything be moved to Python? - ongoing
    • soft links instead of copying - not yet implemented in backend, side effect due to compressed files (compressed files ending in .fits may be delivered) - needs to be checked; frontend creates soft links for all-sky catalogs
    • DAtool / eROSE interface: requires change in fronted
    • Frontend problem: first request by a user occasionally doesn't work
  • Help-desk question (Sept. 3) - download script (bash) does not work on windows
    • alternative to script: lftp or wget - documentation?
    • bash script should not be opened with Windows editor!
  • delivered catalogs - ermldet? output (Sept. 11) - send file pattern to Alain (done)
  • Download size to big? - talk to Frank
  • Ask for feedback ath Thrusdasy meeting
• CalPV data release / public software release
  • Documentation
    • Products
      • So far only for event files and source catalogs (both need updating)
      • for other products: task descriptons
    • Caldb
      • List of caldb files with briegf descrition and task which access them
      • bti in CalPV
      • Paper about calibration
    • Software
      • Wiki / Cook book - public Wiki page or PDF document? Tied to software release
      • Task descriptions: update for release, then fix version for software release
        • May need updating (make consistent - needs reviewing), life HTML versions need to be copied to static version for user release

    • Documentation should be on SciServer (either Web pages or in notebook)

  • Source code (GNU GPL issues)
    • Source should not be a problem, as long as we don't copy source code from public libraries
    • For distributing binaries: we need to provide a way to reproduce the binaries from source)
    • Check specific licenses of libraires; we need to provide our code under a compatible license (which on: GNU GPL?)
    • ESA: less-GPL?
  • New users release
    • task versions
      • astrolib-2.6, evtool-2.10.1, erolib-2.24.4
      • testing new LAPACK (apetool, ersensmap), only polynomials used
      • apetool, ersensmap: newest versions?
    • Docker version for all architectures, source build for Linux (Jonas and Alain will interact)
• User requests
  • Making attitude files public
    • make Georg's script a cronjob / put in directory next to eposure HiPS maps
    • Chandreyee will look into this
  • Orbit file access (Axel Schwope) - use caldb download script?
    • cronjob to copy them in directory under caldb / add to download script
• 946 processing
  • 946 eRASS1 processing
    • Some faulty exposure maps - fixed?
    • Failed 946 TEL chain runs?
    • HiPS maps

      • Comparing 946/945 exposure & images

      • Bigger /tmp - is this from the HiPS scripts?
    • Spurious sources in one great circle - Georg will provide input
    • Source catalogs, ca. 2% of sources have positional error 0, some also have count error 0
  • eRASS2 processing
    • Features in exposure maps (bright spots / pointings?)
  • CalPV reprocessing
  • Copying sky tiles to new disks (ero_ARCHIVE6/7)
  • SOU chain
    • dalist task fixed, fixing soft links

• eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

  • expmap
    • PE015
    • Pattern selection - (v1.34 - not in new users release)
    • Bring coordinate transformations into line with evatt
    • Parallel version
  • evtool: image autosizing - in new users release (2.10.1)
  • telatt (PE025): DATE_OBS, DATE_END missing in input
    • telatt will report that not GTI are present and terminate
  • Ownership flags, keywords, column, filenames:

    • srctool: setting owner character in filenames requires SASS_PARINT env. in lower for versions<1.61

      • erolib expects SASS_PAINT in upper case
      • param accepts lower and upper case
    • catprep: set output and output2 (947 pipeline)
  • Recent task uploads
    • srctool-1.61 (light curve errors)

    • erolib-2.24.4, evtool-2.10.1

    • erolib-2.29.1

• Camera resets / held cameras

  • BTI document

• Longterm activities
  • Light leak mitigation - outcome of threshold test
  • Time dependent RMFs/ARFs (srctool?)
  • Background analysis: projecting closed data onto the sky (XMM SAS) - dedicated meeting?
  • Moving to git

Previous meeting:

• Open actions
  • Act20200811/1_GL: investigate whether ersensmap output maps are correct - ongoing
    • algorithm seems correct, but pointing PSF is not implemented (work already started)
    • how well do the ersensmap maps agree with survey/field scan source catalogs?
    • could be put in 947 survey pipeline immediately but pointing pipeline requires update
• New actions
• NRTA
• DAtool
  • (Jul. 28 meeting)
    • Region requests issues
      • owner filtering / evtool: use MPE owner event flag
      • Were all of Georg's but reports fixed?
    • catalogs can't yet be filtered, whole catalog will be returned
      • Should this be implemented? So far only event files are filtered (via evtool).
      • Catalog filtering could be implemented in Python based DAtool 2
    • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
    • move simple copying request to Python / should everything be moved to Python? - ongoing
    • soft links instead of copying - not yet implemented in backend, side effect due to compressed files (compressed files ending in .fits may be delivered) - needs to be checked; frontend creates soft links for all-sky catalogs
    • DAtool / eROSE interface: requires change in fronted
    • Frontend problem: first request by a user occasionally doesn't work
  • Help-desk question (Sept. 3) - download script (bash) does not work on windows
    • alternative to script: lftp or wget - documentation?
    • bash script should not be opened with Windows editor!
  • delivered catalogs - ermldet? output (Sept. 11) - send file pattern to Alain
• Catalogs file naming convention
• Moving task archive to git / build infrastructure
  • git access for users?
  • Access to users but only accessible via MPE acount (to access gitlab)
  • Problems due to open source libraries?
• 945 vs 946 rates?
• New eSASS users release
  • code freeze / 946 vs. user release task versions?
  • Make a dedicated public user release consitent with the data release
  • Provide public eSASS in docker container (Jonas)
• Public data release / public software release
  • Consistent CalPV processing task versions
  • Which qualitiy checks?
    • Need to define a list of checks to perform
  • Which eSASS version?
• User requests
  • Making attitude files public
    • make Georg's script a cronjob / put in directory next to eposure HiPS maps
  • Orbit file access (Axel Schwope) - use caldb download script?
    • cronjob to copy them in directory under caldb / add to download script
• 946 / 947 processing
  • eRASS1 946 EXP and DET chain processing finished (almost)
  • eRASS2 946 processing ongoing (current till yesterday)
  • Next: 946 SOU chain(s), 947 reprocessing

• eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

  • expmap
    • PE015
    • Look into pattern selection in expmap - is in new users release
    • Bring coordinate transformations into line with evatt
    • Time dependent bad pixels - was already implemented!
    • Parallel version
  • evtool: image autosizing - in new users release (number of pixels is dynamic)
  • telatt (PE025): DATE_OBS, DATE_END missing in input
    • Decide how DAT_OBS/DATE_END and TSTART/TEND should be set (currently matching GTI)
    • telatt will report that not GTI are present and terminate
  • srctool: errorbars in lightcurves
    • Options: remove errors altogether (Paul's suggestion), only remove them if counts are to low, compute asymmetric errors (TBD)
    • Next srctool version: no error bars if less than 25 counts per bin
  • all tasks (also DATool): reviewing ownership flags,column, keywords, filenames (event lists, catalogs, srctool products)
    • srctool: setting owner character in filenames requires SASS_PARINT env. in lower case (pipeline) - done
    • catprep: set output and output2 (947 pipeline)
  • flaregti-1.20 / PE022 fixed?
  • Recent task uploads

    • erolib-2.29, evtool-2.12, expmap-1.34, erolib-2.30 (bug?)

• Camera resets / held cameras

  • BTI document

• Longterm activities
  • Light leak mitigation - outcome of threshold test
  • Time dependent RMFs/ARFs (srctool?)
  • Background analysis: projecting closed data onto the sky (XMM SAS) - dedicated meeting?
  • Public data release / public software release
  • Moving to git