Framework Release Notes - CMSSW_2_1_X

• General:
  • This release contains many months worth of development and changes. Some of them are from work planned for this release and some are more minor things done on user request. I list the major things below followed by the many small things.
• Main changes and new features:
  • Major Changes
    • Migration to the python configuration system. After 2_1_X the old configuration system will no longer work. Please see https://twiki.cern.ch/twiki/bin/view/CMS/WorkBookConfigFileIntro#PythonConfig for more information.
    • Product IDs are now independent of branch IDs so that they become per event definitions. This will allow loosening of some merging restrictions. There is no longer a need to use the firstFreeID, so it is removed from the parameter list of input sources.
    • The EventMetaData tree has been collapsed into a single branch. Also, the EventStatusInfo tree has disappeared, being absorbed into the EventMetaData tree. Similar changes apply to the corresponding Run and Lumi trees. This was done to reduce the amount of memory taken by the meta data system.
    • Add another space saving measure for the size of the EventMetaData tree. Products that are dropped have their provenance dropped as well along with the all decendent's provenance. We call this EventMetaData tree the pruning.
    • Internal improvements to the mixing module IO which reduce it's memory cost. It will now only use memory for the hit collection branches in the secondary file that it needs. With this change it is now possible to generate pilup samples in production again.
    • In the PoolOutputModule the default compression level is now 7. It used to be 1.
    • RandomNumberGenerator configuration interface changed to make it more clear, see https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideEDMRandomNumberGeneratorService
    • An input drop feature has been added. It uses the same syntax as the output drop commands however it does behave differently. On input the products listed for dropping will also have all of their decendent products dropped. This is so that the current process can not use the deleted product nor any of the products derived from it, provenance would be lost if we allowed that. If you want to fine tune the event contents written to the file continue to use the output keep/drop commands. Drop on input saves process memory and allows greater control over what can be used by the current process.
    • New version of the CMSSW I/O layer has replaced the old one, with the following now available however not enabled by default:
      • Automatic stage-in of all input files at job startup.
      • Support for the new I/O model in ROOT 5.
      • Caching and/or prefetching via TTreeCache.
      • Improved I/O adaptors for local files, rfio, dcap.
      • File access adaptors for storm / gpfs (at cnaf).
      • Added I/O adaptor for xrootd (inactive within cmssw).

For more details see the https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideStorageStats and https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideTFileAdaptor twiki pages

• Minor features:
  • There is a new job mode command line parameter for cmsRun : cmsRun –mode=infos whatever_cfg.py The defaults for this mode differ from –mode=grid (or no mode specified at all) by the threshold for statistics being INFO rather than WARNING, and by the default limit for unspecified categories of INFO exceptions being 5 rather than 0.
  • edmParameterSetDump now dumps service configurations too.
  • Support selected specific events for processing with the addition of a eventsToProcess parameter to PoolSource.
  • Users can now access the input file name when a new EDM/ROOT file is opened by implementing !respondToOpenInputFile in their module and using the FileBlock passed into that function to grab the file name with the edm::FileBlock::fileName() function.
  • Add new function edm::ParameterSet::getProcessParameterSet() to enable users to easily get the process parameter set.
  • PrintEventSetupContent now also prints IOV intervals.
  • Add the orbit numer to !EventAuxiliary, as requested by Ivan Mikulec. Also add a read only accessor to it and the !bunchID from the Event.
  • Make streamer file meta data consistent with root file meta data. The ParameterSet registry, and the Module Description registry are saved in the streamer file header, while the EventHistory registry, and EntryDescription registry are saved not as registries, but event by event. These changes will make it easier to do HLT DQM but further work is needed.
  • For data management, add the ability to specify addition TFCs the additional one will override the rules for the SITE.
  • A number of features have been added to the Framework job report (FWJR):
    • The Memory Service reports a summary to the FWJR
    • The Timing Service reports a summary to the FWJR
    • A message logger summary can now be sent to the FWJR. This is enabled by setting !messageSummaryToJobReport = true in the MessageLoggerService
    • Input run and lumi numbers are now properly recording in the FWJR.
    • The FWJR now contains a hash which represents the branch content of an output file.

Updates to Externals:

Major bug fixes:

• Known issues:
  • When histograms are stored in the run tree (as is done for Offline DQM harvesting), and the processing job is fed many runs out of time sequence the cmsRun process will run out of memory. Without the time ordering it is impossible to integrate over many runs because the cache of products becomes too large. In a furture version of the ProdAgent the files sequence will become time ordered, when this happens the FW will default to a mode that minimizes the product cache size probably in CMSSW_2_1_1.

---

Last updated: 30 July 2008 Liz Sexton-Kennedy