test
CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Public Attributes
ValidationMatrix_v2.ReleaseComparison Class Reference
Inheritance diagram for ValidationMatrix_v2.ReleaseComparison:

Public Member Functions

def __init__
 
def compare
 
def was_compared
 

Public Attributes

 clear_db
 
 db_name
 Download needed files. More...
 
 dry
 
 no_url
 
 use_external_script_to_compare_files
 
 work_path
 Create working directory if not given. More...
 

Detailed Description

Generates release comparison information and stores it on SQLite3 .db file.

Definition at line 120 of file ValidationMatrix_v2.py.

Constructor & Destructor Documentation

def ValidationMatrix_v2.ReleaseComparison.__init__ (   self,
  work_path = None,
  db_name = None,
  clear_db = False,
  dry = False,
  no_url = False,
  use_external = False 
)

Definition at line 122 of file ValidationMatrix_v2.py.

122 
123  def __init__(self, work_path=None, db_name=None, clear_db=False, dry=False, no_url=False, use_external=False):
124  self.work_path = work_path
125  self.db_name = db_name
126  self.clear_db = clear_db
127  self.dry = dry
128  self.no_url = no_url
129  self.use_external_script_to_compare_files = use_external
ValidationMatrix_v2.ReleaseComparison.no_url
no_url
Definition: ValidationMatrix_v2.py:127
ValidationMatrix_v2.ReleaseComparison.use_external_script_to_compare_files
use_external_script_to_compare_files
Definition: ValidationMatrix_v2.py:128
ValidationMatrix_v2.ReleaseComparison.work_path
work_path
Create working directory if not given.
Definition: ValidationMatrix_v2.py:123
ValidationMatrix_v2.ReleaseComparison.__init__
def __init__
Definition: ValidationMatrix_v2.py:122
ValidationMatrix_v2.ReleaseComparison.db_name
db_name
Download needed files.
Definition: ValidationMatrix_v2.py:124
ValidationMatrix_v2.ReleaseComparison.dry
dry
Definition: ValidationMatrix_v2.py:126
ValidationMatrix_v2.ReleaseComparison.clear_db
clear_db
Definition: ValidationMatrix_v2.py:125

Member Function Documentation

def ValidationMatrix_v2.ReleaseComparison.compare (   self,
  rel1,
  frags1,
  rel2,
  frags2,
  st_tests,
  url = None,
  title = None 
)

Definition at line 141 of file ValidationMatrix_v2.py.

References utils_v2.check_disk_for_space(), ValidationMatrix_v2.ReleaseComparison.clear_db, compare_using_files_v2.RootFileComparison.db_name, ValidationMatrix_v2.ReleaseComparison.db_name, ValidationMatrix_v2.ReleaseComparison.dry, cmsRelvalreport.exit, utils_v2.get_size_to_download(), utils_v2.get_version(), utils_v2.init_database(), join(), list(), utils_v2.make_file_pairs(), python.multivaluedict.map(), ValidationMatrix_v2.merge_dbs(), ValidationMatrix_v2.ReleaseComparison.no_url, ValidationMatrix_v2.partial_db_name(), utils_v2.recursive_search_online(), python.multivaluedict.remove(), utils_v2.search_on_disk(), ValidationMatrix_v2.ReleaseComparison.use_external_script_to_compare_files, compare_using_files_v2.RootFileComparison.was_compared(), ValidationMatrix_v2.ReleaseComparison.was_compared(), compare_using_files_v2.RootFileComparison.work_path, and ValidationMatrix_v2.ReleaseComparison.work_path.

141 
142  def compare(self, rel1, frags1, rel2, frags2, st_tests, url=None, title=None):
143  print '\n################# Searching for files ###################'
144  if self.no_url:
145  print 'Searching for files on disk at %s' % (self.work_path)
146  files1, files2 = search_on_disk(self.work_path, rel1, frags1, rel2, frags2)
147  file_pairs = make_file_pairs(files1, files2)
148  else:
149  print 'Searching for files online at:'
150  files_with_urls1, files_with_urls2 = recursive_search_online(url, rel1, frags1, rel2, frags2)
151  file_pairs = make_file_pairs(files_with_urls1, files_with_urls2)
152  files_with_urls1.update(files_with_urls2)
153  files1, files2 = zip(*file_pairs)
154  paired_files_with_urls = [(file, files_with_urls1[file]) for file in files1 + files2]
155 
156  if self.dry:
157  print 'DRY: nothing to do. Exiting.'
158  exit()
159 
160  ## Create working directory if not given.
161  if not self.work_path:
162  self.work_path = '%s___VS___%s' % (get_version(files1[0]), get_version(files2[0]))
163  if self.db_name:
164  self.db_name = join(self.work_path, self.db_name)
165 
166  if not exists(self.work_path):
167  print '\n################### Preparing directory ###################'
168  print 'Creating working directory: %s ...' % self.work_path,
169  makedirs(self.work_path)
170  print 'Done.'
171 
172  print '\n################# Downloading the files ###################'
173  total_size, files_to_download = get_size_to_download(self.work_path, paired_files_with_urls)
174  check_disk_for_space(self.work_path, total_size)
175 
176  ## Download needed files.
177  q = Queue()
178  show_status_bar.q = q
179  auth_download_file.q = q
180  auth_download_file.work_dir = self.work_path
181 
182  Process(target=show_status_bar, args=(total_size,)).start()
183  Pool(2).map(auth_download_file, files_to_download)
184  if total_size:
185  print "Done."
186 
187  ## Create database
188  print '\n################# Preparing Database ###################'
189  if not self.db_name:
190  self.db_name = '%s___VS___%s.db' % (get_version(file_pairs[0][0]), get_version(file_pairs[0][1]))
191 
192  if self.clear_db:
193  print 'Clearing DB: %s...' % self.db_name,
194  open(join(self.work_path, self.db_name), 'w').close()
195  print 'Done.'
196 
197  ## Compare file pairs.
198  self.db_name = init_database(join(self.work_path, self.db_name))
199 
200  # TODO: Use multiprocessing for this task.
201  for st_test_name in st_tests.split(','):
202  print '\n################# Comparing Releases (%s) ###################' % st_test_name
203  st_test = tests[st_test_name]()
204 
205  some_files_compared = False
206  file_comparison_ids = []
207  if self.use_external_script_to_compare_files:
208  # Compare files using compare_using_files_v2.py
209  arg_list = [list(pair) + [self.work_path, partial_db_name(self.db_name, i),
210  self.clear_db] for i, pair in enumerate(file_pairs)]
211  pool = Pool(7)
212  pool.map(call_compare_using_files, arg_list)
213 
214  # Merge databases
215  print '\n################# Merging DBs (%s) ###################' % st_test_name
216  for i, pair in enumerate(file_pairs):
217  tmp_db = partial_db_name(self.db_name, i)
218  print 'Merging %s...' % (basename(tmp_db),),
219  file_comparison_ids.append(merge_dbs(self.db_name, tmp_db))
220  remove(tmp_db)
221  print 'Done.'
222  some_files_compared = True
223  else:
224  file_comparison = RootFileComparison(self.db_name)
225 
226  for file1, file2 in file_pairs:
227  # TODO: If files are not found desplay nice message.
228  # TODO: Maybe subprocces would control the unwanted reports of RootFileComparison.compare()
229  file1_path = join(self.work_path, file1)
230  file2_path = join(self.work_path, file2)
231 
232  if not file_comparison.was_compared(file1, file2, st_test_name):
233  print "Comparing:\n%s\n%s\n" % (file1, file2)
234  file_comparison_id = file_comparison.compare(file1_path, file2_path, st_test)
235  file_comparison_ids.append(file_comparison_id)
236  some_files_compared = True
237  else:
238  print "Already compared:\n%s\n%s\n" % (file1, file2)
239 
240  ## Calculate statistics for the release.
241  release1 = get_version(file_pairs[0][0])
242  release2 = get_version(file_pairs[0][1])
243  if some_files_compared:
244  release_comparison_id = self.was_compared(release1, release2, st_test_name)
245  conn = sqlite3.connect(self.db_name)
246  c = conn.cursor()
247  if not release_comparison_id:
248  print 'Inserting release "%s VS %s" description.\n' % (release1, release2)
249  if not title:
250  title = "%s__VS__%s" % (release1, release2)
251  c.execute('''INSERT INTO ReleaseComparison(title, release1, release2,
252  statistical_test) VALUES (?, ?, ?, ?)''', (title,
253  release1, release2, st_test_name))
254  release_comparison_id = c.lastrowid
255  c.executemany('''UPDATE RootFileComparison SET release_comparison_id = ?
256  WHERE id == ?''', [(release_comparison_id, fid) for fid in file_comparison_ids])
257  conn.commit()
258  conn.close()
259 
start
Definition: start.py:1
utils_v2.search_on_disk
def search_on_disk
Definition: utils_v2.py:253
ValidationMatrix_v2.partial_db_name
def partial_db_name
Definition: ValidationMatrix_v2.py:70
utils_v2.check_disk_for_space
def check_disk_for_space
Definition: utils_v2.py:459
ValidationMatrix_v2.ReleaseComparison.no_url
no_url
Definition: ValidationMatrix_v2.py:127
compare_using_files_v2.RootFileComparison
Definition: compare_using_files_v2.py:37
ValidationMatrix_v2.ReleaseComparison.use_external_script_to_compare_files
use_external_script_to_compare_files
Definition: ValidationMatrix_v2.py:128
ValidationMatrix_v2.ReleaseComparison.work_path
work_path
Create working directory if not given.
Definition: ValidationMatrix_v2.py:123
ValidationMatrix_v2.ReleaseComparison.was_compared
def was_compared
Definition: ValidationMatrix_v2.py:130
ValidationMatrix_v2.ReleaseComparison.db_name
db_name
Download needed files.
Definition: ValidationMatrix_v2.py:124
ValidationMatrix_v2.ReleaseComparison.dry
dry
Definition: ValidationMatrix_v2.py:126
ValidationMatrix_v2.ReleaseComparison.clear_db
clear_db
Definition: ValidationMatrix_v2.py:125
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
ValidationMatrix_v2.ReleaseComparison.compare
def compare
Definition: ValidationMatrix_v2.py:141
utils_v2.recursive_search_online
def recursive_search_online
Definition: utils_v2.py:213
utils_v2.make_file_pairs
def make_file_pairs
Definition: utils_v2.py:105
utils_v2.init_database
def init_database
Utils.
Definition: utils_v2.py:388
utils_v2.get_version
def get_version
Definition: utils_v2.py:432
ValidationMatrix_v2.merge_dbs
def merge_dbs
Definition: ValidationMatrix_v2.py:74
utils_v2.get_size_to_download
def get_size_to_download
Definition: utils_v2.py:443
list
How EventSelector::AcceptEvent() decides whether to accept an event for output otherwise it is excluding the probing of A single or multiple positive and the trigger will pass if any such matching triggers are PASS or EXCEPTION[A criterion thatmatches no triggers at all is detected and causes a throw.] A single negative with an expectation of appropriate bit checking in the decision and the trigger will pass if any such matching triggers are FAIL or EXCEPTION A wildcarded negative criterion that matches more than one trigger in the trigger list("!*","!HLTx*"if it matches 2 triggers or more) will accept the event if all the matching triggers are FAIL.It will reject the event if any of the triggers are PASS or EXCEPTION(this matches the behavior of"!*"before the partial wildcard feature was incorporated).Triggers which are in the READY state are completely ignored.(READY should never be returned since the trigger paths have been run
def ValidationMatrix_v2.ReleaseComparison.was_compared (   self,
  release1,
  release2,
  st_test_name 
)

Definition at line 130 of file ValidationMatrix_v2.py.

References compare_using_files_v2.RootFileComparison.db_name, and ValidationMatrix_v2.ReleaseComparison.db_name.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

130 
131  def was_compared(self, release1, release2, st_test_name):
132  conn = sqlite3.connect(self.db_name)
133  c = conn.cursor()
134  c.execute('''SELECT id FROM ReleaseComparison WHERE release1=? AND
135  release2=? AND statistical_test=?''', (release1, release2, st_test_name))
136  release_comparison_id = c.fetchone()
137  conn.close()
138  if release_comparison_id:
139  return release_comparison_id[0]
140  return False
ValidationMatrix_v2.ReleaseComparison.was_compared
def was_compared
Definition: ValidationMatrix_v2.py:130
ValidationMatrix_v2.ReleaseComparison.db_name
db_name
Download needed files.
Definition: ValidationMatrix_v2.py:124

Member Data Documentation

ValidationMatrix_v2.ReleaseComparison.clear_db

Definition at line 125 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

ValidationMatrix_v2.ReleaseComparison.db_name

Download needed files.

Compare file pairs.

Create database

Definition at line 124 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare(), and ValidationMatrix_v2.ReleaseComparison.was_compared().

ValidationMatrix_v2.ReleaseComparison.dry

Definition at line 126 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

ValidationMatrix_v2.ReleaseComparison.no_url

Definition at line 127 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

ValidationMatrix_v2.ReleaseComparison.use_external_script_to_compare_files

Definition at line 128 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

ValidationMatrix_v2.ReleaseComparison.work_path

Create working directory if not given.

Definition at line 123 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

Generated for CMSSW Reference Manual by doxygen 1.8.5