CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Groups Pages
List of all members | Public Member Functions | Public Attributes
ValidationMatrix_v2.ReleaseComparison Class Reference
Inheritance diagram for ValidationMatrix_v2.ReleaseComparison:

Public Member Functions

def __init__
 
def compare
 
def was_compared
 

Public Attributes

 clear_db
 
 db_name
 Download needed files. More...
 
 dry
 
 no_url
 
 use_external_script_to_compare_files
 
 work_path
 Create working directory if not given. More...
 

Detailed Description

Generates release comparison information and stores it on SQLite3 .db file.

Definition at line 121 of file ValidationMatrix_v2.py.

Constructor & Destructor Documentation

def ValidationMatrix_v2.ReleaseComparison.__init__ (   self,
  work_path = None,
  db_name = None,
  clear_db = False,
  dry = False,
  no_url = False,
  use_external = False 
)

Definition at line 123 of file ValidationMatrix_v2.py.

123 
124  def __init__(self, work_path=None, db_name=None, clear_db=False, dry=False, no_url=False, use_external=False):
125  self.work_path = work_path
126  self.db_name = db_name
127  self.clear_db = clear_db
128  self.dry = dry
129  self.no_url = no_url
130  self.use_external_script_to_compare_files = use_external
ValidationMatrix_v2.ReleaseComparison.no_url
no_url
Definition: ValidationMatrix_v2.py:128
ValidationMatrix_v2.ReleaseComparison.use_external_script_to_compare_files
use_external_script_to_compare_files
Definition: ValidationMatrix_v2.py:129
ValidationMatrix_v2.ReleaseComparison.work_path
work_path
Create working directory if not given.
Definition: ValidationMatrix_v2.py:124
ValidationMatrix_v2.ReleaseComparison.__init__
def __init__
Definition: ValidationMatrix_v2.py:123
ValidationMatrix_v2.ReleaseComparison.db_name
db_name
Download needed files.
Definition: ValidationMatrix_v2.py:125
ValidationMatrix_v2.ReleaseComparison.dry
dry
Definition: ValidationMatrix_v2.py:127
ValidationMatrix_v2.ReleaseComparison.clear_db
clear_db
Definition: ValidationMatrix_v2.py:126

Member Function Documentation

def ValidationMatrix_v2.ReleaseComparison.compare (   self,
  rel1,
  frags1,
  rel2,
  frags2,
  st_tests,
  url = None,
  title = None 
)

Definition at line 142 of file ValidationMatrix_v2.py.

References utils_v2.check_disk_for_space(), ValidationMatrix_v2.ReleaseComparison.clear_db, compare_using_files_v2.RootFileComparison.db_name, ValidationMatrix_v2.ReleaseComparison.db_name, ValidationMatrix_v2.ReleaseComparison.dry, beamvalidation.exit(), utils_v2.get_size_to_download(), uploadConditions.get_version(), utils_v2.init_database(), join(), utils_v2.make_file_pairs(), ValidationMatrix_v2.merge_dbs(), ValidationMatrix_v2.ReleaseComparison.no_url, ValidationMatrix_v2.partial_db_name(), print(), utils_v2.recursive_search_online(), MatrixUtil.remove(), utils_v2.search_on_disk(), ValidationMatrix_v2.ReleaseComparison.use_external_script_to_compare_files, compare_using_files_v2.RootFileComparison.was_compared(), ValidationMatrix_v2.ReleaseComparison.was_compared(), compare_using_files_v2.RootFileComparison.work_path, ValidationMatrix_v2.ReleaseComparison.work_path, and ComparisonHelper.zip().

142 
143  def compare(self, rel1, frags1, rel2, frags2, st_tests, url=None, title=None):
144  print('\n################# Searching for files ###################')
145  if self.no_url:
146  print('Searching for files on disk at %s' % (self.work_path))
147  files1, files2 = search_on_disk(self.work_path, rel1, frags1, rel2, frags2)
148  file_pairs = make_file_pairs(files1, files2)
149  else:
150  print('Searching for files online at:')
151  files_with_urls1, files_with_urls2 = recursive_search_online(url, rel1, frags1, rel2, frags2)
152  file_pairs = make_file_pairs(files_with_urls1, files_with_urls2)
153  files_with_urls1.update(files_with_urls2)
154  files1, files2 = list(zip(*file_pairs))
155  paired_files_with_urls = [(file, files_with_urls1[file]) for file in files1 + files2]
156 
157  if self.dry:
158  print('DRY: nothing to do. Exiting.')
159  exit()
160 
161  ## Create working directory if not given.
162  if not self.work_path:
163  self.work_path = '%s___VS___%s' % (get_version(files1[0]), get_version(files2[0]))
164  if self.db_name:
165  self.db_name = join(self.work_path, self.db_name)
166 
167  if not exists(self.work_path):
168  print('\n################### Preparing directory ###################')
169  print('Creating working directory: %s ...' % self.work_path, end=' ')
170  makedirs(self.work_path)
171  print('Done.')
172 
173  print('\n################# Downloading the files ###################')
174  total_size, files_to_download = get_size_to_download(self.work_path, paired_files_with_urls)
175  check_disk_for_space(self.work_path, total_size)
176 
177  ## Download needed files.
178  q = Queue()
179  show_status_bar.q = q
180  auth_download_file.q = q
181  auth_download_file.work_dir = self.work_path
182 
183  Process(target=show_status_bar, args=(total_size,)).start()
184  Pool(2).map(auth_download_file, files_to_download)
185  if total_size:
186  print("Done.")
187 
188  ## Create database
189  print('\n################# Preparing Database ###################')
190  if not self.db_name:
191  self.db_name = '%s___VS___%s.db' % (get_version(file_pairs[0][0]), get_version(file_pairs[0][1]))
192 
193  if self.clear_db:
194  print('Clearing DB: %s...' % self.db_name, end=' ')
195  open(join(self.work_path, self.db_name), 'w').close()
196  print('Done.')
197 
198  ## Compare file pairs.
199  self.db_name = init_database(join(self.work_path, self.db_name))
200 
201  # TODO: Use multiprocessing for this task.
202  for st_test_name in st_tests.split(','):
203  print('\n################# Comparing Releases (%s) ###################' % st_test_name)
204  st_test = tests[st_test_name]()
205 
206  some_files_compared = False
207  file_comparison_ids = []
208  if self.use_external_script_to_compare_files:
209  # Compare files using compare_using_files_v2.py
210  arg_list = [list(pair) + [self.work_path, partial_db_name(self.db_name, i),
211  self.clear_db] for i, pair in enumerate(file_pairs)]
212  pool = Pool(7)
213  pool.map(call_compare_using_files, arg_list)
214 
215  # Merge databases
216  print('\n################# Merging DBs (%s) ###################' % st_test_name)
217  for i, pair in enumerate(file_pairs):
218  tmp_db = partial_db_name(self.db_name, i)
219  print('Merging %s...' % (basename(tmp_db),), end=' ')
220  file_comparison_ids.append(merge_dbs(self.db_name, tmp_db))
221  remove(tmp_db)
222  print('Done.')
223  some_files_compared = True
224  else:
225  file_comparison = RootFileComparison(self.db_name)
226 
227  for file1, file2 in file_pairs:
228  # TODO: If files are not found desplay nice message.
229  # TODO: Maybe subprocces would control the unwanted reports of RootFileComparison.compare()
230  file1_path = join(self.work_path, file1)
231  file2_path = join(self.work_path, file2)
232 
233  if not file_comparison.was_compared(file1, file2, st_test_name):
234  print("Comparing:\n%s\n%s\n" % (file1, file2))
235  file_comparison_id = file_comparison.compare(file1_path, file2_path, st_test)
236  file_comparison_ids.append(file_comparison_id)
237  some_files_compared = True
238  else:
239  print("Already compared:\n%s\n%s\n" % (file1, file2))
240 
241  ## Calculate statistics for the release.
242  release1 = get_version(file_pairs[0][0])
243  release2 = get_version(file_pairs[0][1])
244  if some_files_compared:
245  release_comparison_id = self.was_compared(release1, release2, st_test_name)
246  conn = sqlite3.connect(self.db_name)
247  c = conn.cursor()
248  if not release_comparison_id:
249  print('Inserting release "%s VS %s" description.\n' % (release1, release2))
250  if not title:
251  title = "%s__VS__%s" % (release1, release2)
252  c.execute('''INSERT INTO ReleaseComparison(title, release1, release2,
253  statistical_test) VALUES (?, ?, ?, ?)''', (title,
254  release1, release2, st_test_name))
255  release_comparison_id = c.lastrowid
256  c.executemany('''UPDATE RootFileComparison SET release_comparison_id = ?
257  WHERE id == ?''', [(release_comparison_id, fid) for fid in file_comparison_ids])
258  conn.commit()
259  conn.close()
260 
start
Definition: start.py:1
utils_v2.search_on_disk
def search_on_disk
Definition: utils_v2.py:256
ValidationMatrix_v2.partial_db_name
def partial_db_name
Definition: ValidationMatrix_v2.py:71
utils_v2.check_disk_for_space
def check_disk_for_space
Definition: utils_v2.py:462
ValidationMatrix_v2.ReleaseComparison.no_url
no_url
Definition: ValidationMatrix_v2.py:128
compare_using_files_v2.RootFileComparison
Definition: compare_using_files_v2.py:38
ValidationMatrix_v2.ReleaseComparison.use_external_script_to_compare_files
use_external_script_to_compare_files
Definition: ValidationMatrix_v2.py:129
ComparisonHelper::zip
OutputIterator zip(InputIterator1 first1, InputIterator1 last1, InputIterator2 first2, InputIterator2 last2, OutputIterator result, Compare comp)
Definition: L1TStage2CaloLayer1.h:41
ValidationMatrix_v2.ReleaseComparison.work_path
work_path
Create working directory if not given.
Definition: ValidationMatrix_v2.py:124
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
ValidationMatrix_v2.ReleaseComparison.was_compared
def was_compared
Definition: ValidationMatrix_v2.py:131
ValidationMatrix_v2.ReleaseComparison.db_name
db_name
Download needed files.
Definition: ValidationMatrix_v2.py:125
ValidationMatrix_v2.ReleaseComparison.dry
dry
Definition: ValidationMatrix_v2.py:127
ValidationMatrix_v2.ReleaseComparison.clear_db
clear_db
Definition: ValidationMatrix_v2.py:126
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:19
ValidationMatrix_v2.ReleaseComparison.compare
def compare
Definition: ValidationMatrix_v2.py:142
utils_v2.recursive_search_online
def recursive_search_online
Definition: utils_v2.py:216
utils_v2.make_file_pairs
def make_file_pairs
Definition: utils_v2.py:108
utils_v2.init_database
def init_database
Utils.
Definition: utils_v2.py:391
ValidationMatrix_v2.merge_dbs
def merge_dbs
Definition: ValidationMatrix_v2.py:75
utils_v2.get_size_to_download
def get_size_to_download
Definition: utils_v2.py:446
def ValidationMatrix_v2.ReleaseComparison.was_compared (   self,
  release1,
  release2,
  st_test_name 
)

Definition at line 131 of file ValidationMatrix_v2.py.

References compare_using_files_v2.RootFileComparison.db_name, and ValidationMatrix_v2.ReleaseComparison.db_name.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

131 
132  def was_compared(self, release1, release2, st_test_name):
133  conn = sqlite3.connect(self.db_name)
134  c = conn.cursor()
135  c.execute('''SELECT id FROM ReleaseComparison WHERE release1=? AND
136  release2=? AND statistical_test=?''', (release1, release2, st_test_name))
137  release_comparison_id = c.fetchone()
138  conn.close()
139  if release_comparison_id:
140  return release_comparison_id[0]
141  return False
ValidationMatrix_v2.ReleaseComparison.was_compared
def was_compared
Definition: ValidationMatrix_v2.py:131
ValidationMatrix_v2.ReleaseComparison.db_name
db_name
Download needed files.
Definition: ValidationMatrix_v2.py:125

Member Data Documentation

ValidationMatrix_v2.ReleaseComparison.clear_db

Definition at line 126 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

ValidationMatrix_v2.ReleaseComparison.db_name

Download needed files.

Compare file pairs.

Create database

Definition at line 125 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare(), and ValidationMatrix_v2.ReleaseComparison.was_compared().

ValidationMatrix_v2.ReleaseComparison.dry

Definition at line 127 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

ValidationMatrix_v2.ReleaseComparison.no_url

Definition at line 128 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

ValidationMatrix_v2.ReleaseComparison.use_external_script_to_compare_files

Definition at line 129 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

ValidationMatrix_v2.ReleaseComparison.work_path

Create working directory if not given.

Definition at line 124 of file ValidationMatrix_v2.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

Generated for CMSSW Reference Manual by doxygen 1.8.5