conddb2hdf5.DBPayloadIterator Class Reference

Inheritance diagram for conddb2hdf5.DBPayloadIterator:

Public Member Functions
def	__init__ (self, args, payloads)
def	__iter__ (self)
def	__next__ (self)

Private Attributes
	_args
	_cacheChunking
	_nextIndex
	_payloadCache
	_payloadHashs
	_payloadHashsIndex
	_safeChunkingSize

Detailed Description

Definition at line 157 of file conddb2hdf5.py.

Constructor & Destructor Documentation

__init__()

def conddb2hdf5.DBPayloadIterator.__init__	(		self,
			args,
			payloads
	)

Definition at line 158 of file conddb2hdf5.py.

    def __init__(self, args, payloads):
        self._args = args
        self._payloadHashs = payloads
        self._payloadCache = {}
        self._payloadHashsIndex = 0
        self._cacheChunking = 1
        self._safeChunkingSize = 1
        self._nextIndex = 0

Member Function Documentation

__iter__()

def conddb2hdf5.DBPayloadIterator.__iter__

(

self

)

Definition at line 166 of file conddb2hdf5.py.

    def __iter__(self):
        return self

__next__()

def conddb2hdf5.DBPayloadIterator.__next__

(

self

)

Definition at line 168 of file conddb2hdf5.py.

References mps_alisetup.SetupAlignment._args, tkal_create_file_lists.FileListCreator._args, conddb2hdf5.DBPayloadIterator._args, conddb2hdf5.DBPayloadIterator._cacheChunking, conddb2hdf5.DBPayloadIterator._nextIndex, conddb2hdf5.DBPayloadIterator._payloadCache, conddb2hdf5.DBPayloadIterator._payloadHashs, conddb2hdf5.DBPayloadIterator._payloadHashsIndex, conddb2hdf5.DBPayloadIterator._safeChunkingSize, and conddb2hdf5.canonicalProductName().

    def __next__(self):
        if self._nextIndex >= len(self._payloadHashs):
            raise StopIteration()
        payloadHash = self._payloadHashs[self._nextIndex]
        if not self._payloadCache:
            self._cacheChunking = self._safeChunkingSize
            queue = mp.Queue()
            p=mp.Process(target=external_process_get_payloads_objtype_data, args=(queue, self._args, self._payloadHashs[self._payloadHashsIndex:self._payloadHashsIndex+self._cacheChunking]))
            p.start()
            table = queue.get()
            p.join()
            #table = get_payloads_objtype_data(session, payloadHashs[payloadHashsIndex:payloadHashsIndex+cacheChunking])
            #print(table)
            self._payloadHashsIndex +=self._cacheChunking
            for r in table:
                self._payloadCache[r[0]] = (r[1],r[2])
        objtype,data = self._payloadCache[payloadHash]
        if len(data) < 1000000:
            self._safeChunkingSize = 10
        del self._payloadCache[payloadHash]
        self._nextIndex +=1
        return DBPayload(payloadHash, canonicalProductName(objtype.encode("ascii")), data)

Member Data Documentation

_args

conddb2hdf5.DBPayloadIterator._args

private

Definition at line 159 of file conddb2hdf5.py.

Referenced by conddb2hdf5.DBPayloadIterator.__next__(), conddb2hdf5.DBTag.dataProducts(), conddb2hdf5.DBDataProduct.payloads(), and dqm-mbProfile.Profile.update().

_cacheChunking

conddb2hdf5.DBPayloadIterator._cacheChunking

private

Definition at line 163 of file conddb2hdf5.py.

Referenced by conddb2hdf5.DBPayloadIterator.__next__().

_nextIndex

conddb2hdf5.DBPayloadIterator._nextIndex

private

Definition at line 165 of file conddb2hdf5.py.

Referenced by conddb2hdf5.DBPayloadIterator.__next__().

_payloadCache

conddb2hdf5.DBPayloadIterator._payloadCache

private

Definition at line 161 of file conddb2hdf5.py.

Referenced by conddb2hdf5.DBPayloadIterator.__next__().

_payloadHashs

conddb2hdf5.DBPayloadIterator._payloadHashs

private

Definition at line 160 of file conddb2hdf5.py.

Referenced by conddb2hdf5.DBPayloadIterator.__next__(), and conddb2hdf5.DBDataProduct.payloads().

_payloadHashsIndex

conddb2hdf5.DBPayloadIterator._payloadHashsIndex

private

Definition at line 162 of file conddb2hdf5.py.

Referenced by conddb2hdf5.DBPayloadIterator.__next__().

_safeChunkingSize

conddb2hdf5.DBPayloadIterator._safeChunkingSize

private

Definition at line 164 of file conddb2hdf5.py.

Referenced by conddb2hdf5.DBPayloadIterator.__next__().