rdkit.ML.Data.DataUtils module

Utilities for data manipulation

FILE FORMATS:

  • .qdat files contain quantized data suitable for

feeding to learning algorithms.

The .qdat file, written by _DecTreeGui_, is structured as follows:

  1. Any number of lines which are ignored.

  2. A line containing the string ‘Variable Table’

    any number of variable definitions in the format:

    ‘# Variable_name [quant_bounds]’

    where ‘[quant_bounds]’ is a list of the boundaries used for quantizing

    that variable. If the variable is inherently integral (i.e. not quantized), this can be an empty list.

  3. A line beginning with ‘# —-’ which signals the end of the variable list

  4. Any number of lines containing data points, in the format:

    ‘Name_of_point var1 var2 var3 …. varN’

    all variable values should be integers

Throughout, it is assumed that varN is the result

  • .dat files contain the same information as .qdat files, but the variable values can be anything (floats, ints, strings). These files should still contain quant_bounds!

  • .qdat.pkl file contain a pickled (binary) representation of the data read in. They stores, in order:

    1. A python list of the variable names

    2. A python list of lists with the quantization bounds

    3. A python list of the point names

    4. A python list of lists with the data points

rdkit.ML.Data.DataUtils.BuildDataSet(fileName)

builds a data set from a .dat file

Arguments

  • fileName: the name of the .dat file

Returns

an _MLData.MLDataSet_

rdkit.ML.Data.DataUtils.BuildQuantDataSet(fileName)

builds a data set from a .qdat file

Arguments

  • fileName: the name of the .qdat file

Returns

an _MLData.MLQuantDataSet_

rdkit.ML.Data.DataUtils.CalcNPossibleUsingMap(data, order, qBounds, nQBounds=None, silent=True)

calculates the number of possible values for each variable in a data set

Arguments

  • data: a list of examples

  • order: the ordering map between the variables in _data_ and _qBounds_

  • qBounds: the quantization bounds for the variables

Returns

a list with the number of possible values each variable takes on in the data set

Notes

  • variables present in _qBounds_ will have their _nPossible_ number read from _qbounds

  • _nPossible_ for other numeric variables will be calculated

rdkit.ML.Data.DataUtils.CountResults(inData, col=-1, bounds=None)

#DOC

rdkit.ML.Data.DataUtils.DBToData(dbName, tableName, user='sysdba', password='masterkey', dupCol=-1, what='*', where='', join='', pickleCol=-1, pickleClass=None, ensembleIds=None)

constructs an _MLData.MLDataSet_ from a database

Arguments

  • dbName: the name of the database to be opened

  • tableName: the table name containing the data in the database

  • user: the user name to be used to connect to the database

  • password: the password to be used to connect to the database

  • dupCol: if nonzero specifies which column should be used to recognize duplicates.

Returns

an _MLData.MLDataSet_

Notes

  • this uses Dbase.DataUtils functionality

rdkit.ML.Data.DataUtils.FilterData(inData, val, frac, col=-1, indicesToUse=None, indicesOnly=0)

#DOC

rdkit.ML.Data.DataUtils.InitRandomNumbers(seed)

Seeds the random number generators

Arguments

  • seed: a 2-tuple containing integers to be used as the random number seeds

Notes

this seeds both the RDRandom generator and the one in the standard Python _random_ module

rdkit.ML.Data.DataUtils.RandomizeActivities(dataSet, shuffle=0, runDetails=None)

randomizes the activity values of a dataset

Arguments

  • dataSet: a _ML.Data.MLQuantDataSet_, the activities here will be randomized

  • shuffle: an optional toggle. If this is set, the activity values will be shuffled (so the number in each class remains constant). Otherwise random values between the min and max of the dataset will be assigned

  • runDetails: an optional CompositeRun object

Note

  • _examples_ are randomized in place

rdkit.ML.Data.DataUtils.ReadGeneralExamples(inFile)

reads the examples from a .dat file

Arguments

  • inFile: a file object

Returns

a 2-tuple containing:

  1. the names of the examples

  2. a list of lists containing the examples themselves

Note

  • this attempts to convert variable values to ints, then floats. if those both fail, they are left as strings

rdkit.ML.Data.DataUtils.ReadQuantExamples(inFile)

reads the examples from a .qdat file

Arguments

  • inFile: a file object

Returns

a 2-tuple containing:

  1. the names of the examples

  2. a list of lists containing the examples themselves

Note

because this is reading a .qdat file, it assumed that all variable values are integers

rdkit.ML.Data.DataUtils.ReadVars(inFile)

reads the variables and quantization bounds from a .qdat or .dat file

Arguments

  • inFile: a file object

Returns

a 2-tuple containing:

  1. varNames: a list of the variable names

  2. qbounds: the list of quantization bounds for each variable

rdkit.ML.Data.DataUtils.TakeEnsemble(vect, ensembleIds, isDataVect=False)
>>> v = [10,20,30,40,50]
>>> TakeEnsemble(v,(1,2,3))
[20, 30, 40]
>>> v = ['foo',10,20,30,40,50,1]
>>> TakeEnsemble(v,(1,2,3),isDataVect=True)
['foo', 20, 30, 40, 1]
rdkit.ML.Data.DataUtils.TextFileToData(fName, onlyCols=None)

#DOC

rdkit.ML.Data.DataUtils.TextToData(reader, ignoreCols=[], onlyCols=None)

constructs an _MLData.MLDataSet_ from a bunch of text #DOC

Arguments

  • reader needs to be iterable and return lists of elements (like a csv.reader)

Returns

an _MLData.MLDataSet_

rdkit.ML.Data.DataUtils.WriteData(outFile, varNames, qBounds, examples)

writes out a .qdat file

Arguments

  • outFile: a file object

  • varNames: a list of variable names

  • qBounds: the list of quantization bounds (should be the same length

    as _varNames_)

  • examples: the data to be written

rdkit.ML.Data.DataUtils.WritePickledData(outName, data)

writes either a .qdat.pkl or a .dat.pkl file

Arguments

  • outName: the name of the file to be used

  • data: either an _MLData.MLDataSet_ or an _MLData.MLQuantDataSet_

rdkit.ML.Data.DataUtils.permutation(nToDo)