NSB Entropy Estimation

NSB Entropy Estimation

The nsb-entropy project is devoted to implementation and practical use of the NSB algorithm for estimation of entropy and related information-theoretic quantities from undersampled discrete data. Originally, the algorithm was presented in

I Nemenman, F Shafee, and W Bialek. Entropy and inference, revisited. In TG Dietterich, S Becker, and Z Ghahramani, editors, Advances in Neural Information Processing Systems 14, Cambridge, MA, 2002. MIT Press. arXiv: physics/0108025. (PDF, 141973 bytes)

More details about the algorithm can be found in this collection of theory papers.

There are two implementations of the method (C++ and MatLab/Octave). The MatLab/Octave code (thanks to Fernando Montani for porting the original Octave source to MatLab/Octave compatible version) is pretty basic in its functionality, works cross-platform, but is not actively being developed anymore due to its slow speed as well as inability of Octave and MatLab to index discrete structures that can exist in a space with cardinality of more than 2^32 (which we routinely encounter in linguistics or neuroscience applications). The C++ code is much faster and is currently being developed. It has all the bells and whistles invented after the initial creation of algorithms and can transparently deal with complicated data files. As a cost, one needs to compile the code and install additional software libraries (GMP and GSL), both available under the GPL.

Both implementations can be downloaded from the project development/download pages, hosted by SourceForge. They are distributed under GPL v.2, the Free Software License.

The following is being requested from all users of either branches of the software:

Submit bug reports.
Submit changes you make to the code for incorporation into the main project tree.
If the software has been used for published and/or otherwise publicly available research, please submit publication data and other information, which will be recorded on this web site in the "This software has been used by..." section.
If you produce published research using this software and/or any other implementation of the NSB algorithm, please cite the following publication: I Nemenman et al. Entropy and information in neural spike trains: Progress on the sampling problem. Phys. Rev. E, 69:056111, 2004, together or instead of the original publication. Unlike the original publication, this later paper was published in a journal tracked by ISI and other citation indexes.

If you are willing to help with the ongoing development, please contact Ilya Nemenman. Currently, the most important contribution you can make is to port the software to Windows and/or Unix-like operating systems besides Mac OS X, which is the native OS for the project.

Thanks for your interest!

Ilya Nemenman

09/08/2005