Information theory in machine learning

Jan 1, 2001
Source: Dilokthanakul et al. (2016)

Statistical inference and information theory are intimately related. So much so, that we typically jump from one to the other without even noticing (or paying attention to) the differences between them.

In part of my work I exploit this connection in two ways: using information theory to design better machine learning algorithms; and studying machine learning systems from the lens of information theory.

Machine learning
Pedro Mediano
Pedro Mediano
Coffee-powered beast-machine

Computational neuroscientist interested in synergy, information theory, and complexity.

Related

Publications

Deep active inference agents using Monte-Carlo methods
Active Inference just got real, thanks to Monte Carlo trees and information theory.
PDF Code
Deep active inference agents using Monte-Carlo methods
Learning, compression, and leakage: Minimizing classification error via meta-universal compression principles
Connections between information theory, supervised learning, and privacy
PDF
Relational forward models for multi-agent learning
Using graph neural networks to understand and improve multi-agent RL.
PDF
Relational forward models for multi-agent learning
The partial information decomposition of generative neural network models
Tracking the information decomposition of neural networks as they learn.
PDF DOI
The partial information decomposition of generative neural network models
Deep unsupervised clustering with Gaussian mixture variational autoencoders
Active Inference just got real, thanks to Monte Carlo trees and information theory.
PDF Code
Deep unsupervised clustering with Gaussian mixture variational autoencoders

Events

Synergy and information in complex neural systems
An ambitious research programme using complexity science to link brains and machines.
Mar 12, 2022 12:00 AM Department of Computing, Imperial College
Slides Video
Synergy and information in complex neural systems