Shallow iterative networks

History / Edit / PDF / EPUB / BIB /
Created: August 18, 2017 / Updated: March 22, 2020 / Status: in progress / 1 min read (~195 words)

This is only a conceptual idea. The idea being that we've put a lot of effort into working with deep neural architectures, while I think it would be more appropriate to be able to work with shallow architectures. This idea stems from the duality of recursion and iteration, where one can be replaced by the other and vice-versa. In the case of deep learning, we can associate it with recursion, where each layer will bring more and more complexity as we go further down the network. Iterative learning would likely be based on auto-encoders, where the output is similar to the input in format so that it can be then fed into the network again. The questions then becomes:

  • When to stop?
  • What to do when we stop to get something out?
    • Do we pass it to another network that takes care of making this information available in a consumable format? In other words, convert the idea/thought format into textual/audio/visual format?

  • Is an iterative network different than a RNN where time is simply replaced by the evolution of the input?