BrainMeta'   Connectomics'  

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
> Long short-term memory, Question about a recurrent neural network architecture
oaks
post Mar 19, 2009, 08:51 AM
Post #1


Newbie
*

Group: Basic Member
Posts: 2
Joined: Mar 19, 2009
Member No.: 31975



Hi!
Maybe someone here can help me. I'm programming a long short-term memory (LSTM) neural network for my masters thesis. On paper LSTM sounds good, but from what I've seen I'm getting doubful. I'm having serious problems replicating the results in the original LSTM research papers. I've implemented an addition test, periodic function generation test and a gliding addition test, as described in the papers. In the addition test each sequence fed to the network consists of random numbers and the network has to remember two marked numbers and return their sum at the end of a sequence.

I found two bugs in the original lstm code and fixed them. When running my network without peephole connections and forget gates it produces exactly same results as the original lstm code.

The results I'm getting are qualitatively not as good as the ones described in the original papers. Also I haven't yet been able to get the network to learn the periodic function generation task at all. I've tried with all possible combinations of parameters I could think of.

My questions are: is this normal behavior? Is LSTM really this hard to train or is it more likely there is a bug in the original code and my code? Or am I just doing something wrong in the training?

Also, I haven't decided what to use the network on. A large part of my thesis will be applying the network on some problem. Any tips and ideas would be appreciated.

Sorry about the long rant smile.gif
User is offlineProfile CardPM
Go to the top of the page
+Quote Post
Rick
post Mar 19, 2009, 04:23 PM
Post #2


Supreme God
*******

Group: Basic Member
Posts: 5916
Joined: Jul 23, 2004
From: Sunny Southern California
Member No.: 3068



While I have had some experience with neural network programming, I am not familiar with LSTMs. I suggest you contact some of the paper authors and ask them directly. You can often find professor and grad student email addresses at university Web sites. Just drill down to the departments and you will often find contact information.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post
oaks
post Mar 19, 2009, 11:17 PM
Post #3


Newbie
*

Group: Basic Member
Posts: 2
Joined: Mar 19, 2009
Member No.: 31975



Ok thanks.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



Lo-Fi Version Time is now: 23rd November 2017 - 10:43 AM


Home     |     About     |    Research     |    Forum     |    Feedback  


Copyright BrainMeta. All rights reserved.
Terms of Use  |  Last Modified Tue Jan 17 2006 12:39 am

Consciousness Expansion · Brain Mapping · Neural Circuits · Connectomics  ·  Neuroscience Forum  ·  Brain Maps Blog
 · Connectomics · Connectomics  ·  shawn mikula  ·  shawn mikula  ·  articles