These are examples of the solvers used with DeepHisto

solver.ff.prototxt --> solver for field fraction study
solver.nd.prototxt --> solver for the neuronal density study

Where to put them
==================

I put the solver under

deephisto/<study>/caffe/solver/

So for the neuronal density study for example it looks like this:

/home/dcantor/projects/deephisto/neuronal_density/caffe/solver/solver.nd.prototxt


How to run caffe with a solver
===============================

Then I execute it like:

cd /home/dcantor/projects/deephisto/neuronal_density/caffe
caffe train --solver=solver/solver.nd.protxt --gpu=3

What are snapshots and how to use them
=======================================
Snapshots are generated by caffe to save the state of the network (weights) as training evolves. The idea is to be able to recover from a point in training if something goes wrong. Thus you can resume training.

An example of how to start training from a snapshot:

caffe train --solver=solver/solver.nd.protoxt --gpu=3 ----snapshot=/data/dcantor/deephisto/nd/_iter_525000.solverstate

Where to store snapshots
========================
I moved the snapshots to the /data partition in grumpy since I was running out of space. However when you finish training you can copy and paste the last snapshot into your path:

deephisto/<study>/caffe/data/<network_name>

this way you can use it to visualize the results. Let me show you how...


testing your network
=====================

cd deephisto/code/ 

then:

python interpreter.py -c <configuration file>

for example:

python interpreter.py -c config_neuronal_density.ini


This will show you the following menu

Study NEURONAL DENSITY
-------------------------------

Welcome to the network observer. Type man to show this message
------------------------------------------------------------------------------

 Commands: 

   data  [dir]                 : sets the data directory (where patches are)

   Single patch:
   =============
   load [directory]  [epoch]   : loads a trained network
   ping [patch]                : gets a random prediction from the validation set
   net                         : shows the structure of the network


   Panel:
   ======
   peek [directory] [epoch]    : loads the panel for a given epoch
   epoch [e]                   : sets the current epoch to e
   rand                        : changes the data in the panel


   Animation:
   ==========
   next                                     : next epoch   
   play [directory] [start] [stop]  [step]  : sets up an animation of the net through its epochs


   Subject:
   ========
   run [subject] [slice] [avg|max|med|mod] : runs the network on this slice patcy-by-patch

   man                                     : shows this message                          
   exit                                    : exits the interpreter
------------------------------------------------------------------------------

1. The first thing you need to do is to set your *data* directory. DeepHisto will look for this directory under deephisto/<study>/patches/ 
This name is the same as the one in the configuration file under:

[patches]
name                : 28x28
 
You set the data directory with:

> data 28x82 <-- enter

when you hit enter, deephisto will confirm the directory selected and show you the size of the training and validation sets. 

Note
-----
Remember that such sizes can be configured in the configuration file, however you would have to run step 6 (create dataset) for the new partition between training and validation to take effect. Anyways, after executing:

data 28x28 <-- enter

you will see:

data dir set to /home/dcantor/projects/deephisto/neuronal_density/patches/28x28

size training set    :   630567
size validation set  :   207955

2. The second thing you need to do is to load the network and the snapshot.
You do this by:
> load <network> <snapshot number>

for example:
> load nd 1800000

Note
----
Remember that the network name is configured in the configuration file under:
[network]
name                : nd

Since in my example I am stopping training after 1800000 iterations. I should have the snapshot:

_iter_1800000.solverstate

Remember that I generated the snapshots in the /data partition? I need to copy this snapshot to deephisto/<study>/caffe/data/<network name> before being able to use it because that's where deephisto will look for it. I am assuming here that you already checked the 'where to store snapshots' in this document.

> load nd 1800000 <-- enter

will show something like:
Average training image /home/dcantor/projects/deephisto/neuronal_density/dataset/28x28/training_average.png loaded
directory: nd epoch: 1800000  data:28x28

3. Let's take a look at some predictions. You now are ready to invoke the *peek* command

> peek <-- enter

this will show you selections from the validation set with the corresponding predictions.

Note
-----
you can select predictions from a different snapshot as long as that snapshot is available in deephisto/<study>/caffe/data/<network name> 

4. you can change the selection of patches for predictions with the command *rand*

> rand <-- enter