Need understanding on the output

[veenits]

Hi,

Brilliant work with putting all the algorithm implementations together. I'd love to contribute on this project. I have a question though. I need a bit more understanding on the output generated by AKOM (I've commented out the rest of the models and also, I'm just using the FIFA dataset)

---FIFA---
Number of sequences :   5000
Number of distinct items:   2612
Largest item id:    2775
Itemsets per sequence:  40.1966
Distinct item per sequence:     38.6178
Occurences for each item:   1.0408827017592923
Size of the dataset in MB:  20000.803932

Memory history: 286 281 311 454 935 623 432 933 476 962 510 990 490 988 988 
Max memory used: 990mb
Execution time: 8 seconds
        AKOM    
Success     26.659
Failure     73.341
No Match    0.46
Too Small   0.36
Overall     26.44
Size (MB)   6.532
Train Tim   0.499
Test Time   0.002

I have a series of inputs of the order -

1 4 2 5 3
1 3 5 2 3 2 1 5 3
1 5 3 
1 5 2 3

For this above test data, I would like to predict the next item in the sequence 1 4 3 2

I look forward to your inputs

[tedgueniche]

Hi,

Could you be more specific with your question? Are you asking me to describe the various outputs from an experiment (Sucess, Failure, No Match, Too Small) or are you asking me how to use the library to do a single prediction?

The output that you pasted in your message comes from using the Evaluator class. This class is used to evaluate and compare the performances of one or more prediction model. If you want to train a model manually and make a prediction, you should not use the Evaluator. Instead you can write a simple controller. I wrote one for you: https://gist.github.com/tedgueniche/fd9c6fde08119f6a3fa99330344f959d.

I hope this helps

[veenits]

Hi,

Sure. I'll try to be more specific. I have 2 different set of questions.

I.
If you can describe me the output that I posted in my previous comment, that would be really helpful. (The one with the --FIFA-- dataset). What does Success, Failure, No Match, Too Small, Overall mean? How would I say if one of my model works better than the other for the same given dataset?

II.
Thank you for the gist code. That really helped me understand how would I make a single prediction based on my training set. That is exactly what I was looking for. Also, now if I wanted to see the predictions from multiple models, I would

1. Initialize the model
   MarkovFirstOrderPredictor firstOrderModel = new MarkovFirstOrderPredictor();
2. Train my model
   firstOrderModel.Train(trainingSet);
3. Predict a new sequence
   Sequence predicted = firstOrderModel.Predict(stringToSequence(5, "1 4 3 2"));
4. Print the prediction.

Am I correct in mentioning the above 4 steps? Would you change/edit it? I can add as many models as I want to train my dataset and make them spit the predictions, right?

[tedgueniche]

Here are the definitions of each term from the output

Success: is the ratio of successful predictions against the number of
wrong predictions, it is defined as (Number of success) / (Number of success +
number of failure)

Failure: is the inverse of the local accuracy: 1 - (Success)

No Match: is the ratio of unsuccessful prediction against the total num-
ber of tested sequences: (Number of sequence without prediction) / (number of
sequence)

Too Small: is the ratio of sequences too small to be used in the experimentation, it counts any sequence with a length smaller than the parameter consequentSize.

Overall: is our main measure to evaluates the accuracy of a given
predictor. It is the number of successful prediction against the total number of
tested sequences. (Number of success) / ( number of sequence)

The three other measures are self explanatory.

[tedgueniche]

Concerning your approach to use multiple models, you are right. Here's a gist showing a more generic way to use multiple models: https://gist.github.com/tedgueniche/0530001bce15cf684074d1bcb00df851

[veenits]

Thank you for that insights. It helps. We can maybe put together all this information in the documentation for this project.

I have a question about the Too Small parameter. How do I set the consequentSize parameter? Currently, when I test the MarkovAllKPredictor, it gives me No Match/Too Small of 0.0

[tedgueniche]

When running an experiment, you should use or modify one of the existing profiles or make your own. A profile is a set of parameters for the Evaluator class and for one or more models. In the project, there is a profile for each included dataset and a default profile.

If you work on a new dataset called XYZ, you can simply create a XYZProfile.java class that ressemble DefaultProfile.java. The Evaluator class will automatically recognize and use the new profile.

If you get a "No Match/Too Small" of 0.0, it means all the sequences were longer than the consequentSize parameter.

[veenits]

Okay. Thank you for the pointer to Profiles. I was looking at a profile and I found this -

        //Global parameters
        //Pre-processing
        parameters.put("sequenceMinSize", "6");
        parameters.put("sequenceMaxSize", "999");
        parameters.put("removeDuplicatesMethod", "1");
        parameters.put("consequentSize", "1"); 
        parameters.put("windowSize", "5"); 

What does removeDuplicatesMethod, consequentSize and windowSize mean? I am using MarkovAllKPredictor as my algorithm. Are there any MarkovAllKPredictor specific parameters that I can configure?

[tedgueniche]

This Diagram (also available on the homepage of this repository) shows that each sequence typically has 3 parts; a context, a prefix and a suffix. The suffix is what a predictor tries to predict. The prefix is what the predictor uses to predict the suffix. Finally, the context is ignored. The consequentSize parameter defines the size of the suffix. The windowSize parameter defines the size of the prefix. Typically, a larger windowSize should give more accurate results. If you want to strictly predict the next item of a sequence, then you should use a consequentSize of 1. If you want to learn more about the consequentSize, have a look at the method Evaluator.isGoodPrediction

The parameter removeDuplicatesMethod can either be set to 1 or 2. When set to 1, consecutive repetitions are trimmed. When set to 2, all duplicates are removed so that each item become unique.
Here is an example:
original sequence: 1 1 2 3 3 4 1 2
with removeDuplicatesMethod set to 1: 1 2 3 4 1 2
with removeDuplicatesMethod set to 2: 1 2 3 4
(more details at line 148 of SequenceDatabase.java)

At the moment, the MarkovAllKPredictor has a single parameter order. You can either specify it in the model constructor:

//initializing Markov all-k-order with a max order of 2
MarkovAllKPredictor akom = new MarkovAllKPredictor("akom", "order:2");

or via a profile by adding the following line:
parameters.put("order", "5");

I hope this helps.