Multi Segment Filter

[Bobijohn]

Hi Stan

The following is an observation based on experience running the Multi Segment filter. For descriptive simplicity I am going to describe the procedure I use. We will also consider just the first 5 columns.

For each column I pick the 5 most likely segments. I use a spreadsheet of 3125 lines to create the csv file. This plays all possible combinations of segments (5^5). For a pick 6/45 game this will take about 5 hours to run on a full package of 8.3 million tickets.

However, if I filter each column consecutively with the five selected filters using the Segment Combination Filter I will have a ticket set of about 800K. If I run the Multiple Segment Filter on this reduced set it will take about one tenth of the time – only 800K iterations instead of 8.3 million – i.e. parse 800k times through 3125 lines instead of 8.3 million.

So, my suggestion is that the Multi Segment Filter be expanded to automatically filter each column of the input array (thereby creating a pre-filtered set of tickets) before being fed into the existing algorithm. The results will be exactly the same. However, there is a substantial saving in run time especially on the large Lotteries.

Just a thought for your consideration.

Bobijohn

[stan]

i'm not sure i fully understood your suggestion. do you mean adding some extra conditions to the multiple segments filter?
the filter is applied in a single pass over the package tickets so i don't think there's an easy solution for this...

[Bobijohn]

Hi Stan

I think you may be reading more into my suggestion than is necessary.

do you mean adding some extra conditions to the multiple segments filter?

“extra conditions to the multi segment filter.â€￾ No not really. Let’s call it pre-filtering. The multi segment filter engine can be just as it is – only the input file is would be a reduced file (i.e. pre-filtered). The objective is to have the computer automatically do the pre-filtering prior to running the multi segment filter engine.

To accomplish this the computer would scan down each column of the input array, identify the different segments ( 3 legs – e.g. 2-3-1 = 1 segment) and then filter the column by those segments. It would perform this function sequentially for each column just as you would do manually (using the Segment Combination Filter) to reduce the size of the package. This would provide the reduced package (now pre-filtered) which would be used as the input to the multi-segment filter.

With the above in mind, perhaps if you re-read my first post it will make more sense. Needless to say, I will be happy to elaborate further since I believe this enhancement has substantial benefits in terms of reducing the long run times required for large full packages.

Please let me know what you think.

Thanks

Bobijohn

[stan]

ok, my corrected interpretation is:
- scan the input and make a list of unique segment combinations in each column
- filter the package and remove tickets that don't have any of the segment combinations from the step above in either column.

did i get it right this time?

[Bobijohn]

Yes Stan – I believe so.

Have the computer automatically do what you would do manually using the Segment Combination Filter to create the filtered ticket used to then create the csv input file to the Multi Segment Filter.

As I mentioned in my example above the computer run time can be reduced substantially – from just over 5 hours to 45 minutes on a pick 6/45 game and input csv of 3125 lines. This is a substantial difference and well worth serious consideration IMHO.

BTW – Please remember that version 1.06 of the Multi Segment Filter is giving incorrect results. Version 1.05 is fine though. Please refer to the appropriate post by Falcon.

Bobijohn

[stan]

[quote=Bobijohn]
BTW – Please remember that version 1.06 of the Multi Segment Filter is giving incorrect results. Version 1.05 is fine though. Please refer to the appropriate post by Falcon.
[/quote]

yep, it's on my list

[Falcon]

Stan,

We have two threads going on multi segment filtering - apologies for that. Given that there are some difficulties arising in implementing some suggestions might I with Bobijohn's concurrence suggest the following -

The combined Count and Filter activities remain separated as they are in plugin 1.0.5. Whilst the count and filter activity has issues when done under multithreading there are not issues when they are done separately. It is probably more desirable to have speed of process achieved under multi threading than by trying to combine the count and filter functions.

Bobijohn's suggestion above of a sort of pre-filter prior to a multi segment run offers most productivity in terms of time saved. I would regard this suggestion as the one that should receive priority in upgrading the multi-segment filter.

falcon

[stan]

unfortunately, there's similar problem with 'pre-filtering'. i can implement that but only as an extra button in the filter window you'll have to invoke manually. there's no support for this in the filter infrastructure - same as counting passed tickets during filtering.
let me know if you're ok with this solution and i'll start working on it.

another option would 'pre-filtering' on the fly when the filter is invoked. if a ticket doesn't have any segment combination from the input file then the rest of the filter wouldn't bother processing. it might speed the filtering up or slow it down depending how many combinations your input file has.

[Bobijohn]

Hi Stan

unfortunately, there's similar problem with 'pre-filtering'. i can implement that but only as an extra button in the filter window you'll have to invoke manually. there's no support for this in the filter infrastructure - same as counting passed tickets during filtering.
let me know if you're ok with this solution and i'll start working on it.

Yes – I am very happy with the above proposal. Thank you.

Just to confirm - there will still be one run to count tickets and a separate run to filter. Both runs will be multi threaded. There will be one .csv output file showing ticket counts just as before. Also, there will be a new button to initiate pre-filtering as above.

Bobijohn,