Package - Duplicates (Remove / Keep)

[Meisterkeks]

Hello Stan,

I have a package with a total of 15,421,760 tickets. An analysis of the 15,421,760 tickets gives this overview:



These tickets were created from 8 individual filters and then merged with "Append". This allows tickets to occur x times (maximum 8x).

I have to choose the approach because extensive complex filters (e.g., 6 out of 8, 7 out of 8, etc.) of EL either can not be computed (ERROR message) or take much too long (e.g., 1 -2 days).

If I calculate all 8 filters individually and save the results, this is very fast. The results are then merged with "Append". From the overall result are now different tickets for further use interesting / important:

Tickets 1x
Tickets 2x
Tickets 3x
Tickets 4x etc.

Depending on the filter structure / strategy, I then have to use ticket combinations:

Tickets 1x + tickets 2x
or
Tickets 1x + Tickets 3x + Tickets 4x
or
Tickets 2x + tickets 3x
etc….

When I filter the package with 15,421,760 tickets with package duplicates I get the following new packages:
Remove: 8.261.140
Keep: 10.733.195

My request:
Can the feature be extended to get the following results:

Keep 1x
Keep 2x
Keep 3x
Keep 4x
etc. (until ... ??)

Neither EXCEL nor ACCESS have managed to put these tickets together. I do not know any other alternative software.

[Meisterkeks]

addendum:

I now have several strategic filters based on draw results of 1 day or 1 to 4 days. The filter results are tickets between 2 million and 7 million (Lotto 6 out of 49 total = 13.9 million). Again, I have the same situation that I could again (significantly) improve the filter efficiency, if I could match the filter results (packages) directly with each other.

[therealpoorboy]

That would be a great addition to the "Duplicates Keep /Remove" function
By being able to 'user define' the number occurrence of duplicates in file of combinations would be very powerful!

[stan]

How about using Mutual Match filter for that? It does have min/max option.

[Meisterkeks]

Hello Stan,

I can not reduce the tickets accordingly with this filter. The example in the documentation does not use identical tickets:

Mutual Match

This filter opens from menu Package - Reduce - Mutual Match. The filter compares all tickets in the Package with each other. If there are for example 10 tickets in the Package then the first ticket will be compared with tickets 2 to 10, the second ticket will be compared with tickets 3 to 10, the third ticket will be compared with tickets 4 to 10 etc.
Example

Let's assume the Package contains the following tickets:

01-02-03-04-05-06
01-02-03-08-09-10
01-02-03-31-32-33
01-22-23-24-25-16
01-22-23-24-25-26

Filtering with settings Match 3 will leave only two tickets in the Package - the first one (01-02-03-04-05-06) as it matches the following two tickets in three numbers 1, 2 and 3 and the second one as it matches the following ticket in the same three numbers.
Filtering with settings Match 3, Min 2 will leave only the first ticket in the Package as it matches two other tickets in the Package in three numbers.
Filtering with settings Match 3, Max 1 will leave only the second ticket in the Package as it matches one other Package ticket in three numbers.


In my different packages but many identical tickets are included and after the reduction, each ticket should only be a maximum of 1x in the package:

before reduction:
01-02-03-04-05-06
01-02-03-04-05-06
01-02-03-08-09-10
01-02-03-31-32-33
01-02-03-31-32-33
01-22-23-24-25-16
01-22-23-24-25-16
01-22-23-24-25-26

after reduction:

01-02-03-04-05-06
01-02-03-08-09-10
01-02-03-31-32-33
01-22-23-24-25-16
01-22-23-24-25-26

[stan]

So what filter settings did you choose to get those results?

[Meisterkeks]

Hello Stan,

I use a variety of filters with very different structures, for example:









I have created these filters based on extensive analysis from all draws. Sometimes I have very long computation times of these filters. From these filters, I create for each complex filter from all winning numbers (= 13,983,816 tickets) one to three different calculation results (= packages) and save them as .csv file. From these many packages can be determined very efficiently, which tickets at the next draw a "high hit probability" have. For this I need the comparison of different packages.

And right here I have the problem that I can compare only 2 packages (Duplicate Remove or Keep). It would be very helpful if i Could compare 5 packages. So all 5 packages are merged into one package (with append). Then it should be determined which tickets in the new package either

only 1x
or 2x
or 3x
or 4x
or 5x
are included.

This would require help from EL.

[stan]

I thought you tried 'Mutual Match' filter and it didn't work for you. I think that filter should be able to find the multiple duplicates that you're looking for.

[Meisterkeks]

Hello Stan,

I had tested the Mutual Match Filter a few times, but I never had the desired result. Maybe I have not understood this correctly.

To make my question easier to understand, I have now created a simple example. I only use the tickets Index Range 1 to 1,900 for this. With this index range, the example should be equally applicable to all lotteries.

Here is the complete overview of the results I am looking for, by keeping or merging several packages:



I think this makes it clear that it is irrelevant to the results how many 3s, 4s, 5s, 6s are in the packages. This is perhaps the crucial difference to the Mutual Match Filter approach.

I have provided the corresponding EXCEL overview and all packages for download in the systems. I hope that everything works and my question becomes clearer and a solution is found with EL.

[Falcon]

Hi Meisterkeks,

Quite some time ago Stan wrote some code in a language called Perl which had the ability to seek duplicates across multiple packages - and all done at the one pass of the code.

The discussion and Stan's code can be found in this thread viewtopic.php?f=8&t=1010&start=15 - perhaps read through some of the subsequent posts to see if this application performs the duplicate findings you are after.

To make use of the code you will need to have the Perl compiler software within your PC - not sure how you obtain this - perhaps a google search. I believe it was open source (and so free) at the time this was discussed in the forum and it may still be.

Stan's code is independent of EL5 - just a matter of having the packages in the correct location for the app to find them.

cheers

falcon

[Meisterkeks]

Hello Falcon,

Thanks for pointing out, that could solve my questions. I have Perl installed on my PC.




I've been trying to figure out how and where to enter the code from Stan. As a layman, I'm currently completely at a loss. I'm not clear with the descriptions. Are there any hints how to enter and use the code?

[Falcon]

Meisterkeks,

It is quite sometime since I have used this Perl utility as I am now quite happy with the EL5 remove duplicates process but admittedly this current EL5 process is no more than one package compared to one other package to find their duplicates. The benefit of the Perl app being able to compare one package to several others at the one pass. I have unfortunately now forgotten all the fine detail necessary to put this process into action.

But there is a very detailed commentary in the thread source I posted with Stan providing several iterations of his code along the way, the final code appearing at about Page 5 of that thread. If you haven't followed the thread through I suggest you do so and in there I believe you will find all the necessary paths and locations for packages etc. Very good input throughout the thread is provided by poster Bobijohn and in fact he started the whole process off with some concepts earlier in the the thread which are also worth a read.

My apologies but I cannot offer any more than this now. It is clear from the various iterations of Stan's code that one needs to be quite careful of actual content and whether certain lines of code need to be included or not. Some others skilled in Perl may be able to offer assistance here.

cheers

falcon

[Meisterkeks]

Hello Falcon,

so I have read something. Basically, this should be the solution. I will try to find the appropriate solution for me by adapting the patterns accordingly. I think that will be difficult - but I will practice.

Thank you very much for this very interesting cross reference.

[tdnl46w]

With 'Match Tickets in File' and 'Mutual Match'(as stan replied) you can specify how many duplicates a ticket can have in a package. 'Match Tickets in File' compares tickets of the file with the package. 'Mutual Match' compares tickets of the package only. Examples using the EL demo lotto 6/49;

- Match 6: Min 2| Max "blank" = keeps tickets that have 2 or more duplicates.
- Match 6: Min 3| Max 3 = keeps tickets that have exactly 3 duplicates.
- Match 6: Min 1| Max 5 = keeps tickets that have 1 to 5 duplicates.

The best method I found(so far) using EL to find duplicates or range of duplicates, is to set up multiple 'Match Tickets in File' in a complex filter with different Min/Max values(Min:1/Max 1, Min 2/Max 2 etc.). A similar strategy to finding common ticket numbers across different packages, you can combine packages into one and then use Package>Reduce>Best Tickets...which will find the best/worst number combinations of a package, including ticket duplicates.

[Meisterkeks]

Hello,

interesting, with the filter "Match Ticket in File" could work too. However, I have not understood how to make the settings.

Concrete example with three sample files:

File Pack1 = 1,000 tickets; Ticket Index Range 1 - 1,000

File Pack2 = 1,000 Tickets: Ticket Index Range 601 - 1,600

File Pack 1 and 2 Merge: 1,600 tickets; Ticket Index Range 1 - 1,600

Complex filter

Filter 1 Match Ticket in File: Pack1

Filter 2 Match Ticket in File: Pack2

Goal: Accept the tickets Index Range 601 - 1.000, which are included in both files.

I have tested the two filters several times in Match 6/6 + 1 with different settings: either all tickets were always accepted or none. I filled the package for evaluation with "File Pack 1 and 2 Merge" (= 1.600 tickets).



What am I still doing wrong. The documentation of EL unfortunately did not help me either.