High CPU usage issue

[tdnl46w]

Hi stan.

There is an Expert Lotto file called "tempFiles.properties", that appears to get corrupted by writing to many temp files after switching between multiple lotto's. This will cause the program to use high CPU resources and the program becomes almost unusable. However, if I delete the contents of this file(using Notepad on Windows 10) the issue will stop, and the CPU usage goes back to normal. Can you confirm this issue/bug? Thanks. Also, if anyone else wants to add any information, feel free to reply.

[stan]

Are you saying that there's so much content in that file in your case so Expert Lotto has performance problems? If that's the case how large that file becomes? And how many different lotteries do you have installed?:)

[tdnl46w]

Are you saying that there's so much content in that file in your case so Expert Lotto has performance problems? If that's the case how large that file becomes? And how many different lotteries do you have installed?:)

Yes. The file size is usually around 600KB. I currently have 25+ lotto's installed, but this seems to happen with any number of lotto's. The "tempFiles.properties" file has lotto information that usually gets deleted/updated each time the program is closed. However, when switching between lotto's in the program or possibly loading a saved complex filter(?), the file can randomly generate thousands of stats text lines that do not get deleted on program close. Examples below.

This text line gets deleted/updated on close and does not cause the issue.
C\:\\FileFolder\\EL\ FileFolder\\LottoFileFolder\\tmp\\temp4172101749010479412.t539=true

This stats text line does not get deleted/updated on close and causes the high CPU usage when there are thousands of them(e.g. 600KB).
C\:\\EL\ FileFolder\\LottoFileFolder\\tmp\\stats3970561476534239176.tmp=true

I used different names for the folders to make it easier to read, and the numbers after "temp"/"stats" will vary for other lines in the file.

[stan]

Thanks for the details, I'll look into it.

[tdnl46w]

Hi stan.

I found a way to reproduce the high CPU usage issue. The filter setups below are using the EL Demo 6/49 Lotto;...

1. Create a Combined Statistics filter for 'Distance';...

Distance.jpg

2. Then create a new Complex Filter with a single 'Statistical Filter';...

Statistical Filter.jpg

3. Insert a full wheel package, and then right click on the statistical filter and select 'Back-test' using all 960 draws.

When it's finished, the filter will write thousands of .tmp lines to the-> "tempFiles.properties" file. The file size will also increase each time you back-test this filter. The next time you open Expert Lotto, the program appears to try and reduce this file, which takes very high CPU resources depending on the file size.

Edit to add...
Combined Statistics:
Number Movement 1
Number Movement 2
Number Movement 3
Number Movement 4

...will cause the same high CPU usage issue. Also, in both of the above combined statistics setups, combining five of the same statistic doesn't work.

[stan]

Thanks for the steps to reproduce this issue. During back-test or performance testing there can indeed be many temporary files created to hold partial results. However once the calculation is done or finished those files are removed.
Since it's obviously not happening in your case there must be some problem that prevents the deletion of those files so Expert Lotto does the cleanup next time it starts. Can you please email the whole content of Application Log window? Use menu View - Application Log to display and then copy/paste the content here or into an email, thanks. Maybe you have some unexpected version of Java where there's a problem with the file clean up...

[tdnl46w]

During back-test or performance testing there can indeed be many temporary files created to hold partial results. However once the calculation is done or finished those files are removed.

When I reopen EL, it does try to clean up the temp file, as I can see the file size shrinking slowly. However, because there can be thousands of lines in that temp file, the CPU/system uses a lot of resources to remove them. Eventually the file size will go back to normal(1k), but this can take many hours on a slower system.

Maybe you have some unexpected version of Java where there's a problem with the file clean up...

Latest Windows 10 Home Version 22H2
Latest Java 1.8.0_381(64-Bit)
Latest Expert Lotto 5.11[230920](Registered Version)
Latest Expert Lotto Development Version 5 DEV 230920(same issue)

This issue happened from much earlier versions of all of the above as well, so I'm guessing it's a specific filter setup issue(see Note: below).

Have you tried to reproduce this issue on your system? If your system doesn't have this issue, then something on my system is not compatible or it's just to slow to clear the temp file before I start using the program again. My system very outdated by today's standards(FX-6300 CPU/16Gb RAM) so that could be the issue, however it has been very stable for other graphic/CPU intensive applications and games.

Note:There is also the issue with both combined statistics in my above replies, that if you have 1 to 2 combined it works,...but if you combine 3 to 4, then temp file has the issue. If you try to combine 5, both will fail with an error. So the program is working to remove the temp file, but the issue seems to be specifically with the filter setup(s). If you still need the App Log info let me know. Thanks.

edit: Updated program version numbers.

[stan]

I have verified that temp files get deleted even when using the same Java version that you have (Java 8 update 381). The file is removed as soon as the application is done with the stats calculation.
If you are seeing many undeleted files on the next Expert Lotto startup then it means the application can't delete them for some reason. The combined stats only make the problem more prominent as there are *many* such temp files created to calculate the results.
Can you email me your application log after running those combined stats? Also, what kind of antivirus software do you use?

[tdnl46w]

The only file causing the issue is a hidden AppData file in the following directory: C:\Users\Name\AppData\Roaming\.expertlotto\release\config\Preferences\com\expertlotto\core\api\tempFiles.properties. When you back-test the combined statistics setups from the replies above, the "tempFiles.properties" file will have thousands of text lines written to it. Each time you back-test the above setups, the file's size can double(90k to 180k, then 360k...etc.).

When you exit and then reload Expert Lotto, the program will try to delete the contents of the "tempFiles.properties" file(this can be seen in real-time if you watch the file's size in the directory window). If there are hundreds of thousands of lines in the file, EL will use the systems resources(e.g. CPU) to go through the file and delete the contents. This will cause EL to run very slow. The Application Log shows no changes or errors when this issue occurs(sent PM about the log).

I'm guessing this is a bug with the program, because as noted in my last reply for those specific combined statistics,...if you have 1 to 2 combined it works,...but if you combine 3 to 4, then the "tempFiles.properties" file has the issue. If you try to combine 5, both of the above combined statistics setups will fail with an error(Warning ⚠ There was an error while executing back-test).

[stan]

It's not a bug:) That properties file is used to track temporary files that the stats framework creates during calculation. If the deletion of a temp file fails for whatever reason then Expert Lotto will remove that file next time it starts (it reads its file name from that properties file). Without this tracking you'd end up without any disk space left very soon because it would be filled up with orphaned temporary files.
Can you email me your application log after running your combined stats? Perhaps there will be some debug messages that will tell me why the regular cleanup doesn't work in your case.

[tdnl46w]

Hi stan.

The bug/issue I'm trying to describe has something to do with how those specific combined statistics write to the tempFiles.properties file. It seems to write the same amount of lines as the draws you use in a back-test. So if you use 960 draws in a back-test, the combined filter setup will write 960 lines to the tempFiles.properties file each time you run the back-test in the same program session. On separate program sessions, running another back-test may add to whatever is leftover from the last session(etc.). I have emailed my application log and added some more examples below of what I recently tested;

Ex 1. Using two(or three) combined, the tempFiles.properties file does not seem to be affected by the amount of back-tests or draws used. CPU usage is normal.
Combined Statistics:
Distance 2-1
Distance 3-2

Ex 2. However, using four combined, it will write the same amount of lines as draws used in the back-test. The examples in my last replies were using all 960 draws from the EL Demo Lotto 6/49.
Combined Statistics:
Distance 2-1
Distance 3-2
Distance 4-3
Distance 5-4 -> So if you run a back-test with this setup 6 times in a row with 960 back-test draws, then it will write approximately 5760 lines to the tempFiles.properties file. On the next program startup, the CPU usage for EL can be 10% or higher and EL will be noticeably slower.

Ex 3. Using five combined is definitely a bug.
Combined Statistics:
Distance 2-1
Distance 3-2
Distance 4-3
Distance 5-4
Distance 6-5 -> Back-test 1 time and a pop up message window will show,..."Warning ⚠ There was an error while executing back-test". The issue is the same for combining the Number Movements.

It took me a while to track what was causing my system to slow down because when it would happen, I would usually just reinstall EL to solve it.

[stan]

Thanks for the log file however there's nothing abnormal in it. In the next app update I'll log more errors that the temp file clean up may throw so hopefully I'll be able to see what's wrong in your case then.

[tdnl46w]

Hi stan. Thanks for looking into the issue.

The application log file I sent only relates to the tempFiles.properties issue, it does not contain the error that happens when you combine five statistics. So if you setup the following from my last reply...

Ex 3. Using five combined...
Combined Statistics:
Distance 2-1
Distance 3-2
Distance 4-3
Distance 5-4
Distance 6-5 -> Back-test 1 time and a pop up message window will show,..."Warning ⚠ There was an error while executing back-test". The issue is the same for combining five Number Movements.

This does show errors in the application log file, and it may or may not be related to the temp file issue(?).

[stan]

Ok, I missed that. Please email your application log file with that error and I'll see if it's related to the temp files being orphaned on your disk.