Dramatic increase in runtimes


Message boards : Number crunching : Dramatic increase in runtimes

Message board moderation

To post messages, you must log in.
AuthorMessage
Anthony Ayiomamitis

Send message
Joined: 12 Dec 15
Posts: 6
Credit: 507,840
RAC: 0
Message 4748 - Posted: 5 Jan 2016, 16:22:02 UTC
My tasks running on three different machines have suddenly shown a huge increase in runtime and which is nearly double.

Has anyone else noticed something similar?
ID: 4748 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Presrvd
Avatar

Send message
Joined: 15 Jun 15
Posts: 16
Credit: 123,015,840
RAC: 0
Message 4749 - Posted: 5 Jan 2016, 18:43:06 UTC - in response to Message 4748.  
I have the same thing going on with my machines. Seems pretty recent, though. Looks like my machines started doubling their WU times yesterday...
ID: 4749 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Anthony Ayiomamitis

Send message
Joined: 12 Dec 15
Posts: 6
Credit: 507,840
RAC: 0
Message 4750 - Posted: 5 Jan 2016, 19:34:12 UTC - in response to Message 4749.  
Yes, it involves units downloaded as of yesterday.
ID: 4750 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile cliff
Avatar

Send message
Joined: 19 Nov 14
Posts: 93
Credit: 30,066,240
RAC: 0
Message 4752 - Posted: 6 Jan 2016, 3:23:52 UTC - in response to Message 4750.  
Yes, it involves units downloaded as of yesterday.

too true, major increase in run times and for some reason the server is NOT validating completed WU, I've several with both parties showing complete awaiting validation. However that has only been evident today [06-01-16].

Regards,
Regards,
Cliff
ID: 4752 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile darkstarz1

Send message
Joined: 8 Nov 12
Posts: 30
Credit: 63,033,198
RAC: 13,271
Message 4754 - Posted: 6 Jan 2016, 11:13:33 UTC

Last modified: 6 Jan 2016, 11:14:09 UTC
I've seen something similar; also now have 22 wu's awaiting validation, but server status says validator running normally, and no wu's awaiting validation. Somethng broken?
ID: 4754 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Presrvd
Avatar

Send message
Joined: 15 Jun 15
Posts: 16
Credit: 123,015,840
RAC: 0
Message 4758 - Posted: 6 Jan 2016, 22:30:56 UTC - in response to Message 4754.  
Looks like someone kick-started the validator. My queue of WUs is processing at a decent speed so far. Now, if they could only explain why the WUs are taking so much longer to actually compute....
ID: 4758 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
hsdecalc

Send message
Joined: 1 Sep 15
Posts: 2
Credit: 6,166,451
RAC: 1,310
Message 4764 - Posted: 13 Jan 2016, 11:19:25 UTC
I have also similar problems with some WU for NVIDIA-GPU on WIN10.
There are some WUs which hangs. The expired time is counting, but no work on the GPU will be done.
I cancel one WU after 3.5 hours with no power using.
The second one I see, I did suspended. The next WU runs without any problems (16 min).
I resume the suspended WU and it runs also thru to the end!?

Here are some more data:
<wu_name>ps_151223_input_4269_30</wu_name>
</global_preferences>
<app_file>period_search_10112_windows_x86_64__cuda55.exe</app_file>
<app_file>cudart64_55.dll</app_file>
</app_init_data>

Stderr.txt:
CUDA RC12!!!!!!!!!!
CUDA Device number: 0
CUDA Device: GeForce GTX 980 Ti
Compute capability: 5.2
Multiprocessors: 22
Grid dim: 352 = 22*16
Block dim: 128
ID: 4764 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile cliff
Avatar

Send message
Joined: 19 Nov 14
Posts: 93
Credit: 30,066,240
RAC: 0
Message 4765 - Posted: 13 Jan 2016, 23:00:10 UTC - in response to Message 4758.  
@Presrvd
Hi,
My 'guess':-) More data/more complex data in = more cpu time required per WU = greater time to process a WU.
I think there was a post from Radic about using some new and more detailed data a while back.
I've noticed for a while that some batches of data take longer, others shorter times to complete. In fact anywhere on my rig from about 1 hour to 1 hr 30+ mins

It seems to depend on the batch of WU and I assume each batch is for a diff asteroid. Some of which will have multi axis rotation some not so much:-)

Still it would be nice if tasks that use more computer cycles garnered more CR:-) Instead of all tasks getting the same no matter what duration.

I do 3 projects, two on GPU and A@H on cpu an AMD9570 running at 4.958GHz
and I do 2 x WU on the cpu at a time.

My Tuppence worth:-)
Regards,
Cliff
ID: 4765 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Presrvd
Avatar

Send message
Joined: 15 Jun 15
Posts: 16
Credit: 123,015,840
RAC: 0
Message 4766 - Posted: 14 Jan 2016, 0:44:32 UTC - in response to Message 4765.  
My real issue isn't really the credits not changing, even though the workload is. I'm more annoyed with the BOINC client not updating with the WU processing times. I'm now starting to lose entire WUs (which just means wasted processor, and electricity) because my BOINC client is still pulling down WUs based on the old calculating times. It's almost like Asteroids@home has times that are propagating in place of my calculating time. In the last two days, I've lost 30 WUs, and over .5 million processing seconds. That, and the severe lack of communication on this project are frustrating.
ID: 4766 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile cliff
Avatar

Send message
Joined: 19 Nov 14
Posts: 93
Credit: 30,066,240
RAC: 0
Message 4767 - Posted: 15 Jan 2016, 3:53:25 UTC - in response to Message 4766.  

Last modified: 15 Jan 2016, 3:53:51 UTC
My real issue isn't really the credits not changing, even though the workload is. I'm more annoyed with the BOINC client not updating with the WU processing times. I'm now starting to lose entire WUs (which just means wasted processor, and electricity) because my BOINC client is still pulling down WUs based on the old calculating times. It's almost like Asteroids@home has times that are propagating in place of my calculating time. In the last two days, I've lost 30 WUs, and over .5 million processing seconds. That, and the severe lack of communication on this project are frustrating.


yeah, well the bloke running the project has his hands full trying to earn a crust and keeping the project going. Some folks who said they were going to assist dropped out at the last min etc. Plus he's in a new job, so has to keep his nose to the grindstone..
I haven't lost any WU due to increased times 'yet' but its a pita:-(
Regards,
Cliff
ID: 4767 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Richie

Send message
Joined: 25 Jul 14
Posts: 64
Credit: 100,582,080
RAC: 0
Message 4768 - Posted: 15 Jan 2016, 5:03:36 UTC

Last modified: 15 Jan 2016, 5:06:50 UTC
Luckily, the deadline for newly downloaded tasks seems to be always somewhat far in the future. If a host can contact project server at least once a day, it's quite easy to avoid having the tasks end up late.

I take a look at the running tasks daily and find the next task in queue. I have decided for example: That next task in line needs to have the deadline NOT BEFORE tomorrow evening. And same procedure always... every day.

If there are tasks that have deadline earlier than tomorrow evening, I'll cancel enough of them until my requirement is met. Then I let the scheduler download new tasks and those will be placed in the end of the tail.

That way I can leave the computer, relax and sleep long in the morning, knowing there won't be any tasks missing the deadline if I manage to check at the host again before the next evening. Or sometimes I might even change the rule to something like "next deadline at least two days ahead".

If your tasks are at the knife edge at the moment and some of them are ending up late, just cancel a ton of them in cold blood (tasks for the next five days, for example). Those cancelled tasks will reborn somewhere else.
ID: 4768 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Presrvd
Avatar

Send message
Joined: 15 Jun 15
Posts: 16
Credit: 123,015,840
RAC: 0
Message 4771 - Posted: 20 Jan 2016, 16:49:23 UTC - in response to Message 4768.  
I take a look at the running tasks daily and find the next task in queue. I have decided for example: That next task in line needs to have the deadline NOT BEFORE tomorrow evening. And same procedure always... every day.

If there are tasks that have deadline earlier than tomorrow evening, I'll cancel enough of them until my requirement is met. Then I let the scheduler download new tasks and those will be placed in the end of the tail.

That way I can leave the computer, relax and sleep long in the morning, knowing there won't be any tasks missing the deadline if I manage to check at the host again before the next evening. Or sometimes I might even change the rule to something like "next deadline at least two days ahead".

If your tasks are at the knife edge at the moment and some of them are ending up late, just cancel a ton of them in cold blood (tasks for the next five days, for example). Those cancelled tasks will reborn somewhere else.

No offense, but that is entirely too much babysitting for me.

My late WUs are now in the hundreds, and yet there is still no reason or explanation why. From reading through the forum, it looks like this is a norm. Perhaps it is time to retire this project.
ID: 4771 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Anthony Ayiomamitis

Send message
Joined: 12 Dec 15
Posts: 6
Credit: 507,840
RAC: 0
Message 4828 - Posted: 19 Mar 2016, 21:29:32 UTC
I have been away a couple of months. Are the work units still showing the increase in runtime we noticed back in January?
ID: 4828 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Kyong
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 9 Jun 12
Posts: 584
Credit: 52,667,664
RAC: 0
Message 4835 - Posted: 23 Mar 2016, 11:41:23 UTC
I am sorry, I had too many things around me I had to deal with. I am going to investigate it, but I have limited internet connection now so it slows me down and I have to do many things from work.
ID: 4835 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Ousermaatre

Send message
Joined: 15 Feb 13
Posts: 1
Credit: 3,761,666
RAC: 0
Message 4837 - Posted: 23 Mar 2016, 19:10:57 UTC - in response to Message 4835.  
Hi Kyong,

I sent you a PM and I didn't answer could you contact me?

Ousermaatre
ID: 4837 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : Dramatic increase in runtimes