Server out of disk space??


Message boards : Problems and bug reports : Server out of disk space??

Message board moderation

To post messages, you must log in.
AuthorMessage
Profile networkman

Send message
Joined: 18 Jul 13
Posts: 23
Credit: 17,458,800
RAC: 0
Message 3488 - Posted: 12 Aug 2014, 12:56:06 UTC
The going theory over in this thread: http://asteroidsathome.net/boinc/forum_thread.php?id=333 is that the server may have run out of disk space some time prior to 7:32 UTC on Aug 12th.

I'm posting here in case the Admin(s?) check or are alerted to activity in this category of forum message more often.
ID: 3488 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Mike

Send message
Joined: 6 Mar 14
Posts: 58
Credit: 5,931,360
RAC: 0
Message 3505 - Posted: 14 Aug 2014, 2:43:07 UTC - in response to Message 3488.  
"Feeder" hasn't been running for a couple of days now. No new work. Also, db_purge" server not running. I have 4 units completed and another 20 in uploading status -- for 2 days. Frozen.

I restarted BOINC manager & see in the event log: "File upload server: Server is out of disk space".

No notice of planned maintenance -- we need the Admin to fix this.
ID: 3505 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Mike

Send message
Joined: 6 Mar 14
Posts: 58
Credit: 5,931,360
RAC: 0
Message 3511 - Posted: 14 Aug 2014, 2:55:51 UTC - in response to Message 3488.  
"Feeder" hasn't been running for a couple of days now. No new work. Also, db_purge" server not running. I have 4 units completed and another 20 in uploading status -- for 2 days. Frozen.

I restarted BOINC manager & see in the event log: "File upload server: Server is out of disk space".

No notice of planned maintenance -- we need the Admin to fix this.
ID: 3511 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
SuperSluether

Send message
Joined: 5 Jul 14
Posts: 38
Credit: 1,262,880
RAC: 0
Message 3517 - Posted: 14 Aug 2014, 14:25:17 UTC - in response to Message 3511.  
Oh, I remember now. Without the feeder, nobody gets new tasks. Without the purge, old tasks never get deleted. These 2 server functions are essentially the bottleneck, and now they're not working. :/
ID: 3517 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Mike

Send message
Joined: 6 Mar 14
Posts: 58
Credit: 5,931,360
RAC: 0
Message 3518 - Posted: 14 Aug 2014, 16:15:20 UTC
Feeder & db_purge servers are still down - this is day #3. Can't get new work and can't upload completed tasks.

We need the Admin to fix this.
ID: 3518 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 1 Jan 14
Posts: 302
Credit: 32,671,868
RAC: 4
Message 3529 - Posted: 15 Aug 2014, 10:53:19 UTC - in response to Message 3518.  
Feeder & db_purge servers are still down - this is day #3. Can't get new work and can't upload completed tasks.

We need the Admin to fix this.


He is out of town until Sunday, hopefully 'fixing' his family matters.
ID: 3529 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile killik

Send message
Joined: 20 Jun 14
Posts: 3
Credit: 2,785,440
RAC: 0
Message 3562 - Posted: 19 Aug 2014, 11:35:54 UTC
19.8.2014 12:53 "Error reported by file upload server: Server is out of disk space" problem again.
ID: 3562 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile killik

Send message
Joined: 20 Jun 14
Posts: 3
Credit: 2,785,440
RAC: 0
Message 3563 - Posted: 19 Aug 2014, 11:36:00 UTC
19.8.2014 12:53 "Error reported by file upload server: Server is out of disk space" problem again.
ID: 3563 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile killik

Send message
Joined: 20 Jun 14
Posts: 3
Credit: 2,785,440
RAC: 0
Message 3564 - Posted: 19 Aug 2014, 11:36:24 UTC
19.8.2014 12:53 "Error reported by file upload server: Server is out of disk space" problem again.
ID: 3564 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Mike

Send message
Joined: 6 Mar 14
Posts: 58
Credit: 5,931,360
RAC: 0
Message 3569 - Posted: 19 Aug 2014, 17:12:35 UTC
A suggestion from the News Forum: some disk space could be freed up by deleting some the older task records. E.G., I have almost 5 months-worth, some 1600+ task records, most of which could be deleted. I'd be happy with just the past couple of weeks of task records.

People with millions of credits, do you have thousands of old task records?
ID: 3569 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
alexander

Send message
Joined: 28 Apr 13
Posts: 87
Credit: 26,716,176
RAC: 0
Message 3572 - Posted: 19 Aug 2014, 18:35:26 UTC - in response to Message 3569.  
People with millions of credits, do you have thousands of old task records?


My record says 3040 valids. Did not check all of them, as it gets slower and slower moving back in 'time'.
But for shure, I don't need them!
ID: 3572 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile networkman

Send message
Joined: 18 Jul 13
Posts: 23
Credit: 17,458,800
RAC: 0
Message 3573 - Posted: 19 Aug 2014, 19:50:10 UTC - in response to Message 3572.  
My own valid record shows as 24334. I agree, we don't probably don't need to know that far back.

The only tasks I really look at are the ones with errors to make sure I don't have a machine with a hardware/software issue.
ID: 3573 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile BilBg
Avatar

Send message
Joined: 19 Jun 12
Posts: 221
Credit: 623,640
RAC: 0
Message 3574 - Posted: 20 Aug 2014, 0:49:52 UTC
 
The server have 7 x 600 GB IBM SAS 10000 rpm (RAID5)
http://asteroidsathome.net/boinc/server_status.php

7 x 600 GB = 4.2 TB but RAID5 uses some of it for parity information:
http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_5

According the table the usable (for data) capacity is 1 − 1/n = 1 − 1/7 = 6/7
4.2 TB * (6/7) = 3.6 TB
http://en.wikipedia.org/wiki/Standard_RAID_levels#Comparison


So the goal of this server upgrade (~year ago) was fast disk subsystem using expensive but 'small' disks
http://www.amazon.com/IBM-2-5-Inch-Internal-Drives-49Y2048/dp/B004K6QZLI

This link shows (today): List Price: $520.00 , Price: $325.00


I don't think the Download/Upload need fast disk subsystem (only database accesses need it)
A few consumer-grade disks can do for Download/Upload server

For the price of one 600 GB IBM SAS 10000 rpm you can get 3 x 3 TB consumer-grade disks
(they may not be hot-swappable and in case of failure may need server shutdown to be replaced)
http://www.amazon.com/SEAGATE-ST3000DM001-Barracuda-cache-internal/dp/B0067G0YR0


If those 3 x 3 TB disks are in RAID5 the (additional for the server) usable capacity will be 6 TB (for < $400)

Someone willing to make a donation ;)

 



- ALF - "Find out what you don't do well ..... then don't do it!" :)
ID: 3574 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 1 Jan 14
Posts: 302
Credit: 32,671,868
RAC: 4
Message 3575 - Posted: 20 Aug 2014, 11:30:04 UTC - in response to Message 3574.  
 
The server have 7 x 600 GB IBM SAS 10000 rpm (RAID5)
http://asteroidsathome.net/boinc/server_status.php

7 x 600 GB = 4.2 TB but RAID5 uses some of it for parity information:
http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_5

According the table the usable (for data) capacity is 1 − 1/n = 1 − 1/7 = 6/7
4.2 TB * (6/7) = 3.6 TB
http://en.wikipedia.org/wiki/Standard_RAID_levels#Comparison


So the goal of this server upgrade (~year ago) was fast disk subsystem using expensive but 'small' disks
http://www.amazon.com/IBM-2-5-Inch-Internal-Drives-49Y2048/dp/B004K6QZLI

This link shows (today): List Price: $520.00 , Price: $325.00


I don't think the Download/Upload need fast disk subsystem (only database accesses need it)
A few consumer-grade disks can do for Download/Upload server

For the price of one 600 GB IBM SAS 10000 rpm you can get 3 x 3 TB consumer-grade disks
(they may not be hot-swappable and in case of failure may need server shutdown to be replaced)
http://www.amazon.com/SEAGATE-ST3000DM001-Barracuda-cache-internal/dp/B0067G0YR0


If those 3 x 3 TB disks are in RAID5 the (additional for the server) usable capacity will be 6 TB (for < $400)

Someone willing to make a donation ;)  


I just bought two 3tb drives for my own NAS, on sale they were about $250 US together. The problem with buying cheap on sale drives is that when put them in a small low airflow rack server or even a NAS box they can quickly overheat. The drives that are recommended to work in a NAS environment are designed to produce less heat and thus last longer. I am NOT saying your idea won't work, I am just saying the environment they will be in is key.
ID: 3575 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Kyong
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 9 Jun 12
Posts: 584
Credit: 52,667,664
RAC: 0
Message 3576 - Posted: 21 Aug 2014, 7:50:23 UTC
I have been already thinking about adding secondary server mounting folder for data via sshfs. db_purge is set to archive older records than 90 days. Problem is that there are too many results now.
Also prepared workunits and finished results (post-processed) have more than 1 TB. I am just deleting them and storing in my server home where I have also 5 TB RAID5. But extending capacity for it also cost something, I have discussed it but I don't know when the university buy new disks.
ID: 3576 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
MarkJ
Avatar

Send message
Joined: 27 Jun 12
Posts: 129
Credit: 62,725,780
RAC: 0
Message 3579 - Posted: 21 Aug 2014, 12:45:31 UTC - in response to Message 3576.  
I have been already thinking about adding secondary server mounting folder for data via sshfs. db_purge is set to archive older records than 90 days. Problem is that there are too many results now.
Also prepared workunits and finished results (post-processed) have more than 1 TB. I am just deleting them and storing in my server home where I have also 5 TB RAID5. But extending capacity for it also cost something, I have discussed it but I don't know when the university buy new disks.


Would a Drobo be a better idea? It gives storage, handles the data redundancy and you can just attach to it like a network drive. You just put some drives in and connect it to power and a network port. It does the rest. Not too sure if it could handle the speed though.
BOINC blog
ID: 3579 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
alexander

Send message
Joined: 28 Apr 13
Posts: 87
Credit: 26,716,176
RAC: 0
Message 3580 - Posted: 21 Aug 2014, 18:08:09 UTC
Feeder and db_purge are down again.

Kyong, could you please post here what new or additional hardware could help making this project stable with some headroom for increasing load?

BilBG started to post the word 'Donation', Mikey responded direct to that. So maybe the volunteers can help where bureaucracy fails to respond adequate.

Alexander
ID: 3580 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ruud van der Kroef

Send message
Joined: 7 Oct 12
Posts: 4
Credit: 20,691,306
RAC: 875
Message 3585 - Posted: 25 Aug 2014, 23:46:40 UTC - in response to Message 3580.  
Hallo all,

The IBM System x3550 M4 has 8 disk slots (2.5") . The Server Status page says there are 7 disks in a RAID5 configuration, so I assume there is 1 boot disk. (which is already a bad idea: you should have the system disk in RAID1 (mirror) configuration). Currently there are 7x 600GB disks in RAID5. You can replace those with higher capacity disks, but (again I assume) in the current configuration that will be a major job. There is a solution though: zfs. Invented by Sun; originally a abbreviation for Zeta Byte File System.
It is supported in Linux. You should really read yourself into this! It is (simply said) a combination of a volume manager and (software) RAID (and a lot more).
You can (on the fly) replace your disks with higher capacity ones, and in the end have a higher capacity RAID5.
Of course the first time you implement this is also a major job.

For the 'bigger' disk, as the previous poster mentioned: Donation?

Regards,
Ruud
ID: 3585 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile HA-SOFT, s.r.o.
Project developer
Project tester

Send message
Joined: 21 Dec 12
Posts: 176
Credit: 136,462,135
RAC: 1,198
Message 3586 - Posted: 26 Aug 2014, 9:16:26 UTC - in response to Message 3585.  
I think it's one raid5 volume with 7 disks.
ID: 3586 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Kyong
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 9 Jun 12
Posts: 584
Credit: 52,667,664
RAC: 0
Message 3587 - Posted: 26 Aug 2014, 11:29:33 UTC
There is only one volume with RAID5. The last new bay will be filled soon and we are thinking about new solution. When I am home I will check my backup and if there is one folder then I am going to free up almost 1 TB so the problem with running out of space should be solved in the evening. This should be enough before we extended capacity with some new storage server.
ID: 3587 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Problems and bug reports : Server out of disk space??