sever out of disk space


Message boards : Problems and bug reports : sever out of disk space

Message board moderation

To post messages, you must log in.
AuthorMessage
robertmiles

Send message
Joined: 17 Aug 14
Posts: 49
Credit: 5,225,280
RAC: 0
Message 6021 - Posted: 26 Oct 2018, 23:52:17 UTC
It looks like the uploads problem is now PARTIALLY fixed, at the cost of a major slowdown in the human-readable part of the project. Most, but not all, of my uploads have finished, and I've downloaded a few new tasks.
ID: 6021 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
DSO

Send message
Joined: 19 Jun 12
Posts: 1
Credit: 591,840
RAC: 0
Message 6242 - Posted: 4 Apr 2019, 14:27:59 UTC - in response to Message 6021.  
It is happening again:

04.04.2019 16:21:40 | Asteroids@home | [error] Error reported by file upload server: Server is out of disk space
04.04.2019 16:21:40 | Asteroids@home | Temporarily failed upload of ps_190401_input_47930_1_0_0: transient upload error
04.04.2019 16:21:40 | Asteroids@home | Backing off 00:15:49 on upload of ps_190401_input_47930_1_0_0
ID: 6242 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Composer

Send message
Joined: 20 Aug 14
Posts: 20
Credit: 753,120
RAC: 0
Message 6243 - Posted: 4 Apr 2019, 17:13:37 UTC
Out of curiosity, once The results are returned and validated, what is done with any asteroid models/data obtained? Is there some third party that stores information about various asteroids that it is transferred to? If so, who hosts that? (Im assuming that this is what "DAMIT" is, but I am not clear on who actually hosts that. Is it this project as well, or someone else? Possibly NASA or ESA, etc?)

I know nothing about running this kind of project, and as such I want to be clear that I'm not criticizing anyone about the servers running out of disk space, Im just curious as to how that actually happens. Like, it would make sense if the servers for this project are also being used for the long term storage of the results, but if that is handled by some other organization, wouldn't freeing up server space for the project be as simple as transferring data to whatever server is storing the results long term?

Also, couldn't the total storage capacity of the project servers be increased simply by adding additional hard drives? I know really big, server grade ones are probably expensive, but cant you also use regular consumer grade ones as well? Sure its not ideal, since they were not really meant to handle the same load, and they have lower bandwidth, and if you wanted them cheap you would be getting smaller ones, but how expensive would it be to make an additional RAID array out of cheap, old desktop drives with a capacity of around 250 GB each? The server status page says that there are 2 arrays in use right now, both using RAID 5. One is made from 8 600 GB drives, the other from 3 600 GB drives. If I understand how RAID 5 works, that would give a total storage capacity of 4200 GB, right? Say you get 5 250 GB consumer level drives for cheap, and in RAID 5 that would give you an extra 1000 GB of space. And if you wanted to have greater redundancy in the event of a failure since they are all old, consumer grade drives, just tell it to use 2 drives instead of 1 for redundancy, at the cost of 250 GB of storage space.

If any of that is even remotely viable, (and again, I am by no means an expert on this stuff) would it be better to 'donate' by mailing a couple old hard drives rather than a $20 donation potentially? Again, I don't want to tell anyone how to do their job, or imply that I could do it better, (because I really cant) Im just curious about why things are done the way they are done.
ID: 6243 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dataman
Avatar

Send message
Joined: 11 Aug 12
Posts: 8
Credit: 54,899,677
RAC: 0
Message 6244 - Posted: 6 Apr 2019, 14:40:52 UTC - in response to Message 6243.  


I know nothing about running this kind of project, and as such I want to be clear that I'm not criticizing anyone about the servers running out of disk space, Im just curious as to how that actually happens. Like, it would make sense if the servers for this project are also being used for the long term storage of the results, but if that is handled by some other organization, wouldn't freeing up server space for the project be as simple as transferring data to whatever server is storing the results long term?

Also, couldn't the total storage capacity of the project servers be increased simply by adding additional hard drives? I know really big, server grade ones are probably expensive, but cant you also use regular consumer grade ones as well? Sure its not ideal, since they were not really meant to handle the same load, and they have lower bandwidth, and if you wanted them cheap you would be getting smaller ones, but how expensive would it be to make an additional RAID array out of cheap, old desktop drives with a capacity of around 250 GB each? The server status page says that there are 2 arrays in use right now, both using RAID 5. One is made from 8 600 GB drives, the other from 3 600 GB drives. If I understand how RAID 5 works, that would give a total storage capacity of 4200 GB, right? Say you get 5 250 GB consumer level drives for cheap, and in RAID 5 that would give you an extra 1000 GB of space. And if you wanted to have greater redundancy in the event of a failure since they are all old, consumer grade drives, just tell it to use 2 drives instead of 1 for redundancy, at the cost of 250 GB of storage space.

If any of that is even remotely viable, (and again, I am by no means an expert on this stuff) would it be better to 'donate' by mailing a couple old hard drives rather than a $20 donation potentially? Again, I don't want to tell anyone how to do their job, or imply that I could do it better, (because I really cant) Im just curious about why things are done the way they are done.

+1
ID: 6244 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Composer

Send message
Joined: 20 Aug 14
Posts: 20
Credit: 753,120
RAC: 0
Message 6246 - Posted: 7 Apr 2019, 20:50:57 UTC - in response to Message 6244.  
what part about that is the +1 in response to?
ID: 6246 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Remarc
Avatar

Send message
Joined: 19 Sep 14
Posts: 13
Credit: 10,026,859
RAC: 0
Message 6301 - Posted: 11 Jun 2019, 13:45:00 UTC
up
db_purge project1 Не работает (Not Running)
ID: 6301 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dark Angel

Send message
Joined: 11 Apr 18
Posts: 19
Credit: 11,308,420
RAC: 7,032
Message 6303 - Posted: 11 Jun 2019, 21:47:24 UTC
Wed 12 Jun 2019 07:37:35 AEST | Asteroids@home | [error] Error reported by file upload server: Server is out of disk space

Isn't the new server supposed to be here by now?
ID: 6303 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Presrvd
Avatar

Send message
Joined: 15 Jun 15
Posts: 16
Credit: 123,015,840
RAC: 0
Message 6304 - Posted: 11 Jun 2019, 21:57:29 UTC - in response to Message 6244.  


I know nothing about running this kind of project, and as such I want to be clear that I'm not criticizing anyone about the servers running out of disk space, Im just curious as to how that actually happens. Like, it would make sense if the servers for this project are also being used for the long term storage of the results, but if that is handled by some other organization, wouldn't freeing up server space for the project be as simple as transferring data to whatever server is storing the results long term?

Also, couldn't the total storage capacity of the project servers be increased simply by adding additional hard drives? I know really big, server grade ones are probably expensive, but cant you also use regular consumer grade ones as well? Sure its not ideal, since they were not really meant to handle the same load, and they have lower bandwidth, and if you wanted them cheap you would be getting smaller ones, but how expensive would it be to make an additional RAID array out of cheap, old desktop drives with a capacity of around 250 GB each? The server status page says that there are 2 arrays in use right now, both using RAID 5. One is made from 8 600 GB drives, the other from 3 600 GB drives. If I understand how RAID 5 works, that would give a total storage capacity of 4200 GB, right? Say you get 5 250 GB consumer level drives for cheap, and in RAID 5 that would give you an extra 1000 GB of space. And if you wanted to have greater redundancy in the event of a failure since they are all old, consumer grade drives, just tell it to use 2 drives instead of 1 for redundancy, at the cost of 250 GB of storage space.

If any of that is even remotely viable, (and again, I am by no means an expert on this stuff) would it be better to 'donate' by mailing a couple old hard drives rather than a $20 donation potentially? Again, I don't want to tell anyone how to do their job, or imply that I could do it better, (because I really cant) Im just curious about why things are done the way they are done.


Unfortunately, running a database isn't as simple as adding more drives for space. Imagine you have a 4GB mp4 file, and two 2GB thumb drives. Yes, you can split an mp4. Yes, you can split a database, but that doesn't necessarily make it wise to do so. Mixing drives is a terribly bad idea, though, especially used ones. One also has to consider that perhaps there are no more HDD slots on their server. Connecting them via USB just isn't an option. There's no way the disks could handle the I/O over that channel to a database facing the internet with potentially 250k+ computers attempting access. It's just one of those things where you don't want to use 128 shot glasses when you really need a gallon jug... Hope that helps.
ID: 6304 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
keputnam

Send message
Joined: 24 Jun 14
Posts: 23
Credit: 18,794,469
RAC: 3,374
Message 6308 - Posted: 12 Jun 2019, 7:22:39 UTC - in response to Message 6304.  
"Unfortunately, running a database isn't as simple as adding more drives for space."

Actually, it is

You add the physical drives to the DB server, then you extend the size of the database

The question is:

Why is there no scheduled task to purge the database?
ID: 6308 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
robertmiles

Send message
Joined: 17 Aug 14
Posts: 49
Credit: 5,225,280
RAC: 0
Message 6309 - Posted: 12 Jun 2019, 12:34:02 UTC - in response to Message 6308.  
"Unfortunately, running a database isn't as simple as adding more drives for space."

Actually, it is

You add the physical drives to the DB server, then you extend the size of the database

That might work if the DB server still has room for more physical drives. Does it?
ID: 6309 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Presrvd
Avatar

Send message
Joined: 15 Jun 15
Posts: 16
Credit: 123,015,840
RAC: 0
Message 6310 - Posted: 12 Jun 2019, 19:20:05 UTC - in response to Message 6308.  
"Unfortunately, running a database isn't as simple as adding more drives for space."

Actually, it is

You add the physical drives to the DB server, then you extend the size of the database

Over simplification. There's a few things to consider: (1) Is there room on the server for another drive? (2) A new disk would need to be added to the RAID array (3) the current logical volume is SAS. While you technically could add SATA to the SAS array, I don't know a single person that would ever consider it, much less recommend it (4) you're going to take a performance hit if you're using a non-matching drive (10k v 7200--or Bob forbid 5400, SAS v SATA, 64MB v 128MB or 256MB cache, etc.). There are too many unknowns about the server being discussed to just say 'add a drive and extend the database', as per my actual statement, but thanks for cherry-picking the comment.
ID: 6310 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
robertmiles

Send message
Joined: 17 Aug 14
Posts: 49
Credit: 5,225,280
RAC: 0
Message 6441 - Posted: 15 Feb 2020, 4:53:34 UTC
2/14/2020 10:50:46 PM | Asteroids@home | [error] Error reported by file upload server: Server is out of disk space
2/14/2020 10:50:46 PM | Asteroids@home | [error] Error reported by file upload server: Server is out of disk space
2/14/2020 10:50:47 PM | Asteroids@home | [error] Error reported by file upload server: Server is out of disk space
ID: 6441 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
kimkris

Send message
Joined: 18 Jan 20
Posts: 2
Credit: 6,707,418
RAC: 0
Message 6442 - Posted: 15 Feb 2020, 13:29:39 UTC - in response to Message 6441.  
I get the same message, and looking at the Server status page, you can see that the db_purge service is not running, maybe thats is the problem.

I hope he is reading this, and will get it up and running again soon.
ID: 6442 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
robertmiles

Send message
Joined: 17 Aug 14
Posts: 49
Credit: 5,225,280
RAC: 0
Message 6443 - Posted: 15 Feb 2020, 15:48:54 UTC - in response to Message 6442.  
I get the same message, and looking at the Server status page, you can see that the db_purge service is not running, maybe thats is the problem.

I hope he is reading this, and will get it up and running again soon.

How much information do you have on how many workunits have files that are ready to delete and therefore how much space restarting db_purge will save?
ID: 6443 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
robertmiles

Send message
Joined: 17 Aug 14
Posts: 49
Credit: 5,225,280
RAC: 0
Message 6447 - Posted: 16 Feb 2020, 4:08:56 UTC
The project is currently sending out more tasks, even when it cannot accept uploads of results of completed tasks. Is this suitable for freeing up disk space on the server, or is it increasing the amount of disk space in use instead?

Should the project stop sending out any more tasks if the amount of free disk space drops below a certain level?
ID: 6447 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
David

Send message
Joined: 24 Feb 20
Posts: 3
Credit: 27,840
RAC: 0
Message 6456 - Posted: 24 Feb 2020, 20:53:56 UTC - in response to Message 6243.  
"Composer" you said it all,I agree 100%,I'm on a fixed income,& would gladly give my 'Virus'-ridden old computers & harddrives "Windows 98" "95" "Vista" & "Windows 7" in place of cash to help,but all 3 computers are gone now."HP" & a 14-16 inch Monitor.
ID: 6456 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Problems and bug reports : sever out of disk space