Server outage resolved

Author	Message
Kyong Project administrator Project developer Project tester Project scientist Send message Joined: 9 Jun 12 Posts: 578 Credit: 52,667,664 RAC: 0	Message 6857 - Posted: 8 Jul 2020, 11:48:21 UTC Last modified: 8 Jul 2020, 11:49:14 UTC
I am sorry for server outages which was happening recently. There were several factors which often caused lack of free space and some services crashed and I had to do an emergency shutdown on Saturday in order to fix everything so I couldn't inform you about it except for Twitter. There is still a hardware problem with one of servers which I have to fix but the server is not being used now so project should be stable now. Radim Vančo (FoxKyong)
ID: 6857 · Rating: 0 · rate: / Reply Quote

Nflight Send message Joined: 10 Jul 12 Posts: 18 Credit: 6,214,080 RAC: 103	Message 6859 - Posted: 8 Jul 2020, 13:51:36 UTC - in response to Message 6857.
I am still showing nearly >120 work units trying to upload ? User ID #1865 and Computer ID #449480 ____________
ID: 6859 · Rating: 0 · rate: / Reply Quote

Clint Send message Joined: 26 Sep 13 Posts: 4 Credit: 10,123,680 RAC: 391	Message 6860 - Posted: 8 Jul 2020, 14:11:43 UTC
I have over 20 completed work units that won't upload, "error reported by file upload server,: can't open log file /log_project1/file_upload_handler.log errno: 9
ID: 6860 · Rating: 0 · rate: / Reply Quote

afl Send message Joined: 17 Mar 20 Posts: 1 Credit: 75,360 RAC: 0	Message 6862 - Posted: 8 Jul 2020, 14:28:14 UTC Last modified: 8 Jul 2020, 14:29:39 UTC
I also have 3 WUs to upload and 5 WUs to report but nothing works... (this effect is since sunday (July 5.)
ID: 6862 · Rating: 0 · rate: / Reply Quote

sachse Send message Joined: 25 Jan 13 Posts: 1 Credit: 10,222,440 RAC: 976	Message 6863 - Posted: 8 Jul 2020, 15:08:02 UTC
I have 178 WUs that cannot uploaded because of "Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9)". I'm not amused.
ID: 6863 · Rating: 0 · rate: / Reply Quote

Saenger Send message Joined: 18 Jun 12 Posts: 23 Credit: 4,992,995 RAC: 5,561	Message 6864 - Posted: 8 Jul 2020, 16:09:20 UTC
Same here, just on another scale (that's far less relevant with only 3 WUs on my main machine): Mi 08 Jul 2020 18:04:12 CEST \| Asteroids@home \| [fxd] starting upload, upload_offset -1 Mi 08 Jul 2020 18:04:12 CEST \| Asteroids@home \| Started upload of ps_200630_input_116551_11_0_0 Mi 08 Jul 2020 18:04:12 CEST \| Asteroids@home \| [file_xfer] URL: http://asteroidsathome.net/boinc_cgi/file_upload_handler Mi 08 Jul 2020 18:04:14 CEST \| Asteroids@home \| [file_xfer] http op done; retval 0 (Success) Mi 08 Jul 2020 18:04:14 CEST \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) Mi 08 Jul 2020 18:04:14 CEST \| Asteroids@home \| [file_xfer] parsing upload response: <data_server_reply> <status>1</status> <message>can't open log file '../log_project1/file_upload_handler.log' (errno: 9)</message></data_server_reply> Mi 08 Jul 2020 18:04:14 CEST \| Asteroids@home \| [file_xfer] parsing status: -127 Mi 08 Jul 2020 18:04:14 CEST \| Asteroids@home \| [file_xfer] file transfer status -127 (transient upload error) Mi 08 Jul 2020 18:04:14 CEST \| Asteroids@home \| Temporarily failed upload of ps_200630_input_116551_11_0_0: transient upload error Mi 08 Jul 2020 18:04:14 CEST \| Asteroids@home \| Backing off 04:24:19 on upload of ps_200630_input_116551_11_0_0 So I'm just a wee bit not amused, the reporting of my already uploaded WUs has happened quite fine, obviously. ____________ Grüße vom Sänger
ID: 6864 · Rating: 0 · rate: / Reply Quote

Steffen Send message Joined: 25 May 13 Posts: 1 Credit: 19,296,960 RAC: 10,487	Message 6865 - Posted: 8 Jul 2020, 16:16:37 UTC - in response to Message 6857.
It's imposible for me to upload my completed Workunits.
ID: 6865 · Rating: 0 · rate: / Reply Quote

tvcunningham77 Send message Joined: 9 May 20 Posts: 1 Credit: 5,344,800 RAC: 14,858	Message 6866 - Posted: 8 Jul 2020, 16:30:37 UTC
Still unable to upload my completed work units which are about to hit their deadline.
ID: 6866 · Rating: 0 · rate: / Reply Quote

Alan Barnes Send message Joined: 11 Nov 13 Posts: 2 Credit: 2,012,640 RAC: 920	Message 6868 - Posted: 8 Jul 2020, 17:16:33 UTC - in response to Message 6857.
Will work-units completed within the deadline but which failed to upload and which have now timed out due to the server outages eventually be uploaded and attract credit? For example Name ps_200624_input_82381_5_0 Workunit 130936318 Created 24 Jun 2020, 19:35:26 UTC Sent 25 Jun 2020, 13:05:54 UTC Alan Barnes
ID: 6868 · Rating: 0 · rate: / Reply Quote

Lars Vindal Send message Joined: 7 Jun 16 Posts: 3 Credit: 2,292,960 RAC: 754	Message 6869 - Posted: 8 Jul 2020, 17:54:44 UTC
I no longer get the "out of disk space" messages, and most of my previos pending uploads have gone through. BUT... Now I can't upload anything because the server can't read its log files: 08/07/2020 17:28:12 \| Asteroids@home \| Started upload of ps_200630_input_110498_1_1_0 08/07/2020 17:28:12 \| Asteroids@home \| Started upload of ps_200630_input_110498_3_1_0 08/07/2020 17:28:12 \| Asteroids@home \| Started upload of ps_200624_input_88153_14_0_0 08/07/2020 17:28:14 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 08/07/2020 17:28:14 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 08/07/2020 17:28:14 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 08/07/2020 17:28:14 \| Asteroids@home \| Temporarily failed upload of ps_200630_input_110498_1_1_0: transient upload error 08/07/2020 17:28:14 \| Asteroids@home \| Backing off 04:52:58 on upload of ps_200630_input_110498_1_1_0 08/07/2020 17:28:14 \| Asteroids@home \| Temporarily failed upload of ps_200630_input_110498_3_1_0: transient upload error 08/07/2020 17:28:14 \| Asteroids@home \| Backing off 05:58:53 on upload of ps_200630_input_110498_3_1_0 08/07/2020 17:28:14 \| Asteroids@home \| Temporarily failed upload of ps_200624_input_88153_14_0_0: transient upload error 08/07/2020 17:28:14 \| Asteroids@home \| Backing off 00:17:47 on upload of ps_200624_input_88153_14_0_0 Is this related to the low disk space error we had earlier, or is it a misconfiguration somewhere?
ID: 6869 · Rating: 0 · rate: / Reply Quote

Lars Vindal Send message Joined: 7 Jun 16 Posts: 3 Credit: 2,292,960 RAC: 754	Message 6870 - Posted: 8 Jul 2020, 18:00:02 UTC
Prior to these messages, there also was a period with additional log read failures, but related to the scheduler instead.
ID: 6870 · Rating: 0 · rate: / Reply Quote

Steve Dodd Send message Joined: 23 Aug 12 Posts: 8 Credit: 10,001,880 RAC: 0	Message 6871 - Posted: 8 Jul 2020, 18:05:56 UTC
Maybe that server that isn't working still happened to house the log files? ____________
ID: 6871 · Rating: 0 · rate: / Reply Quote

Manxscouse Send message Joined: 2 Feb 14 Posts: 11 Credit: 17,140,320 RAC: 6	Message 6874 - Posted: 8 Jul 2020, 21:21:52 UTC - in response to Message 6857.
Malwarebytes has started saying that that asteroidsathome.net has a trojan. Maybe you can check this as well.
ID: 6874 · Rating: 0 · rate: / Reply Quote

FreeQuest Send message Joined: 31 Jul 18 Posts: 1 Credit: 619,200 RAC: 352	Message 6878 - Posted: 9 Jul 2020, 4:28:23 UTC
Malwarebytes classes your application as a trojan. see for yourself. in order to post this I had to allow your website. Malwarebytes www.malwarebytes.com -Log Details- Protection Event Date: 7/9/20 Protection Event Time: 2:10 PM Log File: 0fed5ace-c19a-11ea-89a6-88ae1d7c2914.json -Software Information- Version: 4.1.0.56 Components Version: 1.0.955 Update Package Version: 1.0.26585 License: Premium -System Information- OS: Windows 10 (Build 18362.900) CPU: x64 File System: NTFS User: System -Blocked Website Details- Malicious Website: 1 , C:\Program Files\BOINC\boinc.exe, Blocked, -1, -1, 0.0.0 -Website Data- Category: Trojan Domain: asteroidsathome.net IP Address: 89.29.55.30 Port: 80 Type: Outbound File: C:\Program Files\BOINC\boinc.exe (end)
ID: 6878 · Rating: 0 · rate: / Reply Quote

Thyme Lawn Send message Joined: 15 Oct 17 Posts: 8 Credit: 2,368,800 RAC: 916	Message 6879 - Posted: 9 Jul 2020, 6:17:36 UTC - in response to Message 6868.
Will work-units completed within the deadline but which failed to upload and which have now timed out due to the server outages eventually be uploaded and attract credit? For example Name ps_200624_input_82381_5_0 Workunit 130936318 Created 24 Jun 2020, 19:35:26 UTC Sent 25 Jun 2020, 13:05:54 UTC Alan Barnes Possibly. That WU has issued a new task to replace your timed out one. If that task is uploaded, reported and validated first the status for your one will change to "Too late" and you won't get any credit. If your task is validated first both will be credited if the replacement starts running before that computer makes its next scheduler request (if it hasn't the server will abort it). ____________ "The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
ID: 6879 · Rating: 0 · rate: / Reply Quote

Maike-Sophy Send message Joined: 13 Jan 17 Posts: 1 Credit: 2,168,160 RAC: 4,736	Message 6880 - Posted: 9 Jul 2020, 7:28:07 UTC
09.07.2020 06:35:52 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 09.07.2020 06:35:52 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 09.07.2020 06:35:52 \| Asteroids@home \| Backing off 00:29:00 on upload of ps_200630_input_113518_2_0_0 09.07.2020 06:35:52 \| Asteroids@home \| Backing off 00:31:05 on upload of ps_200630_input_113611_7_0_0 09.07.2020 08:25:15 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 09.07.2020 08:25:15 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 09.07.2020 08:25:15 \| Asteroids@home \| Backing off 00:54:55 on upload of ps_200630_input_113518_2_0_0 09.07.2020 08:25:15 \| Asteroids@home \| Backing off 00:43:51 on upload of ps_200630_input_113611_7_0_0 These are the log reports i get
ID: 6880 · Rating: 0 · rate: / Reply Quote

Glenn Marsden Send message Joined: 15 Apr 16 Posts: 1 Credit: 3,664,320 RAC: 1,072	Message 6881 - Posted: 9 Jul 2020, 7:46:52 UTC
09/07/2020 08:32:00 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 09/07/2020 08:32:00 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9)
ID: 6881 · Rating: 0 · rate: / Reply Quote

fbac1492 Send message Joined: 20 May 20 Posts: 1 Credit: 600,480 RAC: 0	Message 6882 - Posted: 9 Jul 2020, 7:58:48 UTC
i hope it help's ! 09.07.2020 08:57:05 \| Asteroids@home \| Started upload of ps_200624_input_73675_24_2_0 09.07.2020 08:57:05 \| Asteroids@home \| Started upload of ps_200624_input_73649_6_2_0 09.07.2020 08:57:06 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 09.07.2020 08:57:06 \| Asteroids@home \| [error] Error reported by file upload server: can't open log file '../log_project1/file_upload_handler.log' (errno: 9) 09.07.2020 08:57:06 \| Asteroids@home \| Temporarily failed upload of ps_200624_input_73675_24_2_0: transient upload error 09.07.2020 08:57:06 \| Asteroids@home \| Backing off 00:38:05 on upload of ps_200624_input_73675_24_2_0 09.07.2020 08:57:06 \| Asteroids@home \| Temporarily failed upload of ps_200624_input_73649_6_2_0: transient upload error 09.07.2020 08:57:06 \| Asteroids@home \| Backing off 00:25:07 on upload of ps_200624_input_73649_6_2_0
ID: 6882 · Rating: 0 · rate: / Reply Quote

Brummig Send message Joined: 30 Jan 18 Posts: 8 Credit: 1,247,520 RAC: 667	Message 6884 - Posted: 9 Jul 2020, 8:30:51 UTC
The admin hasn't done any BOINC crunching since 2013 (http://asteroidsathome.net/boinc/show_user.php?userid=1), so he wont be experiencing any upload problems himself, and clearly he doesn't monitor this forum. I've sent him a PM in the hope of waking him up. Does anyone know how to get his attention using the unnamed Twitter account to which he referred?
ID: 6884 · Rating: 0 · rate: / Reply Quote

Zarck Send message Joined: 21 Jun 12 Posts: 15 Credit: 10,053,360 RAC: 18,014	Message 6885 - Posted: 9 Jul 2020, 8:32:22 UTC - in response to Message 6884.
Unable to return calculated units... ____________
ID: 6885 · Rating: 0 · rate: / Reply Quote

Author

Message

Kyong
Project administrator
Project developer
Project tester
Project scientist
Avatar

Send message
Joined: 9 Jun 12
Posts: 578
Credit: 52,667,664
RAC: 0

I am sorry for server outages which was happening recently. There were several factors which often caused lack of free space and some services crashed and I had to do an emergency shutdown on Saturday in order to fix everything so I couldn't inform you about it except for Twitter. There is still a hardware problem with one of servers which I have to fix but the server is not being used now so project should be stable now.

Radim Vančo (FoxKyong)

ID: 6857 · Rating: 0 · rate: