Weird socket issue with multiple services running

jamie commented

2018-07-23 01:10:43 +00:00

(Migrated from git.hazaar.io)

So I've been chasing down a problem with Warlock services. I have a production system that runs for about a day or 3 and then starts having socket issues (broken pipe). It is supposed to bomb out with exitcode 4 if that happens, but it is not. I know why and that is because as a service the Service class is supposed so catch the exception and exit, but I think the Process::send() is being called from somewhere else so I don't know where it's being called from to catch the exception.

Step 1 will be to add a tracelog to where it internally catches the exception and sends the debug message. From there I should be able to trace where it is calling the function and either fix permanently or at least catch the exception so it can exit(4) and restart.

The weird part of this, is that on Windows when closing the server, the services will not exit correctly if there is more than one running at the time. If there is only one service, exit is working.

With multiple services I get:

2018-07-23 11:12:44 - NOTICE - PROCESSING EVENT QUEUE: RQN-6806-6987
----------------------------------------
ERROR #2
File: D:\Source\abe-web\ABE\vendor\hazaarlabs\hazaar-warlock\src\Server\Master.php
Line: 745

socket_recv(): unable to read from socket [10054]: An existing connection was forcibly closed by the remote host.

----------------------------------------
2018-07-23 11:12:48 - NOTICE - Remote host closed connection
2018-07-23 11:12:48 -  DEBUG - DISCONNECT: CLIENT=f333dd28-a703-8a9e-9e87-177da7350b5f SOCKET=Resource id #52
2018-07-23 11:12:48 -  DEBUG - CLIENT_SOCKET_CLOSE: SOCKET=Resource id #52
2018-07-23 11:12:48 -  DEBUG - REMOVE: CLIENT=52
----------------------------------------
ERROR #2
File: D:\Source\abe-web\ABE\vendor\hazaarlabs\hazaar-warlock\src\Server\Master.php
Line: 745

socket_recv(): unable to read from socket [10054]: An existing connection was forcibly closed by the remote host.

----------------------------------------
2018-07-23 11:12:48 - NOTICE - Remote host closed connection
2018-07-23 11:12:48 -  DEBUG - DISCONNECT: CLIENT=067c9d1f-b064-81c6-9a10-ef2c59c6814c SOCKET=Resource id #53
2018-07-23 11:12:48 -  DEBUG - CLIENT_SOCKET_CLOSE: SOCKET=Resource id #53
2018-07-23 11:12:48 -  DEBUG - REMOVE: CLIENT=53
------------------------------
^C
------------------------------
2018-07-23 11:12:49 -  DEBUG - SERVICE=commit EXIT=-1073741510
2018-07-23 11:12:49 - NOTICE - 5b552b8b754c6 - STATUS: COMPLETE
------------------------------
^C
------------------------------
2018-07-23 11:12:49 -  DEBUG - SERVICE=notify EXIT=-1073741510
2018-07-23 11:12:49 - NOTICE - 5b552b8b75cb4 - STATUS: COMPLETE
2018-07-23 11:12:49 -  DEBUG - EXPIRE: NAME=notify TRIGGER=5b552b8cc2b3a
2018-07-23 11:12:49 -  DEBUG - EXPIRE: NAME=notify TRIGGER=5b552b8cd1ad3
2018-07-23 11:12:50 - NOTICE - 5b552b8b754c6 - Cleaning up
2018-07-23 11:12:50 - NOTICE - 5b552b8b75cb4 - Cleaning up

So I've been chasing down a problem with Warlock services. I have a production system that runs for about a day or 3 and then starts having socket issues (broken pipe). It is supposed to bomb out with exitcode 4 if that happens, but it is not. I know why and that is because as a service the `Service` class is supposed so catch the exception and exit, but I think the `Process::send()` is being called from somewhere else so I don't know where it's being called from to catch the exception. **Step 1** will be to add a tracelog to where it internally catches the exception and sends the debug message. From there I should be able to trace where it is calling the function and either fix permanently or at least catch the exception so it can `exit(4)` and restart. The weird part of this, is that on Windows when closing the server, the services will not exit correctly if there is more than one running at the time. If there is only one service, exit is working. With multiple services I get: ```shell 2018-07-23 11:12:44 - NOTICE - PROCESSING EVENT QUEUE: RQN-6806-6987 ---------------------------------------- ERROR #2 File: D:\Source\abe-web\ABE\vendor\hazaarlabs\hazaar-warlock\src\Server\Master.php Line: 745 socket_recv(): unable to read from socket [10054]: An existing connection was forcibly closed by the remote host. ---------------------------------------- 2018-07-23 11:12:48 - NOTICE - Remote host closed connection 2018-07-23 11:12:48 - DEBUG - DISCONNECT: CLIENT=f333dd28-a703-8a9e-9e87-177da7350b5f SOCKET=Resource id #52 2018-07-23 11:12:48 - DEBUG - CLIENT_SOCKET_CLOSE: SOCKET=Resource id #52 2018-07-23 11:12:48 - DEBUG - REMOVE: CLIENT=52 ---------------------------------------- ERROR #2 File: D:\Source\abe-web\ABE\vendor\hazaarlabs\hazaar-warlock\src\Server\Master.php Line: 745 socket_recv(): unable to read from socket [10054]: An existing connection was forcibly closed by the remote host. ---------------------------------------- 2018-07-23 11:12:48 - NOTICE - Remote host closed connection 2018-07-23 11:12:48 - DEBUG - DISCONNECT: CLIENT=067c9d1f-b064-81c6-9a10-ef2c59c6814c SOCKET=Resource id #53 2018-07-23 11:12:48 - DEBUG - CLIENT_SOCKET_CLOSE: SOCKET=Resource id #53 2018-07-23 11:12:48 - DEBUG - REMOVE: CLIENT=53 ------------------------------ ^C ------------------------------ 2018-07-23 11:12:49 - DEBUG - SERVICE=commit EXIT=-1073741510 2018-07-23 11:12:49 - NOTICE - 5b552b8b754c6 - STATUS: COMPLETE ------------------------------ ^C ------------------------------ 2018-07-23 11:12:49 - DEBUG - SERVICE=notify EXIT=-1073741510 2018-07-23 11:12:49 - NOTICE - 5b552b8b75cb4 - STATUS: COMPLETE 2018-07-23 11:12:49 - DEBUG - EXPIRE: NAME=notify TRIGGER=5b552b8cc2b3a 2018-07-23 11:12:49 - DEBUG - EXPIRE: NAME=notify TRIGGER=5b552b8cd1ad3 2018-07-23 11:12:50 - NOTICE - 5b552b8b754c6 - Cleaning up 2018-07-23 11:12:50 - NOTICE - 5b552b8b75cb4 - Cleaning up ```

jamie commented

2018-07-23 01:13:19 +00:00

(Migrated from git.hazaar.io)

changed the description

jamie commented

2018-07-23 01:13:51 +00:00

(Migrated from git.hazaar.io)

changed the description

jamie commented

2018-08-07 11:37:24 +00:00

(Migrated from git.hazaar.io)

I think I may have fixed this in the last few commits. Basically, the reason the exception was not being caught was because PHP changed the way it handles certain errors in PHP7. The end result being that Exception is not the base class we want to use in our catch() structures. This is because there is now a new "exception" type called Error which is used to handle these previously "non-catchable errors". Both Exception and Error now use a common base class called Throwable so for our "catchall" we need to use catch(\Throwable) instead. This fixed the service exit on an error exception.

The socket issues may be because of buffer problems which have also now hopefully been fixed.

I think I may have fixed this in the last few commits. Basically, the reason the exception was not being caught was because PHP changed the way it handles certain errors in PHP7. The end result being that `Exception` is not the base class we want to use in our `catch()` structures. This is because there is now a new "exception" type called `Error` which is used to handle these previously "non-catchable errors". Both `Exception` and `Error` now use a common base class called `Throwable` so for our "catchall" we need to use `catch(\Throwable)` instead. This fixed the service exit on an error exception. The socket issues may be because of buffer problems which have also now hopefully been fixed.

jamie commented

2018-08-07 11:37:25 +00:00

(Migrated from git.hazaar.io)

closed

jamie commented

2018-08-07 23:54:29 +00:00

(Migrated from git.hazaar.io)

mentioned in issue #24

Rows
Columns

Weird socket issue with multiple services running #31