Can it be that 3.5.24 (on OPNsense (FreeBSD)) has broken ICAP support? After a TCP handshake, a Segment with an RST is sent without trying to send an ICAP request. Had this issue a year ago on Ubuntu as well. It results in an inaccessible Web if the service is mandatory (for example an AV engine). c-icap-client is working from the same machine, so the network is running and working. Another user reported the same issue with a commercial AV engine: https://forum.opnsense.org/index.php?topic=4579.msg17571
Additional Information: The squid log says it marks the service as down due too many failed attempts.
I recommend reproducing the problem with a single transaction through an otherwise idle Squid and posting the corresponding ALL,9 logs.
Created attachment 3527 [details] Packet capture on the ICAP Server This is a PCAP from my Laptop running the ICAPrb Server. As you would expect, the error log looks like this: I, [2017-03-23T08:21:55.404297 #2319] INFO -- : [CONNECT] Client from 10.230.231.1:61255 connected to this server I, [2017-03-23T08:24:12.697643 #2319] INFO -- : [CONNECT] Client from 10.230.231.1:54469 connected to this server E, [2017-03-23T08:24:12.714265 #2319] ERROR -- : [PARSER ERROR] Error while parsing request - Error Message is: Connection reset by peer @ io_fillbuf - fd:9
The log does not show anything: squid -N -d 9 -f /usr/local/etc/squid/squid.conf 2017/03/23 08:51:55| Starting Squid Cache version 3.5.24 for amd64-portbld-freebsd11.0... 2017/03/23 08:51:55| Service Name: squid ^C2017/03/23 08:53:03| Shutdown: NTLM authentication. 2017/03/23 08:53:03| Shutdown: Negotiate authentication. 2017/03/23 08:53:03| Shutdown: Digest authentication. 2017/03/23 08:53:03| Shutdown: Basic authentication. Also the cache log keeps empty 2017/03/23 08:53:39| Creating missing swap directories 2017/03/23 08:53:39| No cache_dir stores are configured. 2017/03/23 08:54:00 kid1| Starting Squid Cache version 3.5.24 for amd64-portbld-freebsd11.0... 2017/03/23 08:54:00 kid1| Service Name: squid 2017/03/23 08:54:00| pinger: Initialising ICMP pinger ...
The packet capture you posted shows that Squid terminates an ICAP connection before sending any requests. This could be caused by Squid reconfiguration or other unusual conditions (including Squid bugs). To triage further, I still recommend reproducing the problem with as few transactions as possible going through an otherwise idle Squid (if possible) and sharing the corresponding cache.log with debug_options set to ALL,9 (always possible). Without this, you are not giving developers enough information to work on.