Chuck Kollars: WorldWideWeb Filtering

Although Ipswich Middle/High School (IMHS) ties to not restrict student uses of the web too much, IMHS must do at least a little bit of filtering. IMHS must comply with the CIPA law, which more or less boils down to filtering access to pornography. In order to keep teachers happy, IMHS filters out sites whose principal purpose is to ~~waste~~ spend time. This especially means filtering out "twitch" and other mindless games, typically including almost all Flash-based games. In order to keep students out of trouble, IMHS filters out sites that make it very easy to do something illegal or unwise such as online gambling. And in order to keep students focused on education rather than non-educational uses of the web, IMHS filters out social networking sites.

IMHS' web filtering software runs directly on their firewall computer, with the whole internal LAN on one side and the world wide web WAN on the other. As the filter runs in parallel with other firewall limitations rather than in series with it, IMHS avoids maintenance problems where the filter would allow something yet the firewall continues to stop it.

The Open Source Software IMHS uses —DansGuardian (and Squid)— is covered widely:

(The DansGuardian wiki is often an especially good source for current information and information related to the way IMHS uses the software, and the DansGuardian Wiki FAQ includes a lot of more general information.)

IMHS uses the "transparent-intercepting" configuration family. By simply intercepting all traffic directed to port 80, IMHS gains several advantages. IMHS filters all uses of the web from their network, regardless of whether it comes from one of their computers or from a computer brought in from outside. IMHS makes it easy to use different browsers —even multiple browsers from the same computer— as no browser proxy settings at all are needed. And IMHS makes it easy for some computers (usually those assigned to teachers) to move back and forth from a home network to the school network without having to change any settings.

Of course using the "transparent-intercepting" configuration family has disadvantages too. One is that only port 80 (not even the https port 443) can be filtered. IMHS overcomes that disadvantage by using Shorewall/IPtables to filter the other ports including the https port 443. Another is that IMHS cannot reliably figure out which username is using which computer. IMHS overcomes that disadvantage by using IP addresses instead of usernames as identifiers. (IMHS have arranged that their DHCP never reuses an IP address, so IMHS can do this reliably.)

DansGuardian is a web content filter. Content filtering is an excellent approach for younger schoolchildren, but may not be appropriate at the high school level. So for a time DansGuardian was reconfigured so that during the schoolday it acted as a pure URL filter. DansGuardian reverted to its usual content filter configuration at night while the entire day's activity was replayed. In other words the DansGuardian content filtering capability was used as a review mechanism, pointing out problematic URLs so they could be added to the "pure URL filter" configuration immediately. This "review" process enabled keeping very close tabs on student web activity even while minimizing interference with web traffic.

Additionally, for a while filtering of search terms was implemented. Filtering of search terms is not directly implemented by DansGuardian (or any other known filter), and was technically rather difficult. Implementing search term filtering ultimately required both source code changes to DansGuardian and creation of special tools, and often resulted in exceedingly large (over 1000 characters) "regular expressions".

Even more problematic was that since no one else was doing anything similar, there were no example tool skeletons to follow and no advice was available. Even without any input from anyone else, some rules of thumb became obvious:

to hold overblocking down to a reasonable level, most blocks had to be for combinations of at least two words rather than just a single word
blocks had to work regardless of the order of search terms
blocks had to match only whole words and never inadvertently match partial words with similar spellings
there needed to be some sort of exception mechanism for allowing particular combinations that would otherwise be blocked
search term blocking had to be aware of extraneous punctuation with which users disguised word beginnings and endings

A hard lesson learned was that without a plethora of excellent management tools, filtering of search terms could easily result in erroneous yet unrealized massive overblocking. Search term filtering, which may or may not make sense with pure URL filtering, is clearly unnecessary with content filtering. So when the "review" strategy involving pure URL filtering was abandoned, "search term" filtering was abandoned as well.

Location: N42 40.86' W070 50.35'
(North America> USA> Massachusetts> Boston> North Shore> Ipswich)
Time: UTC-5 (USA Eastern Time Zone)
(UTC-4 summertime --"daylight savings time")
Email comments to Chuck Kollars

All content on this Personal Website (including text, photographs, audio files, and any other original works),

unless otherwise noted, are available to anyone for re-use (reproduction, modification, derivation, distribution, etc.) for any non-commercial purpose under a Creative Commons License.