December 19, 2013 | Peter Bradwell

Ten recommendations to ISPs for dealing with over-blocking

Yesterday's Newsnight has helped demonstrate once again that over blocking by ISPs internet filtering systems is a real and serious issue. We've told the 'UKCCIS' over-blocking group how ISPs should start dealing with the problem.

We started looking closely at internet filtering by mobile networks a couple of years ago. We knew that we could try to learn lessons from the way their default-on systems worked that could be helpful if and when systems for domestic ISPs were rolled out. We found that it was hard to understand what was blocked and why and that over-blocking was a serious problem. We also found that it was hard to get the Government or ISPs to take it seriously. We published a report in May last year, jointly with LSE Media Policy Project, setting these things out.

So far it seems those lessons (we set out five earlier this year) have not been learnt. TalkTalk, BT and Sky now offer network level filters and we're seeing the same issues play out. Yesterday Newsnight helped demonstrate some of the overblocking issues, showing that filters designed to stop pornography also block sex education, sexual health and advice sites.    

We have joined the 'UKCCIS' group that has been set up to try to address over-blocking. (UKCCIS is made up of a number of 'working groups' that are set up to discuss issues related to online child safety.) We'd like to help the group ensure ISPs take concrete steps to deal with inevitable overblocking by their filtering systems. To kick start that process we have sent the group a summary of our concerns about what is happening now and made 10 recommendations for how ISPs could improve the way they deal with over-blocking. You can read what we sent them below.

We are clear that we don't agree with the Government's current approach - mandating network level filters and a 'one click to safety' approach. The 10 ideas below are about dealing with the problems with over-blocking as we see them now, but the Government should be thinking again about the best approach for parental controls. 

Let us know if you have other ideas for dealing with over-blocking in the comments below. 

Concerns about over-blocking and 'one click to safety' filtering

Filtering systems should adhere to four principles: transparency, accountability, choice and responsiveness. The Government's current approach of mandating network level filters and aiming for what David Cameron called 'one click to safety', is not conducive to policies that live up to these principles.

The result could be counter-productive to the Government's aims. For example, all users within a household will be subject to the same level of filtering at a given time, and there is a risk that in frustration at how unresponsive filters can be some account holders may simply turn them off.

Rather than addressing here this broader question about the best solution to the Government's policy goals, below are our top level concerns about the 'one click to safety' approach and some recommendations for addressing these issues.

Concerns about how over-blocking is currently dealt with

1. There is not enough clarity for users about what categories are blocked, what falls within those categories and why, and who makes these decisions. 

2. People who run sites that are blocked incorrectly...:

a. ...have no way of checking if and why their sites are blocked on different ISPs. As far as we are aware, only O2 provide a URL checker. Website owners are going to face multiple ISPs, who will use a variety of filtering systems. 

b. ...can find it difficult to report the problem. Issues can include knowing to speak to and getting a clear response / finding someone at the ISP who understands the issue.

c. ...can find it takes too long to get their site removed from blocking lists. On mobile networks we've seen cases taking a month to get resolved; recently, it took around a week to resolve issues TalkTalk users had accessing Wordpress admin pages (this related to TalkTalk's implementation of the IWF list).

3. There is no clear organisational responsibility for blocking mistakes.

4. It can be hard to find out technical details about how filtering works. This sort of detail may affect someone's decision about which ISP to use, or it may help website operators or users understand filter-related access issues.

Ten recommendations for addressing current over-blocking problems

1. ISPs should provide a one-stop URL checker to help people check if sites are blocked, which checks across ISPs.

2. ISPs should provide clear and consistent information for the user at the point of blocking and on their general customer service pages. At the point of blocking this should cover why a site is blocked and how to report mistakes. On FAQ and customer service pages that should include the categories blocked with explanations and examples of what those categories will block.

3. When mistakes or errors occur due to filtering, clear information should be provided quickly to users and affected sites about what has happened and why. 

4. ISPs should ensure training to ensure that customer service staff understand filter-related problems.

5. ISPs should commit to monitoring performance of their filtering accuracy and responsiveness and to publishing data about this performance. That should include, for example:

a. The number of over-blocking reports received, broken down by filtering category

b. The speed with which mistakes with blocking issues are resolved and sites are taken off blocked lists.

6. ISPs should set common performance standards against these metrics. Performance against these standards should be overseen by independent regulator.

7. ISPs should provide a process for site owners to proactively have their sites whitelisted. 

8. ISPs should offer a process for 'edge cases' (where suitability for under 18s may be disputed, for example) to be resolved. An independent regulator could arbitrate if disputes are not resolved. 

9. ISPs should publish who provides their filtering service and details of the technology involved. 

10. There should be a clear timetable for implementing these changes, we suggest by Spring 2014. Roll out of parental filters to existing customers should not proceed until these measures to mitigate over-blocking are in place.


Comments (1)

  1. Iain Collins:
    Dec 21, 2013 at 07:10 PM

    I concurr - staff should know more about that their own companies are doing, there should be easier ways to report over blocking and there should be provision of a way to check if a URL is filtered and why and there should be greater transparency generally.

    All companies can check which URLs are blocked internally and there is no reason not for it to be public on a web page, it could easily be IP limited to subscribers of the ISP and rate limited to prevent abuse.

    One of the ways to prevent over blocking (cribbed from memory from some
    recommendations I made the best part of a decade ago) is to change the format of lists.

    For example, in the case of lists distributed to ISPs, instead of a simple list of URLs it should be something like a list of IP ranges confirmed as correct by the list vendor (so malicious users can't just present different DNS entries to ISPs to avoid being filtered and so that someone manually checks that IPs belonging to the likes of CDN providers don't get on the list) along with the the hostname and port and path.

    Additionally the hostname and path could be hashed, so the list of specific URLs is never actually provided in the clear to providers (reducing risk). Normalising and hashing URLs sent over HTTP in real time is computationally cheap.

    (NB: This only needs to happen for requests to specific IPs, not all requests, and handling a few thousand requests per second is easy.)

    That there are still incidents of major sites like WordPress repeatedly being impacted - and targeted sites trolling ISPs into blocking IPs of other sites - suggests probably nothing has changed, which means this is going to keep happening for the forseable future.

    This is sad, especially as it was an anticipated problem. I'd like to be more explicit about that but I don't want to go into implementation specifics - it's sad to see these problems happening though.

    Though mistakes happen, there is no good excuse for the overblocking problems written about here dragging on for hours, let alone days. It likely is a result of someone just ingoring a big blinking red alert (which happens /daily/ in operations centers), though I wouldn't rule out that no one even bothered to set up a monitor in the first place either (which is equally as bad).

    Sadly, decent staff in ISP / telco operations centers is a notriously rare commodity - companies just don't invest enough and talented engineers try and escape out of the stress of operations roles for less stressful and better paid work.