Bell Canada traffic throttling saga

Fri,03 Jul 2009 10:36 GMT -4:00 http://www.p2pnet.net/story/24433

- Jon, just wanted to fill you in on this,” said Ottawa Gal a little earlier today. “New CRTC filing in reply to Bell’s non-reply to reopen the ‘throttling case’.”

Re-opening the throttling case?

She’d have been on this. But she’s taking a holiday break –” Not at my computer, can’t get to my computer, can’t get to Email and don’t have your email etched in memory … ”

So »»»

CAIP et al. via PIAC: http://www.dslreports.com/r0/download/1445389~572b62733237bb57791279c632e847e0/2008-108_RV_Reply_2July09FINAL.pdf

J.F. of Vaxination: http://www.dslreports.com/r0/download/1445390~f25d06fcfee813f8c1d16ddaaa08843d/vaxination_r%26v_final.pdf

In his document, Vaxination Informatique’s Jean-François Mezei says he’s in receipt of comments from:

CAIP et al

Distributel Communications

Bell Canada/Bell Alliant

CIPPIC representing Campaign for Democractic Media

Union des Consommateurs

Canadian Film and Television Production Association

Coalition of Internet Service Providers

Many personal comments sent to the CRTC on this issue

“Sympatico” refers to Bell Canada’s retail ISP service, he says, noting, “Despite branding changes, this term is still more recognised and provides clearer distinction between the retail ISP business and Bell Canada’s commercial network services, and going on »»»

Errors in scope

In paragraph 4, Bell Canada states:

4. These comments fail to acknowledge the fact that the CAIP Proceeding was limited in scope to a wholesale tariff dispute between Bell Canada and some of its wholesale ISP customers.

3.  If  Bell now agrees that the scope of the proceeding should be limited to the nature of the GAS service, why did it spend so much time and effort during the proceedings (both in its filings and with the media) trying to paint GAS as a white label reselling of its retail Sympatico service ?

4.  One can also challenge Bell Canada’s use of the word “wholesale”.  Wholesale is defined in the dictionary as the bulk selling of a product/service to be retailed by others.  GAS service just one of many commercial building blocks needed to create a complete solution, it cannot be resold in retail, and cannot be purchased by individuals. GAS does not provide connectivity to the internet. In fact, it is closer to a commercial telecommunications service purchased by banks linking their branches and ATMs to their data centres.  As such, the Commission must consider how precedents set by its GAS decision would apply to other commercial service such as those purchased by banks.

5.  GAS is completely different from Sympatico.

6.  In writing the 2008-108 decision  the Commission failed to recognize the true nature of the GAS service: a commercial, PPPoE based telecommunications service on which Bell Canada was applying retail throttling.  The Commission failed to recognize the differences with Sympatico and this lead to the Commission excluding valid arguments while including arguments applicable to Sympatico but not to GAS.

7.  The CAIP.vs.Bell process was about the use of throttling of P2P  by Bell Canada on traffic flowing on a commercial telecommunications service used by independent ISPs using the PPPoE protocol. Discussions  on throttling as well as its applicability on a commercial PPPoE service were not only within scope, but also required to reach a proper decision.  Discussions about individual ISPs (such as Sympatico) using DPI equipment on their retail customers was outside the scope of  the process.

8.  The 2008-19 process retained the discussion on throttling of P2P (and did not widen the scope of discussion to encompass additional capabilities of DPI equipment), and widened applicability to the retail ISP industry.

9.  However, the way with which the 2 processes were linked, as well as interviews given by senior CRTC officials gave the strong impression that the CRTC had rendered a decision before having fully studied the issue and was going to complete the study of throttling with the wider scope of 2008-19.  This is an image problem which the Commission needs to correct.

Safeguards are necessary

10.  Furthermore, while  reviewing this decision, the Commission must incorporate necessary safeguards to ensure that Bell Canada does not deviate from the revised decision. This was a major flaw in the original 2008-108 decision.

11.  In paragraph 21, Bell Canada classifies as being outside of the scope some of the text provided by Vaxination in 2008 which described various capabilities of DPI equipment. While Bell Canada states that none of those are enabled, the Commission failed to include any safeguards in its 2008-108 decision to ensure that Bell Canada does not enable DPI features beyond flow control.

12.  In paragraph 70 of the 2008-108 decision, the Commission imposed a notification requirement only where DPI equipment changes result in material impact on performance.  Bell Canada remains free to change/enable other features which do not have performance impacts.

13.  Having knowledge of the full capabilities of this DPI equipment,  the Commission failed in its duty to uphold article 7(i) of the Telecommunications act because it did not add any text to prevent Bell Canada from enabling features which would cause serious privacy violations without having to notify anyone since such feature would not have material impact on performance.

14.  The Commission based its 2008-108 decision largely on the fact that GAS and Sympatico were equally throttled, yet, the Commission failed to add any text in the decision to ensure that the throttling parity persisted and that Bell Canada could not eventually remove/change throttling from the Sympatico offering while maintaining it for GAS.

Discrimination cannot be ignored

15.  Contrary to what Bell states in its paragraph 16,   Vaxination’s submission did not imply Comcast’s implementation of throttling was a form of blocking. Vaxination’s submission  quoted an internal CRTC powerpoint slide, obtained via Access to Information, which showed in black and white that the Commission was aware that the FCC considered Comcast’s traffic management to be discriminatory.

16.  There are two issues to be considered independently when comparing Bell against Comcast:

• How the throttling is implemented (blocking issue)

• What  traffic is throttled (discrimination issue)

17.  Differences in how throttling was implemented do not deter from the fact that Bell discriminates just like Comcast did  by targeting only certain types of transmissions. Therefore, the comparison with Comcast is valid since both discriminate between transmissions based on whether they found certain data patterns in the payload of packets.

18.  Comcast was forced to and has implemented  non-discriminatory network management, showing that  Bell’s solution is not the only solution.

19.  The Commission publicly admitted in a CBC Interview  that it based its 2008-108 decision largely on the Bell Canada argument that there was no discrimination because Sympatico and GAS were equally throttled.  But,  the second paragraph in the introduction to the 2008-108 decision states:

The Commission’s determinations in this Decision relate solely to Bell Canada’s traffic-shaping practices in relation to its wholesale GAS, and are based on the evidence filed in this proceeding.

20.  Why then did the Commission place so much weight on a comparison with Sympatico which doesn’t use the tariffed GAS service, has different network interconnection to the ADSL cloud and more importantly, has an undisclosed internal business relationship with Bell Canada that makes any meaningful comparison impossible ?

21.  More importantly, in its decision, the Commission avoided the real discussion on discrimination, namely the fact that certain transmissions were targeted for discriminatory treatment based on some signatures found inside the payload of packets.

22.  The Commission failed to consider the real issue of discrimination which was raised by many parties, as well as the FCC’s own clear conclusion that the throttling of only certain transmissions was discriminatory.  This this a critical omission since the Commission rendered a decision which made throttling legal without actually analysing whether it was legal.

How is throttling implemented ?

23.  In both the CAIP.vs.Bell and the 2008-19 processes, the Commission conveniently avoided requiring the carriers to disclose exactly how they performed the throttling of certain transmissions only.

24.  In the November 20th opening of the 2008-19 consultation process, footnote 3 states:

3 In the context of CAIP’s application, throttling referred to the practice of slowing down the transfer rates of traffic by delaying certain data packets at certain points in the network.

25.  Delaying packets cannot be done because an overdue packet would simply trigger a retransmission because the original packet would be declared lost. The Commission, on the day it rendered 2008-108 and began 2008-19, showed that it did not understand what throttling really was.

26.  How can the 2008-108 decision be allowed to stay when it is clear that the Commission had not even looked at the way throttling was accomplished and whether it was legal within the context of the Telecommunications Act ?

27.  It is clear that the Commission rendered 2008-108 without having studied and understood exactly how throttling was accomplished, and that Bell Canada did not provide any information on how its equipment acted on packets to reduce throughput.

28.  Coming back to paragraph 16 and 17 which discuss the RST technique, Bell Canada, which had refused to confirm until now ANYTHING about how throttling was accomplished, finally agrees to state that it does now use the RST technique.

29.  At this point in time, it is necessary to inject a few facts into this discussion. The “Reset” (RST) technique is essentially the carrier forcing one side of a communication to hang-up and not telling the other side about it so the other side will wait and wait and wait. This is the technique that had been used by Comcast. This is described with more detail in  Appendix 2.

30.  My own experiments in the spring of 2008 (documented in my July 3 2008 submission, page 23) seem to point to Bell Canada’s DPI equipment dropping between 20% and 30% of packets in flows it has targeted for throttling.  See Appendix 1 for a brief description of how TCP manages flow of packets and how it relates to Bell Canada discarding packets.

31.  It must be noted that Bell Canada’s response, while confirming it does not use the Reset bit technique, does not confirm what technique it does use.  The Commission should have never rendered a decision granting the right to throttle before knowing what throttling really involved.

32.  It should also be noted that in the only interrogatory for the 2008-19 process (December 4th 2008), the Commission asked question 8 (c) in such a way as to allow Carriers to answer without having to reveal exactly how packets were treated by their DPI equipment. Knowing how thorttling affects packets  is important to ensure compliance with the Telecommunications Act.  Modifications of packets during transit with the purpose of disrupting communications may not be something the Commission would desire.

33.  Therefore, should the Commission insist on finding a way to allow Bell Canada to continue to throttle GAS service (since the scope of this process is limited to GAS service),  it must first get public confirmation from Bell Canada on how its equipment treats packets to force a reduced throughput in a transmission.

Does throttling block traffic ?

Block: [verb]

• make the movement or flow in (a passage, pipe, road, etc.) difficult or impossible

• put an obstacle in the way of (something proposed or attempted) :

• restrict the use or conversion of (currency or any other asset).

• Sports hinder or stop the movement or action of (an opponent).

34.  Bell Canada has stated many times in this process that it did not block P2P.  In Bell Canada’s comments, paragraph 16, the issue of blocking is brought up again. And in the CBC interview on the day the 2008-108 decision was rendered,  the CRTC co-chairman stated:Katz: The [U.S. Federal Communications Commission] situation is very different. Comcast was actually blocking traffic that was coming across the network. What they may have agreed to do is something totally different. We’ll look at what we think needs to be done in Canada, but Bell was not blocking at all. All they were doing was managing traffic on their network without impacting or influencing the content at all. Big difference.

35.  It must be noted that at the time the decision was rendered, Bell Canada had not confirmed nor published what technique was used to slow traffic down, and thus, the Commission was in no position to state that Bell Canada was not blocking at all. On that same day, the Commission had issued the 2008-19 letter in which they thought that in the context of CAIP’s process, throttling involved the delaying of packets, which is an error in fact.

36.  In the case of Comcast, they were randomly forcing disconnection of TCP flows, allowing eventual reconnection between two peers.  This reduces throughput. But requires the application compensate by re-opening new links to replace those terminated by the DPI equipment.

37.  Assuming that Bell Canada’s technique consists of dropping 20 to 30% of  TCP packets, this force peers

to stop sending while they wait for acknowledgements, reducing throughput.

38.  In both cases, the DPI equipment provide an impediment to the efficient flow of packets, hence one can use the term “block”.  In both cases, the application will eventually be able to reconstitute the contents that was intended to be transferred, albeit with many retransmissions, overhead and orders of magnitudes more time.

39.  Therefore, the Commission erred when it ruled that Bell Canada’s throttling was different enough from Comcast to escape the “blocking” and “discriminatory” labels.

Bell Canada’s real motives

40.  Because Bell Canada throttles specific types traffic 40% of the day, every day of the week, whether there is congestion or not, and because Bell Canada does not throttle other types of traffic which generate equal or greater loads on the network,  the Commission failed to consider whether Bell Canada had goals other than mere congestion management.

41.  In the context of GAS, if an ISP  purchases sufficient GAS/AHSSPI capacity to support its customers accessing innovative leading edge services, Bell Canada should have no reason to claim that users of this ISP are creating congestion problems. Bell Canada insisting on throttling such end users should have raised competition alarms at the Commission.

42.  When a GAS customer purchases a telecommunication service of a certain capacity, the carrier as no business decide what type of traffic can and cannot flow unimpeded on those links. This is a core concept in telecommunications which the Commission failed to uphold.

43.  It is outside the purview of this process to discuss what Sympatico can and cannot do as this process is limited to the GAS service.   And it is exactly because of this that  the Commission must keep GAS and Sympatico totally separate and not allow service features imposed on Sympatico customers to also be imposed on GAS end users. As long as a GAS customers purchases sufficient capacity to support the usage generated by its end users, there should not be any complaints by Bell about some users negatively impacting others.

44.  The Commission failed to uphold the basic need to allow each service provider to independently define its service offering to maintan a competitive environment. Allowing Sympatico, a retail service outside the scope of this process, to impose service features on individual links inside a commercial GAS service is extremely anti competitive.

45.  Therefore, the Commission must send a strong signal to Canadians that it will uphold basic principles of telecommunications which require a carrier to be content agnostic, manage a service according to the protocol which defines that service and not impose any discrimination on certain contents just because it has unilaterally decided it doesn’t like them.

46.  The Commission must also uphold the core principle that if you buy a certain amount of capacity from a common carrier, the later should be made to provide this capacity.

Other Notes

47.  In paragraph 9 of Bell Canada’s June 22nd submission, Bell Canada contents that both CAIP and Vaxination are repeating arguments already presented in the original proceeding, that Bell Canada had already answered those in its 85 page submission of July 11th and that the Commission had already ruled on.

48.  As part of a Review and Vary process, once needs to outline those failures to consider a basic principle which had been raised in the original proceeding . It is the duty of the Applicants in an R&V to point to those issues which the Commission failed to properly address in its decision.  The fact that so many points could be raised again between the 2 Applicants shows how many issues the Commission had chosen to ignore, despite those issues being relevant within the scope of a thorttled GAS service.

49.  In paragraph 10, Bell Canada refers to its  85 page submission of July 11th. Unfortunately, those with enough technical background to tackle the enormous amount of misleading statements and errors contained in that submission did not have the opportunity to submit comments due to the structure of the process, and it was extremely worrisome to see that not only had the Commission chosen to believe all of Bell Canada’s statements and failed to challenge even the most blatant errors, but it included many significant errors in the final decisions, including the most serious one, the assertion that Bell Canada’s DPI equipment did not look at packet contents.

50.  The Commission committed a grave error in fact when it decided to believe Bell Canada’s statement that its DPI equipment did not look at packet contents, despite the overwhelming evidence presented to the Commission to this effect, and despite the very definition of Deep Packet Inspection equipment.

51.  Once the Commission agrees to the truth on this issue, it will be forced to admit that this type of network management is discriminatory, just as the FCC had been forced to admit.

52.  Overall, in Bell’s 85 page submission, the majority of blatant errors were caused by Bell Canada treating GAS as if it was just a resold Sympatico service.  It is ironic to see Bell Canada complain about scope here when its own 85 page submission was guilty of being so far out of scope.  This 85 page document was unfortunately successful at influencing the Commission into comparing GAS with Sympatico and  concluding there was no discrimination.

53.  It is therefore important that the Commission realise and admit that the GAS service is a PPPoE service which is extremely different from Sympatico and take corrective actions to eliminate all comparisons with Sympatico in its analysis, as well as removing all Bell Canada arguments that portray the GAS service as being the same as Sympatico.

About ATM and purchased capacity

54.  In paragraphs 12-15, Bell Canada discusses upgrades to Ethernet switches and ancient ATM DSLAMS.

55.  The Commission is faced with many GAS related files at the moment. Most of them include complaints by Bell Canada about congestion in its ATM lines causing problems etc. The appearance given is that Bell Canada is no longer upgrading capacity in its ATM network. It is understandable that a carrier would want to stop upgrading ancient technology and seek to migrate to current technology.

56.  What is not understandable is why Bell Canada has not upgraded the all of the  GAS infrastructure to Ethernet yet.  American ADSL networks have long ago converted to ethernet.  There are still a number of ISPs in Ontario/Québec whose AHSSPI links are limited to ATM technology because apparently, this is all Bell can  offer in certain wire centres.

57.  Why is this related to this R&V ? Because the Commission failed to do due diligence before rendering its 2008-108 decision and blindly accepted Bell’s statements that there was congestion. It failed to investigate whether Bell Canada had misused funds generated by the GAS service to upgrade portions not used by GAS, and then restrict GAS with throttling because of congestion due to lack of capacity increases on the old segments still used by GAS.

58.  This is the reason my Application had mentioned that the Commission should consider the upcoming costs analysis to be filed by Bell Canada on July 10th to ensure that funds generated by GAS over the last few years have been re-invested in GAS infrastructure upgrades.

Appendix 1

Description of TCP flow control

59.  The 2 peers in a TCP session adjust packet sending rate based on measured round trip time and by detecting undelivered packets. To use a telephone conversation analogy, when the line is good, one speaker will say a complete telephone number without pause and then wait for the other to acknowledge he got all 7 digits. When line conditions are bad, he may say the first 3 digits and wait for acknowledgement and then proceed with the last 4 digits and wait for acknowledgement, And when line conditions are very bad, he will wait for acknowledgement after saying each digit. How many “digits you can say” without an acknowledgement is called a window size.  The lower the window size, the more “pauses” are inserted on the line while the sender waits for the recipient to acknowledge.

60.  TCP was designed to make most efficient use of available capacity. This is not a P2P characteristic, it is a characteristic common to all applications that use TCP, including the web (HTTP),  mail (SMTP) etc.

61.  The flow control is done at the TCP software level, which means that all applications that use TCP as session layer will behave exactly the same way on the Internet in terms of using all available bandwidth and managing congestion and line condition problems.

62.  When the sender detects that it has packets that have been unacknowledged for too long, it will resend the unacknowledged packet(s). (there are other ways to cause sender to resend a packet).  As well, the sender will reduce its window size. This will result in suboptimal use of the line, reducing throughput because the sender will have to pause regularly to wait for acknowledgements.

63.  After an “event” where the window size was reduced, the sender will slowly increase the window size again to test the waters in case line conditions have improved. If this was a temporary “blip”, the transmission eventually resumes at its optimal throughput. If the line conditions have not improved, further packet losses will be encountered which will keep the window size low and thus keep the throughput low.

64.  In spring of 2008, I performed packet traces and found that BitTorrent flows had sustained packet loss rates between 20 and 30%.  To prevent peers from raising throughput after a dropped packet (since there is not much actual congestion), the DPI equipment must constantly drop packets to simulate a permanently poor/congested link.

65.  HOWEVER, for every packet that the DPI equipment purposefully destroys and which never arrives to destination, it means that the sender must eventually resend that packet, and this increases the amount of data that needs to be transmitted. If 25% of packets are destroyed by Bell’s DPI equipment, this increases the amount of data the sender has to send by 25%.

66.  Bell Canada’s actions can also impact some innocent sender in another continent. If he had intended to send 1 megabytes of data to a Canadian user, but Bell Canada’s DPI equipment forces him to send 1.25 megabytes, he may  incur higher  transmission costs (especially if sent from a mobile phone).

67.   It must be noted that on the Internet, dropping packets is the accepted practice to handle congestion. HOWEVER,  on the internet, a transit provider that would have a sustained  a 25% rate of packet loss for 40% of the day every day of the week would quickly lose all its customers and go bankrupt. Only a monopoly would survive such under-provisioning of lines because customers would have nowhere else to go. Congestion problems are normally temporary in nature.

Appendix 2

How the Reset bit works

68.  On the subject of the CAIP et al.’s  question of   “RST injection”.  The Comcast implementation achieved

its goal of slowing down traffic by simulating one peer having “hung up” without warning.

69.  This is done with the DPI equipment modifying the RST bit field in the packet header.1 This tells the packet’s recipient to abruptly end any communications using this TCP session and that it must be torn down unilaterally without handshake.

70.  Consider  computers A and B having setup a TCP session between port A-3000 and B-5000.  All is fine and dandy and packets flow in both directions. But B crashes and reboots.  At that point, B is no longer aware of the former link to A. And when A sends it a packet from its port 3000 to B’s port 5000, B will reply with a packet having the Reset bit set, telling A that this session no longer exists. At this point, A destroys this session without further handshakes with B.  A can then re-establish a new session with B at which point, both A and B will be able to communicate again.

71.  In the context of throttling, when a packet from B to A arrives with a reset bit having been set by the DPI equipment, A immediately stops communicating with B. However, B still has a valid session with A and it may take some time before B realises that A no longer has a valid TCP session with it.

72.  Eventually, A will again connect to B and start exchanging data again. But the disruption caused by the reset bit causes delays and reduced throughput.

1 This also involves recalculating the packet’s checksum and changing it so that the recipient doesn’t realise that the packet was changed in transit.

“I hope to see the Consumer Unions Reply, if any,” says Ottawa Gal.

Thanks for taking the trouble, Jean-François, and thanks Ottawa Gal.

Stay tuned.

Follow on Twitter.

More

First they ignore you, then they laugh at you, then they fight you, then you win ~ Mahatma Gandhi

July, 2009




Subscribe to p2pnet.net | | rss feed: http://p2pnet.net/p2p.rss | | Mobile - http://p2pnet.net/index-wml.php