Thread Links Date Links
Thread Prev Thread Next Thread Index Date Prev Date Next Date Index

Re: [EFM] Re: Pause frame usage in transport networks




Jiang,

The "pause" used by 802.3 is based on 802.3x, not what is being developed 
by 802.3ah.  The 802.3x "pause" is part of the active flow control that has 
been in place and has been implemented by many vendors for some time.  It 
uses specifically defined MAC Control frames similar to what is used for 
the OAM defined by 802.3ah.

This level of active flow control operates very differently than the "end 
to end flow/congestion control defined by IETF for TCP.  802.3x tends to 
prevent data loss, while the TCP functionality requires data loss.  802.3 
active flow control operates  at the "link" level, not the session control 
level.

The question here is whether an end to end "Ethernet" session can be 
considered to be a single "link" across the service provider 
network.  E.etna has it defined as three separate and independently 
operating links.  X.86 has it defined as a single flow controlled link end 
to end.

The functionality of this will have massive consequences relative to the 
delivery of "high end" interactive applications that are seen as the 
economic growth industries of the future.  If 802.3ah does not have much of 
the same inherent characteristics that existing 802.3 standard 
implementations have, then "customer" satisfaction and reliance on 
"Ethernet" as a transmission link protocol will be lost.  If the 
"transmission" standards body insists on treating "Ethernet" in the same 
way that all historic transmission protocols have been treated, then the 
term "Ethernet" will have no more meaning than any other "marketing 
label".  This is an area where the majority of the people in 802.3 have 
never been.  It is also an area that the IETF as well as SG15 and SG16 have 
never been.

Existing GbE has a latency variance of about 1.2us with a data frame loss 
of about 0.00000001 per second at 98% utilization.  Leased circuit 
SONET/SDH imposes a latency variance of 125us with a data packet/frame 
loss/errors of about 0.00000001 per second at 99% utilization.  (These are 
hard numbers that were determined by testing under controlled conditions 
over a period of time.)   This is what 802.3ah will be competing with at 
the link and transmission level.  The fact that most participants of the 
802.3ah TF are thinking in terms of Internet Protocol performance and 
reliability has caused them to loose sight of the fact that that there are 
a lot of other network level protocols that are not designed for 
"battlefield" conditions.  Many of these other protocols much better 
inherent performance and reliability characteristics, or require the 
inherent performance and reliability characteristics that the existing 
802.3 deliver.  In order to achieve architecture designed effective 
utilization of greater than 30% , something close to the inherent 
performance and reliability of the existing GbE and SONET/SDH facilities 
are required.

Thank you,
Roy Bynum

At 10:55 AM 2/20/2003 +0800, Jiang Shengming wrote:
>Hi Roy, Ben, Siamack,
>
>Sorry to interrupt your discussion regarding "pause". I just have some 
>questions
>related to functions similar to "pause" as I understand.
>
>1) According to the MAC standard of 802.3ah (if I understand it correctly),
>any ONU should negotiate with the OLT (here, ONUs and the OLT
>are within the same EPON) through "request-grant" protocol. When the OLT
>"grants" the request from an ONU, does this mean that the OLT has already
>considered the congestion issue in itself?
>
>2) If the above "request-grant" protocol can avoid congestion in the OLT, can
>the similar protocol be used from the OLT to ONUs to avoid congestion caused
>by traffic from the OLT in ONUs?
>
>3) Regarding "ONU at one end of an end to end link "pause" the ONU at the
>other end", is this kind of "pause" necessary if TCP is used for end-to-end
>flow/congestion control (here I guess the above ONUs are within different
>EPONs)?
>
>Thank you all very much for any clarifications to my questions.
>
>Best regards
>
>Shengming Jiang
>ICR
>
>Roy Bynum wrote:
>
>>
>>Ben, Siamack,
>>
>>In these comments I am seeing the movement toward an link facility that 
>>can be labeled as "Ethernet", but has little or none of the inherent 
>>characteristics of reliability and low latency variance.
>>Any Ethernet service that claims to be full duplex, but drops frames 
>>without generating a "collision" when congested will fail meet the basic 
>>reliability characteristic of any 802.3 standard that I am aware 
>>of.  This is NOT Ethernet.
>>
>>The ONU should be allowed to "pause" the GFP/OLT on any one link, and the 
>>GFP/OLT, should be allowed to "pause" the ONU on any one link.
>>With proper configuration of the operands, an ONU at one end of an end to 
>>end "link" should be able to "pause" the ONU at the other end.
>>Without that capability, what is being defined is no more reliable that 
>>what exists today, and is some respects is less reliable than the alternative.
>>
>>When an end to end link starts "dropping" frames, the data packets get 
>>retransmitted in new frames which now adds to the congestion of a link 
>>and thus lowers the effective access bandwidth that can be utilized.
>>This has the effect of lowering the effective committed information rate 
>>even more.  This is one of the primary reasons that experienced WAN 
>>networking architects design IP networks to run as a nominal 30% 
>>utilization.  (I remember reading something by David Boggs about ~30% 
>>utilization being the effective performance ceiling of congestion domain 
>>networks.  I turns out that he knew what he was talking about.)
>>
>>The current defined X.86/OLT does make use of "pause" functionality, and 
>>will allow an ONU at one end of an end to end link "pause" the ONU at the 
>>other end.  By properly use of active flow control, the service 
>>communications link can perform as a non-congestion domain link.  With 
>>properly configured operands at the ONUs, this would be highly reliable 
>>at the cost of lower predetermined bandwidth utilization, but without 
>>retransmissions.  In experiments performed in the 1998-2000 time frame, 
>>effective utilization was found to be a direct ratio of link/circuit 
>>speed to distance (I am having to write this from memory because I no 
>>longer have access to the data.)  Properly configured use of active flow 
>>control can allow architecture designs with much higher effective 
>>bandwidth utilization than 30%.
>>
>>Thank you,
>>Roy Bynum
>>
>>
>>
>>At 10:15 PM 2/18/2003 -0500, Siamack Ayandeh wrote:
>>
>>>Ben,
>>>
>>>Please see my comments interleaved.
>>>
>>>Regards, Siamack
>>>
>>>Ben Brown wrote:
>>>
>>> > Siamack,
>>> >
>>> > This comment is way off track of the original question but I
>>> > feel a need to ask this question. That's why I changed the
>>> > subject. I'll even redraw the network so that we're all using
>>> > the same context.
>>> >
>>> > ONU1 ------ OLT1/GFP ------------------- GFP/OLT2 ------ ONU2
>>> >     Ethernet                SONET                Ethernet
>>> >
>>> > Why are Pause frames used on the Ethernet links? The ONU should
>>> > never be allowed to Pause the OLT as that would back-pressure
>>> > the entire WAN. Since the WAN doesn't support back-pressure,
>>> > packets over the SONET link that exceed the OLT's egress buffers
>>> > would wind up being dropped at the OLT.
>>>
>>>That's basically what I said "OLT can simply buffer and subsequently 
>>>drop packets."
>>>
>>> >
>>> >
>>> > The OLT could Pause the ONU but for what reason? The only reason
>>> > that I can think of would be to enforce the ONU's SLA. The point
>>> > of putting an SLA in place is to enforce some set of rules,
>>> > usually having to do with the minimum and maximum throughput
>>> > guaranteed at the OLT for the ONU. The reason an SLA needs to
>>> > be in place is because each side doesn't really trust the
>>> > other's "handshake" and a legal document of sorts is needed.
>>> > So, if neither side "trusts" the other, why do you rely on
>>> > Pause frames to enforce the SLA? If the ONU will attempt to
>>> > use as much bandwidth as possible, it will likely do so by
>>> > ignoring the Pause frames from the OLT. This means that the
>>> > only way the OLT can truly enforce the SLA is to be able to
>>> > discard the frames that exceed the bandwidth agreed to in
>>> > the SLA. If the OLT is capable of this, why even bother with
>>> > Pause frames?
>>>
>>>You forget two things:
>>>1. The bursty nature of traffic and the fact that peak is greater than 
>>>the committed
>>>rate of service
>>>2. That networks by definition protect themselves i.e. can not rely on 
>>>subscriber
>>>alone
>>>
>>>If the subscriber exceeds its SLA then yes, frames would be dropped. But 
>>>if it's just
>>>a burst, i.e. on average the subscriber is in compliance then buffering 
>>>and Pause
>>>should do the job.
>>>
>>> >
>>> >
>>> > Sorry for being long winded but I'm trying to make a logical
>>> > argument. What assumptions did I make that aren't valid?
>>> >
>>> > Thanks,
>>> > Ben
>>> >
>>> > Siamack Ayandeh wrote:
>>> > >
>>> > > Shahram,
>>> > >
>>> > > It would be helpful to this discussion if you indicate what you had 
>>> in mind
>>> > > with regard to OAM frames so one can think of a pragmatic 
>>> answer.  For example
>>> > > if you are worried about Pause frames:  In this case the SONET/SDH 
>>> (OLT1-OLT2)
>>> > > is often the bottleneck link. So Pause is used on the local link to 
>>> protect the
>>> > > OLT-1/2 buffers. If the egress ONU back pressures the network, then 
>>> OLT can
>>> > > simply buffer and subsequently drop packets. Otherwise fairly large 
>>> buffers
>>> > > would be required to absorb the round trip time of the wide 
>>> area.  If you think
>>> > > about it as a two port bridge, again Pause is not propagated over 
>>> the wide
>>> > > area. If you look at various IETF Pseudowire flavors, again 
>>> following the lead
>>> > > of IEEE, Pause is not propagated over the wide area.
>>> > >
>>> > > Siamack
>>> > >
>>> > > Geoff Thompson wrote:
>>> > >
>>> > > > Matt-
>>> > > >
>>> > > > To further enrich the information that you provided below....
>>> > > >
>>> > > > 802.3 has a long, well established tradition of not supporting media
>>> > > > converters.
>>> > > >
>>> > > > It would be my opinion that any "features" designed to 
>>> specifically support
>>> > > > media converters were out of scope unless they were specifically 
>>> mentioned
>>> > > > in the PAR.
>>> > > >
>>> > > > Geoff
>>> > > >
>>> > > > At 01:05 AM 2/12/2003 -0500, Matt Squire wrote:
>>> > > >
>>> > > > >My 2 cents.
>>> > > > >
>>> > > > >If there is a MAC layer in the ONU, then OAM terminates there.  If a
>>> > > > >vendor builds something without a MAC layer, then it doesn't 
>>> terminate
>>> > > > >there.  If you put MACs in your ONU, you're really building 
>>> something
>>> > > > >akin to a 2-port bridge (likely with STP disabled 
>>> permanently).  If you
>>> > > > >don't, then its more along the lines of a media converter.
>>>Both models
>>> > > > >can work.  Both models can interoperate.  So do it anyway you 
>>> want, and
>>> > > > >maybe the market will agree with you.
>>> > > > >
>>> > > > >- Matt
>>> > > > >
>>> > > > >Shahram Davari wrote:
>>> > > > > >
>>> > > > > > Roy,
>>> > > > > >
>>> > > > > > Since EOS is a private line service, It seems that you agree 
>>> with me
>>> > > > > and Chak that for the EOS it seems from architectural point of 
>>> view the
>>> > > > > OAM MAC frames should not be terminated at OLTs. Is that correct?
>>> > > > > >
>>> > > > > > Yours,
>>> > > > > > -Shahram
>>> > > > > >
>>> > > > > > > -----Original Message-----
>>> > > > > > > From: Roy Bynum [mailto:rabynum@xxxxxxxxxxxxxx]
>>> > > > > > > Sent: Tuesday, February 11, 2003 12:26 PM
>>> > > > > > > To: Chau, Chak; stds-802-3-efm@ieee.org
>>> > > > > > > Subject: RE: [EFM] Question regarding OAM in 802.3ah D1.3
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > Chak,
>>> > > > > > >
>>> > > > > > > Depending on the service definition and who owns the service
>>> > > > > > > facilities,
>>> > > > > > > end to end OAM functionality would not necessarily 
>>> available.  In all
>>> > > > > > > packet services, the service provider would own at least a
>>> > > > > > > portion, if not
>>> > > > > > > all of the communications facilities.  The OAM functionality
>>> > > > > > > would then be
>>> > > > > > > for the use of the service provider, not the customer.  Only
>>> > > > > > > with a Leased
>>> > > > > > > Circuit type of service (also referred to as "Private Line")
>>> > > > > > > would end to
>>> > > > > > > end, OLT1 to OLT2 OAM functionality exist.  For all other 
>>> types of
>>> > > > > > > services, the OAM for Link 1 would not be the OAM for Link 2.
>>> > > > > > >  Other than
>>> > > > > > > with a Leased Circuit service, the services are defined to
>>> > > > > > > alter, filter,
>>> > > > > > > or drop customer originated frames/packets in one way or
>>> > > > > > > another, including
>>> > > > > > > the OAM frames.  This is the gist of the work that is being 
>>> done by
>>> > > > > > > SG15/Q.10 under E.Ethna.  Other than with a Leased Circuit
>>> > > > > > > service, true
>>> > > > > > > "transparency" does not exist.
>>> > > > > > >
>>> > > > > > > Thank you,
>>> > > > > > > Roy Bynum
>>> > > > > > >
>>> > > > > > > At 10:15 AM 2/11/2003 -0600, Chau, Chak wrote:
>>> > > > > > > >Hi David and All,
>>> > > > > > > >
>>> > > > > > > >I though for a PtP application the OAMPDUs message should be
>>> > > > > > > transparent
>>> > > > > > > >form OLT1 to OLT2.
>>> > > > > > > >Or else, this will defeat the purpose of PtP, i.e., no L2 
>>> frame
>>> > > > > > > >processing. Which means that OAM for Link1 can be OAM for
>>> > > > > > > Link2, is that
>>> > > > > > > >correct?  This topic may be reviewed outside of EFM if 
>>> prefered.
>>> > > > > > > >
>>> > > > > > > >Kind Regards,
>>> > > > > > > >Chak
>>> > > > > > > >
>>> > > > > > > >Chak Chau
>>> > > > > > > >FUJITSU, Transmission Development
>>> > > > > > > >Phone: (972) 479-2795
>>> > > > > > > >chak.chau@xxxxxxxxxxxxxxx
>>> > > > > > > >chakavuth.chau@xxxxxxxxxxxx
>>> > > > > > > >-----Original Message-----
>>> > > > > > > >From: David Martin [mailto:dwmartin@xxxxxxxxxxxxxxxxxx]
>>> > > > > > > >Sent: Monday, February 10, 2003 2:59 PM
>>> > > > > > > >To: stds-802-3-efm@ieee.org
>>> > > > > > > >Subject: RE: [EFM] Question regarding OAM in 802.3ah D1.3
>>> > > > > > > >
>>> > > > > > > >Shahram,
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >Agreed, the EoS (e.g. GFP-F) mapping is a simple
>>> > > > > > > port-to-port mapping and
>>> > > > > > > >doesn't include the full MAC sublayer processing (i.e. only
>>> > > > > > > terminates
>>> > > > > > > >IPG, preamble, SFD). Inspecting the MAC DA and filtering off
>>> > > > > > > EFM OAMPDUs
>>> > > > > > > >and processing them is required by the network application,
>>> > > > > > > since the
>>> > > > > > > >Ethernet link / PHY to which they apply is terminated. OAM
>>> > > > > > > for link 1
>>> > > > > > > >cannot be mixed with OAM for link 2 on the other side of the
>>> > > > > > > provider's
>>> > > > > > > >network.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >ONU1 -------------- OLT1/GFP------------------ GFP/OLT2
>>> > > > > > > ------------ ONU2
>>> > > > > > > >           Ethernet                      SONET
>>> > > > > > >       Ethernet
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >It might be more appropriate to continue this privately, or
>>> > > > > > > on the Q.12/15
>>> > > > > > > >reflector.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >...Dave
>>> > > > > > > >David W. Martin
>>> > > > > > > >Nortel Networks
>>> > > > > > > ><mailto:dwmartin@xxxxxxxx>dwmartin@xxxxxxxx
>>> > > > > > > >+1 613 765-2901 (esn 395)
>>> > > > > > > >~~~~~~~~~~~~~~~~~~~~
>>> > > > > > > >-----Original Message-----
>>> > > > > > > >From: Shahram Davari [mailto:Shahram_Davari@xxxxxxxxxxxxxx]
>>> > > > > > > >Sent: Monday, February 10, 2003 3:16 PM
>>> > > > > > > >To: Martin, David [SKY:QW10:EXCH]; stds-802-3-efm@ieee.org
>>> > > > > > > >Subject: RE: [EFM] Question regarding OAM in 802.3ah D1.3
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >David,
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >I like to agree with you, but from layering architectural
>>> > > > > > > point of view,
>>> > > > > > > >the EOS box does not have to implement MAC
>>> > > > > > > >layer (i.e., do any MAC lookup), rather a P-2-P EOS is a
>>> > > > > > > kind of port
>>> > > > > > > >transport in which all traffic coming form an Ethernet port
>>> > > > > > > are send over
>>> > > > > > > >a specific SONET channel.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >Please see further comments in-line:
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >Thanks,
>>> > > > > > > >-Shahram
>>> > > > > > > >-----Original Message-----
>>> > > > > > > >From: David Martin [mailto:dwmartin@xxxxxxxxxxxxxxxxxx]
>>> > > > > > > >Sent: Monday, February 10, 2003 3:00 PM
>>> > > > > > > >To: stds-802-3-efm@ieee.org
>>> > > > > > > >Subject: RE: [EFM] Question regarding OAM in 802.3ah D1.3
>>> > > > > > > >Shahram,
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >This question is somewhat out of scope wrt EFM, but the
>>> > > > > > > answer is yes, the
>>> > > > > > > >EFM OAMPDU flow must be terminated at the Provider Edge.
>>> > > > > > > Otherwise it
>>> > > > > > > >would flow through the provider's SONET network and get
>>> > > > > > > mixed in with a
>>> > > > > > > >separate EFM OAMPDU flow at the far end.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >SD=> Which separate OAMPDU flow? do you mean from ONU2 to 
>>> OLT2?
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >  Note that you have the terms ONU / OLT reversed.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >So the ONU is the customer side?
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >This "filter function" is being defined in the ITU-T SG15 /
>>> > > > > > > Q.12 work in
>>> > > > > > > >draft G.ethna (was G.etna) and in the OIF UNI v2.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >Thanks I will have a look.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >The PE-PE, or SONET portion is not an EFM link, but rather a
>>> > > > > > > SONET path,
>>> > > > > > > >which has its own OAM (i.e. POH).
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >Agree.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >...Dave
>>> > > > > > > >David W. Martin
>>> > > > > > > >Nortel Networks
>>> > > > > > > ><mailto:dwmartin@xxxxxxxx>dwmartin@xxxxxxxx
>>> > > > > > > >+1 613 765-2901 (esn 395)
>>> > > > > > > >~~~~~~~~~~~~~~~~~~~~
>>> > > > > > > >-----Original Message-----
>>> > > > > > > >From: Shahram Davari [mailto:Shahram_Davari@xxxxxxxxxxxxxx]
>>> > > > > > > >Sent: Monday, February 10, 2003 2:13 PM
>>> > > > > > > >To: stds-802-3-efm@ieee.org
>>> > > > > > > >Subject: [EFM] Question regarding OAM in 802.3ah D1.3
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >Hi,
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >802.3ah section 57 says that the OAM defined is for single 
>>> link (or
>>> > > > > > > >emulated link), and should not be forwarded by 
>>> bridges/switches.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >My question is, in case of Ethernet over SONET transport
>>> > > > > > > (GFP + Virtual
>>> > > > > > > >concatenation), should the OAMPDU be terminated at the EOS
>>> > > > > > > device or it
>>> > > > > > > >should be transparently transported?
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >Consider this example:
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >OLT1 -------------- ONU 1------------------ ONU 2 
>>> ------------ OLT 2
>>> > > > > > > >           Ethernet               SONET Ethernet
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >Assume OLT1 and OLT2 are the customer equipments and ONU1
>>> > > > > > > and ONU2 are
>>> > > > > > > >provider
>>> > > > > > > >transport equipments that transport Ethernet over SONET (but
>>> > > > > > > don't do any
>>> > > > > > > >switching/bridging).
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >In this case should ONU1 terminate OAMPDUs from OLT1 or it
>>> > > > > > > should sent
>>> > > > > > > >them transparently to OLT2?
>>> > > > > > > >In other words is OLT1--ONU1 considered a single link? what
>>> > > > > > > about ONU1 to
>>> > > > > > > >ONU2, is this also a link?
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >Thanks in advance,
>>> > > > > > > >Shahram Davari
>>> > > > > > > >Senior Product Research Engineer
>>> > > > > > > >R&D Research Ottawa
>>> > > > > > > >PMC-Sierra, Inc.
>>> > > > > > > >Phone: (613) 271-4018
>>> > > > > > > >Fax:   (613) 271-6468
>>> > > > > > > >
>>> > > > > > >
>>> >
>>> > --
>>> > -----------------------------------------
>>> > Benjamin Brown
>>> > AMCC
>>> > 200 Minuteman Rd
>>> > 3rd Floor
>>> > Andover, MA 01810
>>> > 978-247-8022 - Work
>>> > 603-491-0296 - Cell
>>> > 978-247-0024 - Fax
>>> > 603-798-4115 - Home Office/Fax
>>> > bbrown@xxxxxxxx
>>> > -----------------------------------------
>>
>