Re: Does Ten-Gigabit Ethernet need fault tolerance?
Shawn,
Actually, it is a lot simpler at the MAC. All of the fault/TCE (threshold
crossing event) sensing should be done in PHY, beneath the GMII.
For implementations that fault tolerance is desired, a PHY chip set can be used
that will provide the required protection. This would be used primarily in
extended LAN and MAN/WAN implementations.
For implementations that do not need fault tolerance, a non-protected PHY can be
used. This would work for server farms and other very localized installations.
Thank you,
Roy Bynum
MCI WorldCom
"Rogers, Shawn" wrote:
> Okay, I see now. I've seen this referred to as redundancy or redundant
> links. At the PHY you're adding a Tx differential pair, Rx differential
> pair, two power/ground pairs and a mux control pin for every port (9 pins
> for serial/ 36 pins for WWDM). The MAC or higher layer function controls
> the MUX pin sense. Pretty simple but it does add pins and power to the
> transceiver. There has been some interest shown in adding this to 1GbE
> transceivers.
>
> I'm not knowledgeable about what complexity it adds to MAC nor am I
> knowledgeable about the response time from Loss of Signal (LOS) to Link up
> after MUX toggling, Rx PLL re-lock. Both of these are very important to
> providing a viable solution.
>
> This does pose some questions for WWDM solutions similar to those raised in
> Austin:
> What if one of the four lambdas goes down? Do you have redundancy for
> lambda's or for the entire port? My guess is for the entire port, meaning
> that if one lambda were lost, then the entire port would be switched to the
> redundant link.
>
> Shawn
>
> -----Original Message-----
> From: Roy Bynum [mailto:rabynum@xxxxxxxxxxx]
> Sent: Monday, July 19, 1999 6:59 AM
> To: Rogers, Shawn
> Cc: stds-802-3-hssg@xxxxxxxx
> Subject: Re: Does Ten-Gigabit Ethernet need fault tolerance?
>
> Shawn,
>
> I use the term "L1 restoration link services" to refer to the generic
> process by
> which the L1 processes of SONET/SDH does "protection" switching. SONET/SDH
> do
> this without requiring instruction from upper layers. I will attempt to
> give a
> simple explanation of the service process of one of the architectural
> implementations.
>
> SONET/SDH monitors the operable status of alternate links, "working" and
> "protection". SONET/SDH "sends" traffic over both links, but only receives
> it
> on the "working" link. SONET/SDH monitors for L0/L1 fault and other
> threshold
> crossing "events". When SONET/SDH detects an "event" it moves the reception
> of
> traffic from the "working" to the "protection" link. SONET/SDH then sends
> an
> alarm message to upper layer monitoring applications.
>
> With the exception of the upper layer monitoring applications, most of this
> is
> done in the SONET/SDH chip set. Because it is done directly in the chip
> set, at
> L1, it is very fast. This is as close to true "fault tolerance" as can be
> achieved by a communications process. Not upper layer processing is
> required by
> SONET/SDH. The speed that SONET/SDH does protection switching can not be
> duplicated by upper layer applications. I do not consider the fault
> restoration
> processes of upper layer protocols to be "fault tolerant". By definition
> and
> comparison, they can not be. At best, upper layer processes are "fault
> acceptable".
>
> When we talk about fault tolerance for 802.3, we need to think of it as
> operating at the PHY level. In many ways, fibers/wavelengths are "cheap".
> Inside buildings and in WAN/MAN areas of the country they will be becoming
> even
> "cheaper". Using the chip set to implement the process makes L1 restoration
> link services "cheap" and simple to implement.
>
> Thank you,
> Roy Bynum
> MCI WorldCom
>
> "Rogers, Shawn" wrote:
>
> > Roy, sorry but I blinked and missed a tread. What is L1 restoration link
> > services?
> > Shawn
> >
> > -----Original Message-----
> > From: Roy Bynum [mailto:rabynum@xxxxxxxxxxx]
> > Sent: Sunday, July 18, 1999 6:02 PM
> > To: Joe Gwinn
> > Cc: stds-802-3-hssg@xxxxxxxx
> > Subject: Re: Does Ten-Gigabit Ethernet need fault tolerance?
> >
> > Joe,
> >
> > I have a question? Your RTFC flooding algorithm sounds a lot like the
> > restoration
> > algorithms that have been proposed for DXC traffic restoration in the
> > telephony
> > industry. These have not been used because of the complexity of
> constraints
> > introduced in designing the architecture. This might work in very simple
> > systems,
> > but I have reservations about successful implementations in large, high
> > bandwidth,
> > enterprise networks.
> >
> > In doing economic modeling, it was discovered that using simple L1
> > working/protect
> > restoration link services actually cost less to implement in large complex
> > architectures. Nodal failure is more often better handled at higher
> layers.
> >
> > In addition, simple L1 restoration implementations are easier to maintain
> > for
> > not-so-technical people. Part of the reason for the success of 802.3 is
> the
> > simplicity of maintenance. The success of 10GbE will be dependent on a
> > continuing
> > of that simplicity.
> >
> > Thank you,
> > Roy Bynum
> > MCI WorldCom
> >
> > Joe Gwinn wrote:
> >
> > > Jonathan,
> > >
> > > At 2:22 PM 99/7/16, Jonathan Thatcher wrote:
> > > >
> > > >A question and a suggestion:
> > > >
> > > >1. Are you suggesting that Fault Tolerence is a requirement for 10 Gig
> > > >Ethernet or for all Ethernet? Or, if FT is added to 10Gig Ethernet, is
> > it of
> > > >any particular value if 10, 100, and 1000 BASE-* don't have it?
> > >
> > > I am suggesting fault tolerance as an optional enhancement for 10GbE
> only,
> > > mainly because it's early enough in 10GbE's standards development
> timeline
> > > that FT could be included without pain, if the committee so desires.
> > >
> > > Another reason is that I would like to be able to buy FT/DT 10GbE
> products
> > > a few years from now, for use in military systems. If you recall from
> the
> > > London GbE meeting, I intended to suggest this FT technology to GbE, but
> > > the technology couldn't be released in time, and so missed the GbE
> > > standards train.
> > >
> > > As for the other ethernet standards, I propose nothing, although there
> is
> > > no reason that they could not also take advantage of the offered FT
> > > technology, should they so desire.
> > >
> > > The RTFC technology allows some segments of an overall network to be FT,
> > > and does not require all to be FT, so there is no reason for an
> > all-or-none
> > > approach. In a network containing multiple FT segments, the segments
> > react
> > > to changes and roster independently of one another.
> > >
> > > >A1: If all Ethernet: you should ask for a call for interest in 802.3
> and
> > > >bring presentations supporting the requirement (5 criteria, etc).
> > > >
> > > >A2: If only 10 Gig Ethernet: you should bring a presentation supporting
> > the
> > > >requirment to the next 802.3 HSSG meeting. Expect questions about how
> > this
> > > >supports the 5 criteria. Expect questions about why only 10 Gig
> Ethernet.
> > >
> > > The famous 5 criteria, lifted from slide 15 of thatcher_1_0399.pdf:
> > >
> > > 3.4.1. Broad Market Potential -- FT is already in ATM/SONET, 802.3ad,
> > > Rapid Reconfiguration in 802.1, etc, so there seems to be preexisting
> wide
> > > agreement that fault tolerance is desirable and has a sufficiently broad
> > > market potential.
> > >
> > > 3.4.2. Compatibility with IEEE Standard 802.3 -- Based on my experience
> > > with 802.3z, I believe the offered technology is compatible, but the
> > > committee is the expert here.
> > >
> > > Some facts: The current RTFC implementations use standard TriQuint
> > > Fibre-Channel parts and Finisar optical transceivers for the gigabit
> > links,
> > > plus some code in a standard-issue FPGA. Only Fibre Channel layers FC-0
> > > and part of FC-1 are used, just as GbE does (although the details of use
> > of
> > > FC-1 differs). The network segments (containing NICs, hubs, and fibers)
> > > are either in "data mode" (with normal lan traffic), or in "rostering
> > mode"
> > > (where the new roster of NICs, hubs, and fibers are configuring
> themselves
> > > into a working segment), and the protocols used in those two modes are
> > > wholly independent of one another. This is detailed in RTFC Principles
> of
> > > Operation.
> > >
> > > 3.4.3. Distinct Identity -- No problem. No other fault and damage
> > > tolerance algorithm works this way, and thus confers unique advantages.
> > > For one, the technology is noticably simpler than all other FT
> > technologies
> > > I am aware of, and is a whole lot more robust (in that it also supports
> > > DT).
> > >
> > > Perhaps the key difference between this and other message-based fault
> > > distributed tolerance schemes is that all other schemes attempted to be
> > > stingy with mesages (because they are expensive in most distributed
> > > systems), while RTFC is a flooding protocol with just enough population
> > > control to prevent network saturation. The use of flooding allowed a
> > > radical simplification of the algorithm, and the implementation of true
> > > damage tolerance rather than just fault tolerance.
> > >
> > > 3.4.4. Technical Feasibility -- It has been implemented, and is in use
> in
> > a
> > > military application, with others under consideration.
> > >
> > > 3.4.5. Economic Feasibility -- It has been implemented, and the
> algorithm
> > > is quite simple, as detailed in RTFC Principles of Operation. We are
> > > basically talking about making a gate array slightly larger in those
> hubs
> > > supporting fault tolerance (for which one can charge extra).
> > >
> > > I guess the only requirement, in the sense that all of 10GbE would have
> to
> > > follow it, is for the NICs to do their part in rostering, a simple task
> > > easily buried in the NIC's state machines. The rest is for an optional
> > > variety of hub where one does the rest of the rostering algorithm.
> > >
> > > If by "requirment" you mean only a one-liner like "10GbE shall support
> > > Fault Tolerance", it wouldn't be much of a presentation. I doubt that
> > > anyone will argue that fault tolerance is undesirable; their question
> will
> > > be "At what price?". I claim the price is small, and the payoff large.
> > In
> > > the final analysis, the matter will turn on how hard it is to implement
> > the
> > > algorithm, a matter of details.
> > >
> > > I don't know that I will be able to attend many meetings, so I won't be
> a
> > > very active proponent of my own technology. As I said before, no
> salesman
> > > will call. But email is another matter.
> > >
> > > More to the point, a few brave souls will no doubt read the RTFC
> > Principles
> > > of Operation, and if they think that there is something there that 10GbE
> > > either wants or needs, and the rest of the committee comes to agree, the
> > > technology will find its way into GbE. Otherwise, it won't. How else
> > > could it be?
> > >
> > > Basically, this technology is a gift, yours if you wish it. I feel it
> is
> > > of great value to 10GbE, and will be very interested to know what people
> > > think after they have had time to absorb the core of the technology, and
> > to
> > > see the implications.
> > >
> > > Joe
> > >
> > > PS: I'll be on travel, to an unrelated standards meeting, the week
> 19-23
> > > July 1999.
> > >
> > > >> -----Original Message-----
> > > >> From: gwinn@xxxxxxxxxx [mailto:gwinn@xxxxxxxxxx]
> > > >> Sent: Friday, July 16, 1999 2:15 PM
> > > >> To: stds-802-3-hssg@xxxxxxxx
> > > >> Subject: Does Ten-Gigabit Ethernet need fault tolerance?
> > > >>
> > > >>
> > > >>
> > > >> The purpose of this note is to present a case for inclusion of fault
> > > >> tolerance in 10GbE, and to offer a suitable proven technology for
> > > >> consideration. However, no salesman will call.
> > > >>
> > > [snip]
> > > >The basic technical document, the RTFC Principles of Operation, is on
> the
> > > >GbE website as "http://grouper.ieee.org/ groups/802/3/
> 10G_study/public/
> > > >email_attach/ gwinn_1_0699.pdf" and "http://grouper.ieee.org/
> > > >groups/802/3/10G_study/ public/ email_attach/ gwinn_2_0699.pdf".