Re: On two clock issue
Roy,
I agree that not even SONET/SDH is truly synchronous in the mechanical gear
train sense. As I understand it,
1) Data is transmitted with placeholders for padding flags and padding at a
prescribed timing accuracy. This accuracy is such that
2) The receiver, also using its own local time base of prescribed accuracy
can, once it get synchronized, find these timing marks "by Braille", i.e.,
by just counting off time cycles. In a point to point (terminating) link, I
guess the principal use of this is to ignore them. If passing data through,
these must be preserved or modified.
3) More importantly, "3R" regenerators and add/drop multiplexers can modify
the padding flags and amounts to keep the system in long-term synch over
multiple hops without having to fiddle with the actual data content in the
cells, since these padding flags and bits or bytes occur at specified (i.e.,
"known" times and, consequently, positions, in the cell) from the start of
synch attainment.
Without this cut/pad to fit capability at all levels in SDH, you have the
old problem of elasticity buffer crawl that you get with, say, back-to-back
RS-232 UARTS running off separate crystals. ("Trad" Ethernet uses the IPG to
allow for avoidance of multi-hop and speed-dependent data pileup or suckout,
and RS-232 uses the start and stop bits for the same purpose.)
The respective clocks' precision(s) must be compatible with the allowable
amount of padding, in addition to other timing disturbances.
Is my understanding incorrect?
On I Gb we allow quite a lot of time variation (typically 200 ps peak to
peak), and even somewhat more at low frequencies. (The receiver PLL can
follow low frequencies, but you have to have some limit to keep FIFO sizes
reasonable. We measure our crystal clock oscillators (of the $8.00 variety)
with a 634 kHz single pole high-pass filter. I think that Fibre Channel also
does this. The 100 PPM is a "DC" frequency specification, so the overall
speeds (baud rates) of the two directions of the link could differ by as
much as 0.1% and still be conforming.)
Anyway, what timing accuracy do you think is needed? What do you get for
specifications at the PMD level in systems you do?
Thanks,
Larry Miller
-----Original Message-----
From: Roy Bynum <rabynum@airmail.net>
To: Larry Miller <l_d_miller@email.msn.com>
Cc: HSSG <stds-802-3-hssg@ieee.org>
Date: Tuesday, July 20, 1999 4:17 PM
Subject: Re: On two clock issue
>Larry,
>
>Actually, Packet Over SONET/SDH interfaces in routers today do not have the
same
>level of synchronization as true SONET/SDH. Payload synchronization alarms
are
>turned off on the transport systems to allow this. In reality, the router
>interfaces use line timing derived from the transport system, just like any
>other WAN interface. Synchronization is achieved in the SONET/SDH
transport
>system, not the data end system. The issue of tight synchronization
>requirements is unfounded.
>
>Thank you,
>Roy Bynum
>MCI WorldCom
>
>Larry Miller wrote:
>
>> Roy,
>>
>> Ethernet is (so far) asynchronous.
>>
>> Each transmitter runs at its own speed and phase (within the allowed
>> tolerances, which are huge compared to SONET, typically +/-100 ppm), and
>> each receiver recovers its clock from the received data.
>>
>> And, Yes, they free run (each Gigabit Ethernet receiver switches over to
its
>> local transmit clock in the absence of incoming signals to minimize
receive
>> PLL acquisition time), and, Yes, the transmit and receive data rates
differ.
>> One of the main purposes of the Inter-Packet Gap (IPG is to allow for
this).
>>
>> The only exception I know of is 1000BASE-T where the timing/phase
accuracy
>> of the echo cancellation and line compensation filters (simultaneous
>> transmit/receive on same copper wires) is so tight that one end of the
link
>> has to supply clock for both directions, and a master-slave relationship
is
>> negotiated and set up for the link (but not synchronous to the cosmos--
>> still +/-100 ppm for the clock used for both directions).
>>
>> The POS line cards I have seen need considerable buffering and flow
control
>> to get into/out of the synchronous world.
>>
>> Some of you guys are trying to hammer Ethernet into SONET/SDH in a big
way.
>>
>> This interface between the synchronous and asynchronous worlds, as has
been
>> pointed out by many here, will not go away, but will simply move into
>> greater or less costly or inconvenient points in the data paths.
>>
>> Since the synchronous world requires terrifically accurate time bases
>> (single Ethernet links can tolerate very large timing differences in
>> comparison, like 10X that of SONET without even thinking about it much--
>> hey, it's A-synchronous) you should expect and will get a lot of
resistance
>> to forcing the Ethernet world into the expensive SONET performance
>> requirements and costs.
>>
>> Cheers,
>>
>> Larry Miller
>> Nortel Networks
>>
>> -----Original Message-----
>> From: Roy Bynum <rabynum@airmail.net>
>> To: rtaborek@transcendata.com <rtaborek@transcendata.com>
>> Cc: HSSG <stds-802-3-hssg@ieee.org>
>> Date: Monday, July 19, 1999 8:50 PM
>> Subject: Re: On two clock issue
>>
>> >
>> >Rich,
>> >
>> >Just a note of clarification. If the receive clock is separate from the
>> system clock,
>> >what controls the transmit clock. Does each link transmit bit rate
provide
>> the clock
>> >to the other system's receiver? Would this tend to cause free running
and
>> separate
>> >transmit and receive data rates? How do existing OC48C POS interfaces
use
>> line timing
>> >from the carrier LTE to control receive as well as transmit speeds? If
>> 802.3 were to
>> >be used on anything other than enclosed LANs, what would be available to
>> provide line
>> >timing? Are there alternatives?
>> >
>> >Thank you,
>> >Roy Bynum
>> >MCI WorldCom
>> >
>> >Rich Taborek wrote:
>> >
>> >> Devendra,
>> >>
>> >> We're taking a few too many tangents here. To bring everyone else up
to
>> speed, the
>> >> root issue is a MAC/PLS rate of 10.0 Gbps vs. 9.584640 Gbps. I
asserted
>> that a
>> >> 9.584640 Gbps would complicate the clock design of Ethernet equipment
due
>> to the
>> >> requirement to support a rate which is not a multiple of slower
Ethernet
>> MAC/PLS
>> >> local clocks.
>> >>
>> >> Receive clocks are separate from local clocks. I believe that most
>> Ethernet
>> >> equipment supporting multiple interfaces DOES NOT use the receive
clock
>> from any
>> >> interface as its local clock.
>> >>
>> >> Devendra Tripathi wrote:
>> >>
>> >> > Rich,
>> >> >
>> >> > One clarification here. The use of two clocks is almost always
there.
>> >> > Ofcourse
>> >> > some designs carry the retreived clock all the way to host fifo
>> interface
>> >> > and some
>> >> > terminate it early. I am doubtful, though, if anyone uses that clock
>> for
>> >> > transmit
>> >> > purpose. Because in that case, the reference clock (which is taken
as
>> transmit
>> >> > clock by PHYs) also has to be retrieved one, thus making a loop in
the
>> PLL.
>> >> >
>> >> > Thanks,
>> >> > Tripathi.
>> >> >
>> >> > At 10:17 AM 7/19/99 -0700, you wrote:
>> >> > >
>> >> > >Roy,
>> >> > >
>> >> > >This one's pretty simple. The MAC/PLS clock for 10 Mbps Ethernet is
>> based
>> >> > on the
>> >> > >transport of bits. At 100 Mbps, it's nibbles (4 bits). At 1 Gbps,
it's
>> >> > octets. At 10
>> >> > >Gbps, several proposals suggest 8, 16 and 32 bit chunks. The local
>> clock
>> >> > for Ethernet
>> >> > >equipment which performs speed matching is easily derived from
MAC/PLS
>> >> > clock of the
>> >> > >slowest interface supported. Deriving a 9.584640 is not so
>> >> > straightforward. Reverse
>> >> > >deriving lower speed clocks is also not so straightforward. The
>> simplest
>> >> > solution is
>> >> > >to use two clocks. This solution increases implementation cost and
>> will
>> >> > significantly
>> >> > >complicate clocking design.
>> >> > >
>> >> > >Best Regards,
>> >> > >Rich
>> >> > >
>> >> > >--
>> >> > >
>> >> > >Roy Bynum wrote:
>> >> > >
>> >> > >> Rich,
>> >> > >>
>> >> > >> I hear much about the single system clock issue. In the past,
the
>> data
>> >> > transport
>> >> > >> signal clock and the data system bus and processing clock were
>> separate.
>> >> > For many
>> >> > >> systems the data processing would exceed the transport signal in
>> order
>> >> > to maintain
>> >> > >> control. If that is the case, then your argument about a single
>> clock
>> >> > does not
>> >> > >> hold. I could be wrong. Would one of the system implementation
>> people
>> >> > please
>> >> > >> define how a data signal clock is derived in today's systems. Is
>> >> > internal 802.3
>> >> > >> frame processing done at a higher rate than the transport signal
>> rates
>> >> > in today's
>> >> > >> GbE L2 switches? Is there a real issue with a separate transport
>> signal
>> >> > rate or
>> >> > >> even a separate transport data rate?
>> >> > >>
>> >> > >> Thank you,
>> >> > >> Roy Bynum
>> >> > >> MCI WorldCom
>> >> > >>
>> >> > >> Rich Taborek wrote:
>> >> > >>
>> >> > >> > Hon Wah,
>> >> > >> >
>> >> > >> > Thanks for kicking this issue off again!
>> >> > >> >
>> >> > >> > Hon Wah Chin wrote:
>> >> > >> >
>> >> > >> > > Perhaps a difficult number to remember, but with the +-
100ppm
>> >> > tolerance
>> >> > >> > > and a bit rate that needs only to fit within about 200ppm of
the
>> >> > nominal
>> >> > >> > > SONET number we should be able to choose a round number with
4
>> >> > digits in it.
>> >> > >> > >
>> >> > >> > > ---
>> >> > >> > >
>> >> > >> > > As I understand the presentations in Montreal on speed,
>> >> > >> > > a strong advantage of choosing this OC-192 payload rate is
>> >> > >> > > to transport the signal over SONET OC-192 equipment. This
would
>> >> > >> > > be from a "10Gb/s Ethernet" port out to SONET gear, which is
>> really
>> >> > >> > > a PMD external interface rather than a definition for the
>> MAC/PLS
>> >> > interface
>> >> > >> > > and data rate.
>> >> > >> >
>> >> > >> > In my conversations with several folks on both sides of the
issue
>> >> > during the
>> >> > >> > Montreal meetings, I've come to the conclusion that the root
>> reasons
>> >> > to select
>> >> > >> > either a 10 or 9.584640 Gbps are purely ease-of implementation
>> based
>> >> > and have no
>> >> > >> > architectural basis whatsoever. I believe this to be true on
both
>> >> > sides of the
>> >> > >> > argument with the choice of one over the other, rendering the
>> >> > implementation
>> >> > >> > (i.e. product cost) of the losing side only slightly more
>> difficult.
>> >> > Please
>> >> > >> > allow me to explain the basis of this contention:
>> >> > >> >
>> >> > >> > 1) SONET, and specifically synchronous transport, is legacy in
the
>> MAN
>> >> > and WAN,
>> >> > >> > will never be replaced by Ethernet completely or even quickly.
>> >> > Ethernet will
>> >> > >> > make inroads into "green-field" applications, but SONET will be
>> king
>> >> > for some
>> >> > >> > time to come;
>> >> > >> >
>> >> > >> > 2) Ethernet, and specifically packet-based transport, is legacy
in
>> the
>> >> > LAN, is
>> >> > >> > growing in its dominance in the LAN, and will likely gain
market
>> share
>> >> > in the
>> >> > >> > LAN as well as encroach on other non-traditional Ethernet
>> transports
>> >> > including
>> >> > >> > MAN, SAN, and some WAN. I don't include WAN access in WAN.
Instead
>> I
>> >> > include WAN
>> >> > >> > access in LAN or MAN;
>> >> > >> >
>> >> > >> > 3) The existing WAN infrastructure does a great job of
>> transporting
>> >> > Ethernet
>> >> > >> > packets end-to-end today. However, much protocol conversion and
>> >> > equipment to map
>> >> > >> > between packets and TDM bits exists in mapping Ethernet to the
WAN
>> at
>> >> > each end.
>> >> > >> > Considerable savings can be realized by architecting a more
>> seamless
>> >> > Ethernet to
>> >> > >> > SONET connection. This issue seems to be at the root of the 10
vs.
>> >> > 9.584640 Gbps
>> >> > >> > issue.
>> >> > >> >
>> >> > >> > 4) There seems to be no intent by either side to consider any
>> other
>> >> > changes but
>> >> > >> > speed as a HSSG objective. Therefore, Ethernet will remain a
>> simple,
>> >> > general
>> >> > >> > purpose, packet-based transport, and SONET will remain a
specific
>> purpose
>> >> > >> > (MAN/WAN), synchronous transport no matter which way the
decision
>> goes.
>> >> > >> >
>> >> > >> > 5) Consider a Ethernet to OC-192 line card (feeding a fiber or
>> >> > wavelength) in
>> >> > >> > operation. Assume that receive and transmit paths are separate
on
>> the
>> >> > SONET side
>> >> > >> > and related (i.e. full duplex) on the Ethernet side:
>> >> > >> > a) Ethernet -> SONET @ 9.584640 Gbps: The Ethernet side can
>> >> > continuously feed
>> >> > >> > the SONET link with no flow control required.
>> >> > >> > b) Ethernet -> SONET @ 10 Gbps: The Ethernet side must be
flow
>> >> > controlled to
>> >> > >> > prevent over-feeding the SONET link
>> >> > >> > c) SONET -> Ethernet @ 9.584640 or 10 Gbps: The Ethernet side
>> can
>> >> > continuously
>> >> > >> > source SONET data but will flow control or drop packets
downstream
>> >> > whenever the
>> >> > >> > network is congested.
>> >> > >> >
>> >> > >> > Therefore, the issue boils down to one of implementation of
>> existing
>> >> > Ethernet
>> >> > >> > mechanisms such as 802.3x flow control or a reasonable
facsimile
>> on
>> >> > the line
>> >> > >> > card versus complicating the implementation of all Ethernet
>> products
>> >> > which must
>> >> > >> > support a MAC/PLS rate which is not a multiple of 10. These
>> >> > implementation
>> >> > >> > difficulties include multiple clocks which may "beat" against
each
>> >> > other, not
>> >> > >> > being able to easily feed 10 slower links into one faster one,
and
>> >> > numerous
>> >> > >> > other difficulties which are best listed by Ethernet product
>> >> > implementers.
>> >> > >> >
>> >> > >> > My intention is not to make light of the problem but rather to
>> agree
>> >> > with a
>> >> > >> > solution direction along the line proposed by Dan Dove of HP at
>> the
>> >> > Montreal
>> >> > >> > meeting. I believe that Dan's general direction was to tradeoff
a
>> simple
>> >> > >> > architectural change with respect to MAC operation to enable
cost
>> >> > effective 10
>> >> > >> > Gbps to SONET implementations. I don't particularly agree with
>> resolving
>> >> > >> > implementation cost issues between two dominant legacy
protocols
>> by
>> >> > tweaking
>> >> > >> > with the underlying architecture, but I'll raise my hand in
>> support of
>> >> > this
>> >> > >> > solution to the problem.
>> >> > >> >
>> >> > >> > Such a solution would enable the implementation of a 10 Gbps
>> Ethernet
>> >> > to SONET
>> >> > >> > OC-192 line card without requiring a full MAC.
>> >> > >> >
>> >> > >> > I'll let Dan fill in the details of his proposal so I don't get
it
>> >> > wrong if it
>> >> > >> > is still applicable.
>> >> > >> >
>> >> > >> > Best Regards,
>> >> > >> > Rich
>> >> > >> >
>> >> > >> > --
>> >> > >> >
>> >> > >> > > Given a raw continuous bit stream at the PMD, some scheme for
>> >> > >> > > framing packets would be needed. 10M used a carrier, 100M
used
>> coding,
>> >> > >> > > 1000M used coding. Using coding where the PMD speed is fixed
at
>> >> > 9.58Gb/s
>> >> > >> > > would mean a further speed reduction (probably 10-20%) at the
>> MAC/PLS
>> >> > >> > > interface. The discussion at the meeting has already started
to
>> >> > consider ways
>> >> > >> > > of
>> >> > >> > > reducing the useful throughput at the MAC/PLS below the data
>> >> > clocking rate.
>> >> > >> > > An
>> >> > >> > > alternative framing scheme presented to HSSG, which has a
>> smaller
>> >> > throughput
>> >> > >> > > reduction, requires a packet length header -- a departure
from
>> >> > previous 802
>> >> > >> > > practice.
>> >> > >> > >
>> >> > >> > > In considering the advantage of leveraging SONET OC-192
>> transport
>> >> > >> > > we should also consider the issues which come up in actually
>> getting
>> >> > >> > > the hoped-for benefits. It would also be worthwhile to
>> carefully
>> >> > consider
>> >> > >> > > what volume forecasts for the OC-192 components can be
>> documented, in
>> >> > >> > > evaluating the advantage to be gained. Counting IEEE802.3
>> 10Gb/s data
>> >> > >> > > ports (however the definition works out) to get 2 million
ports
>> sounds
>> >> > >> > > good, but I found the forecast of 2,000,000 OC-192 ports in
2000
>> rather
>> >> > >> > > surprising.
>> >> > >> > >
>> >> > >> > > -hwc
>> >> > >
>> >> > >-------------------------------------------------------------
>> >> > >Richard Taborek Sr. Tel: 650 210 8800 x101 or 408 370 9233
>> >> > >Principal Architect Fax: 650 940 1898 or 408 374 3645
>> >> > >Transcendata, Inc. Email: rtaborek@transcendata.com
>> >> > >1029 Corporation Way http://www.transcendata.com
>> >> > >Palo Alto, CA 94303-4305 Alt email: rtaborek@earthlink.net
>> >> > >
>> >>
>> >> --
>> >>
>> >> Best Regards,
>> >> Rich
>> >>
>> >> -------------------------------------------------------------
>> >> Richard Taborek Sr. Tel: 650 210 8800 x101 or 408 370 9233
>> >> Principal Architect Fax: 650 940 1898 or 408 374 3645
>> >> Transcendata, Inc. Email: rtaborek@transcendata.com
>> >> 1029 Corporation Way http://www.transcendata.com
>> >> Palo Alto, CA 94303-4305 Alt email: rtaborek@earthlink.net
>> >
>> >
>