THE TERASTREAM IPv6 NATIVE NETWORK ARCHITECTURE

(1)

THE TERASTREAM IPv6 NATIVE NETWORK ARCHITECTURE

How to build a modern service provider using IP v 6 and Optics

Ian Farrer, Deutsche Telekom AG

(2)

12-Sep-2012

One truly converged network - de-layered, IP and Optical are one, bits over wavelengths, digital over analog

The same technology for LAN and WAN - for LAN: IP packets in Home, Office, Data Center; for WAN: IP packets in Metro, Country, Continent

Digital services for consumers and businesses - communication, information, cloud-compute and -storage

Real-time OSS - instantaneous service provisioning, guaranteed good user experience

Cloud-era economics - service flexibility, fast-paced innovation, agile implementation, reduced system complexity, lower cost

Stay with the IP Internet Architecture using IPv6 for all functions and services Have a standardized toolbox, “services in the data center”

Standard computers instead of specialized appliances

Look forward, what can we do, not backwards what we used to do No L1, L2, L3 dependencies

No “best effort” you get what you pay for (Best effort is just a service class..)

TeraStream

Packet Cloud Architecture Fundamentals

(3)

12-Sep-2012

Principle Applied to TeraStream design

Reduce the amount of technologies used Use IP and optical transmission only No OTN, L2, MPLS switching

Use IPv6 for all internal functions and

services No native IPv4 support in the network

IPv4 is a service

IPv6 based “carrier Ethernet service”

Avoid internal interfaces Minimize non-customer, non-peering facing interfaces Distribute Internet peerings, offload external traffic ASAP

Size the network to handle all IP traffic

without IP packets losses Dimension the network for peak hour IP traffic, no oversubscription, packet loss is extreme exception

Integrate optical networks and IP networks as

much as possible Integrate IP and optical layers into routers to simplify the network, avoid redundant mechanisms e.g. failure handling, reduce total cost

Use one network for all services – Internet, IP

TV, business, … Single converged packet network

Note: Dominant traffic drives the design!

Deterministic and short routing path for all on-

net traffic Network distance between R1 access routers is at most two R2 backbone

routers away and R1 is multi homed to two R2

Service policy for packets are outside the

payload Encode service type, traffic class, direction etc in the IPv6 address

Data Centers are directly connected to

backbone routers DCs connect directly to R2s to avoid building internal IP interfaces for very large amount of traffic

TeraStream Design Principles

(4)

THE TERASTREAM ARCHITECTURE

L2:MSAN

OLT Mobile FTTx

xDSL

DWDM

L2 Agg L2 Switch L2 Agg

R1 R1

R2 R2

intern

R1 R1

Infrastructure Cloud Infrastructure Cloud

Node B

R1 R1

R2 R2

(5)

12-Sep-2012

R1

! Terminate access interfaces

! Runs IPv6 routing only, integrates optical

! Access services

! IPv6 - dealt with natively

! IPv4 – IPv4 over IPv6 softwire between HGW / CPE and DC, R1 not involved

! non-IP - L2-over-IPv6 encapsulation

! User configuration

! using Netconf / Yang

! Driven by real-time OSS i.e. self-service portal

All Access

MSAN OLT

LTE

Data Center User

Area

DC

R1 R2

IPv6 Packet Network

IPv6 + Optical IPv6

routing

4o6-swire DC

Peering IPv6 / IPv4 + Optical L2-over-IPv6 Tunnel

IPv6 4o6-swire Non-IP

DC

L2oIPv6 encap

R2

•  Connects R1s, Data Centers and Internet peerings

•  Runs IPv6 and IPv4 routing, integrates optical

•  Closely integrated with Data Centers

Optimized handling of locally sourced services

•  High scale IP bandwidth

Data Center / Services

•  Distributed design

fully virtualized x86 compute and storage environment

•  Network support functions - DNS, DHCP, NMS

•  Real-time OSS incl. user self-service portal

•  Cloud DC applications, XaaS services

•  Complex network services e.g. high- touch subscriber handling

TeraStream key functional elements

Non IP services

TeraStream – Design in a Nutshell

(6)

TRANSPORT

(7)

10-SEP-2013

R1 " R2: OPTICAL FIBER LINKS

R1 ! R2 optical fiber links

R2a/b R2b/a

R1 R1 R1 R1 R1 R1

80/96 W DM channels

R2b/a

R1 R1 R1 R1 R1 R1 R1 R1 R1 R1

R1 R1 R1 R1 R1 R1

R1 R1 R1 R1 R1 R1 R1 R1 R1 R1 R1 R1 R1 R1

R1 R1 R1 R1 R1 R1 R1

R1 R1 R1 R1 R1 R1 R1 R1 R1

R1 R1

(8)

100G Coherent DWDM Interop and Pluggable Technology

OEO OEO

Vendor A Vendor B Vendor C

Up to 600Km

Optical

Components DSP

FEC

&

Framer

Optical IF Electrical Analog IF

Device IF e.g. CAUI&CEI

System IF e.g. CAUI/CEI

•  Agree on a common set of

parameters for the 100G line side

•  Enable innovation by many players in the silicon optics arena

•  Hard FEC, typ 800km

•  If price is right, use in data center

•  Coding

•  Carrier Recovery

•  Acquisition (blind)

•  Reach

•  Framing (works with both OTU4.4 and OTU4.10)

•  Forward Error Correction

(Hard FEC Staircase)

(9)

12-Sep-2012

TeraStream user facing router R1

L2 device:

MSAN, OLT, Eth Swtch

Router

10 GE

(250VLANs)

Max 250 cust. per 10GbE link

•IP traffic shaped to capabilities of L2 device

•5000 customers connections per R1

•20 * 10GE port for L2 device

•4 * 100GE for R2 link R1

per customer queues: 100GE

100GE

100GE 100GE

R2-B R2-A

best- effort audio video reserved

traffic shaping _{per 100GE:}

TeraStream user facing router R1

Service Guarantee Architecture Configure per queue:

- Delay - Drop

- Bandwidth

- Reorder

- Etc...

(10)

TeraStream Cloud Service Center 10

Traffic flow patterns:

•  R1 " Peers and Other R2 going north " south (example: IPv6 Internet traffic)

•  R1 " Data Center services going south " east (example: DHCP)

•  R1 " Data Center " Peers going south " east " west " north (example: IPv4 Internet traffic)

DNS (local)

R2

Data Center

IMS / SBC DHCP

Lw4o6 (IPv6oIPv6) CDN

DNS (global)

Peering

Other R2 full Mesh

R1

north

south

west east

DC Internal East-West L3

L2 SDN Overlay ? Security Domains:

MZ, DMZ, Exposed Hosts separated, firewalled

R2 router and traffic patterns

(11)

21-MAR-2013

TERASTREAM HOMEGATEWAY PLATFORM

Existing CPE procurement models are far too slow to keep up with the necessary rate of development

The current state of IPv6 support on commercial platforms leaves … room for improvement!

We believe that we are not the only carrier with this problem

DT are currently developing a carrier grade’ CPE software platform Based on OpenWRT, extended with IPv6 and SP functionality

Plan to register this with Linux Foundation A fully open source initiative

We are looking for other interested carriers to build a development community around the platform

So . . .

(12)

IPv6

(13)

21-MAR-2013

IPv6 NATIVE NETWORKING

IPv6 is fundamental, not an Afterthought

A new network - designed from the ground up specifically for IPv6

IPv6 for ALL internal interfaces (incl. management, control plane etc) – with just a bit of IS-IS for IGP

Treat IPv6 as a new protocol – It can be much more than just ‘IPv4, but bigger’

Don’t let native IPv4 into the network No Dual-Stack

Treat IPv4 as a long-tail overlay service – (more on this later)

Once you let ANY IPv4 into the design, you’ll never get rid of it!

(14)

21-MAR-2013

SERVICE DIFFERENTIATION BASED ON ADDRESSES USING IPv6 ADDRESS SPACE AS LABELS

Provider (56 bits)

Subnet (8) ^User

User – Host (64 bits)

Network Structure bits Servicebits

Registry/IANA Assigned

P Public 0=SP-intern, 1=extern

I Infrastructure 0=end user, 1=infrastructure packet E Endpoint/Service 0=endpoint, 1=service

SSS Service Type 0=res, 1=internet, 4=video, 5=L2, 6=voice, 7=mgmt

M 0=fixed, 1=mobile endpoint

Examples: Source Destination

PIESSS PIESSS

---

User -> IMS 000110 011110

IMS -> User 011110 000110

User -> User (best effort) X00001 X00001 User -> Internet (best effort)100001 XXXXXX Internet -> User (best effort)XXXXXX 100001

Lan-Lan service 010101 010101

(15)

21-MAR-2013

MULTIPLE PREFIXES ON THE CLIENT…

Brings benefits, but also a new (old!) problem

The Solution? - “Prefix Colouring”

!  Adds additional metadata to DHCP prefix allocations

!  Allows applications and users to select a source prefix based on this metadata

!  Source address selection is decoupled from the destination address prefix matching

!  Can also help with source address selection in multi-homing deployments

!  Described in ‘draft-lepape-6man-prefix-metadata’

Current source address mechanisms are based on variants of longest source/destination prefix matching policy

!  This places constraints on your addressing architecture

!  Often require additional policy to be provisioned to the client

!  Doesn’t give users or applications information about the ‘properties’ or ‘suitability’ of a prefix for use How do you ensure that the client selects the right source address for each different service?

(16)

Home network with video class service

In this example, two different services are being run on the same network.

The service provider wishes that traffic is sourced from different prefixes by the home network clients for Video on demand service as against general Internet access

The homenet has several prefixes delegated – (potentially one each for voice, video and Internet)

This example show

Set Top box

Video Service Prefix Class: Internet (e.g. 0)

http://datatracker.ietf.org/doc/draft-lepape-6man-prefix-metadata/

Prefix Class: Video(e.g. 10)

(17)

21-MAR-2013

IPv4 AS A SERVICE – LIGHTWEIGHT 4o6 SOFTWIRES

21-MAR-2013

R1 R2

Home Network

v4 CPE

host v4

Internet v6

Lightweight 4o6 Softwire Tunnel

Concentrator (lwAFTR)

Infrastructure Cloud

v4

IPv4 in IPv6 Softwire Tunnel

(18)

Customer connection usage example

L2 device:

MSAN, OLT, EthSwtch R1

Scenario: New customer connects to DT IP Infrastructure

1) Customer connects his device, generates IPv6 packet and forwards them to R1

2) R1 receives the IP packet and recog nizes a VLAN as not “in serv ice”

3) IP packet is encapsulated at R1, wit h the IPv6 informing the VLA N / L2 d ev ice from where the request came from, and forwards it to Data Center

4) Data center respond s w ith DHCP or Web server.

Customer can only reach the walled garden.

Router

R2

VLAN per customer

10GbE

Server (at R2) LocalIPv6

IP pa cket IP packet

Data Center

NetConf/

Yang

Max 250 cust. per 10GbE link

inservice customer MAC

Scenario: customer registers

1) Web server at Data Center generates a request to OSS to configure a new customer via NetConf / Yang at router R1, Line ID.

2) The OSS via NetConf configures the R1 as “in service” for a customer located at a specific interface (IPv6 address).

3) From now on, the customer is outside the walled garden and can reach other Internet addresses.

OSS

- DHCP - DNS

Initial tunnel

Internet access

- WEB srv.

If Not IPv6, use the network as a PTP Ethernet

(19)

21-MAR-2013

DEPLOYING v6 ONLY – SOME LESSONS LEARNT SO FAR…

!  There are still mainstream products that do not have complete IPv6 implementations:

!  Transport interfaces tend to have more complete implementations

!  Management and control plane functionality may not be so good

!  The level of IPv6 testing in shipping products is not on a par with IPv4 – we’ve found some pretty hairy bugs!

!  Vendor’s need constant ‘encouragement’ to resolve these problems

!  If you are planning any kind of similar rollout, get your requirements fixed

and test well in advance!

(20)

NETWORK FUNCTION VIRTUALISATION

“THE INFASTRUCTURE

CLOUD”

(21)

21-MAR-2013

INFRASTRUCTURE CLOUD

NETWORK FUNCTION VIRTUALIZATION

R2

40% of traffic

OTT Apps Self Provisioning

IPv4 Softwire

Network Services (DNS, DHCP)

Video

Business VPN Services

Content

Mobile Core

& Services IMS

(22)

12-Sep-2012

REAL-TIME OSS

(23)

12-Sep-2012

NCS – The Model-Driven Network Abstraction Layer

NCS

Network Engineer Other OSS

Functions

YANG Service Models

YANG Device Models

Two way formal service

mappings •  Real-time

•  Read-write

•  Two phase transactions

•  Validations, rollbacks NETCONF, IOS(-XR) CLI, REST, etc

Network CLI, Web UI NETCONF, REST

Internet₁ Internet₂ VPN

R1₁ R1₂

API

Model-Driven Network Abstraction Layer

(24)

12-Sep-2012

NCS in Terastream

Confidential Information | September 29, 2013

R1 R1 R1

R2 R2

N C S

NCS CLI : EVPL Service over R1 and R2

NCS REST

NCS NETCONF NCS WebUI

Transaction

Define “Services” in a data-model language (Yang)

(25)

12-Sep-2012

TERASTREAM

CROATIA TRIAL

(26)

100 Gb/s network using IP and Optical integration

Full integration of Network and Cloud technologies for

service production

Native IPv6 network delivering consumer service

Built in a record time - decision in September, launch on Dec 10th 500 customers with up to Gigabit access speeds

TERASTREAM PILOT SUCCESFULLY LAUNCHED

@ HRVATSKI TELEKOM ON DEC 10, 2012

Agile execution – small cross-functional teams (DT, HT, Cisco, Combis)

Continued development and iterative improvement Technology refinements

New vendors being integrated More customers connected

(27)

12-Sep-2012

(28)

10-SEP-2013

THE TERASTREAM IPv6 NATIVE NETWORK ARCHITECTURE