Archive for October, 2013

Sign up for ipSpace webinars and get a 20% discount

October 31, 2013 4 comments

I posted about ipSpace webinars a couple of days ago and Ivan has kindly agreed
to provide a 20% discount for the readers of this blog. Follow this link to sign up
for the yearly subscription at a 20% discount.

This is not an opportunity you want to miss.
The code is active until Wednesday, December 11, 09:30 AM (EST)

IPSPACE Webinars with Ivan Pepelnjak

October 27, 2013 1 comment


Want to learn more about data center networks? How do you scale a data center?
How do you deploy IPv6 in the real world? What are overlays and which protocols
can be used to create overlays?

If these kind of questions interest you then I recommend that you look into
getting a subscription from Ivan Pepelnjak at ipSpace.

Who is Ivan?

If you have been in the networking industry for a while you probably alread know of
Ivan. Here is a quick summary:

In my personal opinion Ivan is one of the most knowledgable persons of the networking 
community. He has a strong level of integrity and will tell things as he sees them not
drinking any of the vendors koolaid.

Why should I get a subscription?

The networking industry is moving at a rapid pace right now. It’s difficult to keep up
with all of the new solutions/vendors/protocols. Doing so may require to read through
drafts, RFCs, white papers and so on. By using the webinars you get to tap in to
the knowledge of Ivan which makes it easier to keep up with the industry.

If you have a certain area you are interested in you can buy a recording for that.
If you only have interest in IPv6 you can get the IPv6 webinars which will cost you
somewhere around 30$ to 50$ per recording.

Personally I have really enjoyed the data center webinars beacuse I didn’t have
much exposure to data centers so that helped me a lot to learn about those through
the webinars. The yearly subscription is $199 which is a good price in my opinion.
For more information go to

Why am I promoting Ivan?

This post may seem as an advertisement for Ivan which in one way it is. The reason
for writing it is that many of us struggle to keep up with all the changes in the
industry. SDN here, data center there, VXLANs, NVGRE, networking virtualization.
How does it all fit together?

I wouldn’t write this if I didn’t think that this is a really great resource to stay
current and for $199 you get access to both webinars and case studies. That’s a
fair deal in my opinion. So go to the webinars page and see if there is anything of interest for
you. I don’t think you will regret signing up. If you want to have a demo before signing up
I recommend that you check out the great webinar he did on NSX architecture.

Cisco VIRL – Let’s go viral!

October 25, 2013 3 comments

Cisco VIRL

Most of you have probably heard something about VIRL already. There were
some rumors going around and then at Cisco Live, Joel Obstfeld from Cisco did a demo of Cisco VIRL. VIRL is a tool running on Openstack and KVM according to Ivan Pepelnjak.

With VIRL it’s possible to run virtual IOS, NXOS, IOS-XE and IOS-XR. Refer to posts by Ivan, Bob and Tom for more information about VIRL.

What can it be used for?

Here are some ideas on how to use it:

  • Create Proof of Concept (PoC) to demonstrate to potential customers
  • Training
  • Try changes in a simulated environment before you deploy it
  • Test features before deploying them in your network
  • Create a replica of customers network to be familiar with it
  • Recreate errors so that TAC can work on them outside of production

These are just some things that I thought VIRL would be useful for.

What can I do to spread the word?

I would like to encourage everyone to spread the word on VIRL, let’s go viral! For any
professional networking individual the importance of this tool could be huge. Blog about
VIRL, tweet about it. Approach your Cisco account team, let them know you are interested
in trying it out. The more pressure we put on Cisco, hopefully they make the right decisions
when releasing it.

What do I want to see from VIRL?

From what I’ve read about VIRL most of aspects of it seem very promising. I just hope Cisco
gets the packaging right. I hope to see a free version that everyone can use for training
and certifications. This would be huge for the community and would make it easy to choose
Cisco as the preferred vendor.

There might be an appliance or commercial version. This will cost some money. That is fine,
it costs money to produce code and support it. I would still urge Cisco to not overprice it
though. The value you would be adding to enterprises would be coming back tenfold in fewer
TAC cases and increased product sales.

Will it be crippled? For the CSR1000v there is a bandwidth cap which is a great way of
producing a free router without crippling it further. I hope to see the same from VIRL.
Don’t cripple it in other ways and don’t make it a licensing jungle.

I hope that VIRL can be easily deployed together with VMs for testing things like multicast
and injecting BGP routes and so on.

So go on, spread the word! And let’s hope VIRL becomes everything we want it to be.

Categories: Announcement Tags: , , , , ,

One year as a CCIE

October 23, 2013 6 comments

One year ago today I passed the lab in Brussels! Time flies. That means I have
one year to recert. My plan is to use the DE written to recert.

Good luck to anyone pursuing the CCIE!

Categories: Announcement

Network Campus Design

October 18, 2013 13 comments


Modern networks need to be enabled for voice and video. These applications
do not tolerate a lot of loss before quality becomes unacceptable. This
requires us to build networks that are scalable, resilient and converge
quickly. This post will describe key points of building a network that
fullfills those requirements.

Hierarchical Network Design

It’s important to think of the network in terms of building blocks and
hierarchy. Define different building blocks like campus, small remote site,
medium remote site and large remote site so that not every new network
needs an unique design. Building a hierarchical network will:

  • Make it easier to understand, grow and troubleshoot
  • Create small fault domains, clear separation of layer 2 and 3.
  • Allows for load sharing and redundancy
  • Deterministic traffic patterns and convergence

The key point is to not end up with a network looking like this:


Like anyone working in the real world I know that budget constraints, lack of
available fibre or many other factors can limit our network designs. We can’t
always win but we should make it clear to the company/management that we can’t
support voice and video unless we design the network as it should be. Do you want
to be on call for a network that is poorly designed and maybe you are the only
one that knows how it works? In Optimal Routing Design, Russ White talks about
the 2 AM test. If someone calls you up at 2 AM do you know how your network works?
If you don’t it’s a sign that it is too complex.

The different building blocks of a hierarchical network

Traditionally networks have been built in a three tier model. This model
consists of access, distribution and core. In smaller networks it may be
acceptable to have a layer that acts as both distribution and core and
this is called the collapsed core model.

Access Layer

The role of the access layer is to:

  • Provide connectivity into the network
  • Enforce security to prevent ARP/IP spoofing
  • Boundary for trust for QoS model
  • Provide PoE for phones and access points

Distribution layer

The role of the distribution layer is to:

  • Aggregate wiring closets(access layer) and uplinks to core
  • Provide high availability, load sharing and QoS
  • Protect the core from high density peering and problems in access layer
  • Summarize routes towards core and fast convergence
  • Provide first hop redundancy towards the access layer

Core Layer

The role of the core layer is to:

  • Provide connectivity between all the building blocks
  • Provide high performance and high availability
  • Aggregate the distribution layer
  • A separate core layer helps with scalability

When do I need a core layer?

There is no text book answer to this question but consider the following


There are currently two building blocks. Every distribution layer switch has
three IGP neighbors and 3 links. What happens if we add another building


The network went from three IGP peers to five IGP peers. Also the total number
of links went to 15 from previously 6. You can see that this gets out of hand
pretty quickly.

Different Campus Designs

There are a few common designs that can be used to build the campus access and
distribution layer. Which design that fits best will depend on if there is a
need to span VLANs and how modern equipment you have in your network.

Layer 3 Distribution

Layer 3 distribution

This design has no VLANs spanning the switches. This is what we want to have
but usually there is some requirement that keeps us from doing this. It could be
some application, Vmotion or maybe one common wireless network across the entire
campus. There is no layer 2 loop in this design which means we are not relying
on STP for convergence. Here are some points to consider for this design:

  • Tune CEF to avoid polarization leading to underusing links
  • Summarize routes towards the core
  • Don’t peer IGP across links unless you intend to use them
  • Set the trunk mode to on/nonegotiate
  • Ports toward users hardcoded to access and enable portfast
  • Configure root guard or BPDU guard towards users
  • Enable security features such as DHCP snooping and DAI

Layer 2 Distribution

Layer 2 distribution

It’s quite common that some VLANs need to span the campus. This means that there
must exist an L2 link between distribution switches. There is now a loop in the
topology so convergence is dependant on spanning tree. Some points to consider
for this design:

  • Tune CEF to avoid polarization leading to underusing links
  • Summarize routes towards the core
  • Don’t peer IGP across links unless you intend to use them
  • Set the trunk mode to on/nonegotiate
  • Ports toward users hardcoded to access and enable portfast
  • Configure root guard or BPDU guard towards users
  • Enable security features such as DHCP snooping and DAI
  • Align STP Root and HSRP primary on the same distribution switch
  • Put Root Guard on downlinks (facing access switches)
  • Put Loop Guard on uplinks (facing distribution switches)

Routed Access

Routed access

The routed access design has no layer 2 links. It’s all routing which means
convergence is fast, no links are blocking and equal cost routing can be used.
The drawback is that no VLANs can span the topology. If MPLS is enabled in the core
some VLANs could still be able to span through the use of EoMPLS, VPLS etc. Key
points to consider for this design:

  • How much more will routed access cost me?
  • Do I need the performance/convergence gain?
  • Do I have the need to span any VLANs?
  • How many routes do my access layer devices support?
  • Summarize routes towards the core
  • Summarize routes towards the access
  • Tune CEF to avoid polarization
  • Don’t peer IGP across links unless you intend to use them
  • Ports toward users hardcoded to access and enable portfast
  • Configure root guard or BPDU guard towards users
  • Enable security features such as DHCP snooping and DAI

Layer 2 Distribution with MLAG

Layer 2 MLAG

Newer designs can utilize newer features like stacking, VSS and vPC. This means that
VLANs can span access switches but there is no physical loop because MLAG is used.
This gives us the advantage of a layer 2 distribution without the disadvantage of
relying on spanning tree for convergence. There is no need to run HSRP becaues the
distribution layer is acting as one device. The key points are similar to the layer 2

  • Tune CEF to avoid polarization leading to underusing links
  • Summarize routes towards the core
  • Set the trunk mode to on/nonegotiate
  • Ports toward users hardcoded to access and enable portfast
  • Configure root guard or BPDU guard towards users
  • Enable security features such as DHCP snooping and DAI
  • Put Root Guard on downlinks (facing access switches)
  • Put Loop Guard on uplinks (facing distribution switches)

If doing a new design I would definitely go with some form of stacking or VSS or vPC
if deploying Nexus switches. This gives us the flexibility of using layer 2 in distribution
but still not needing to rely on STP and FHRP for convergence.

Recommendations for Fast Convergence

  • Use only point to point interconnections
  • Use fiber between all devices for fast convergence (debounce timer)
  • Tune the carrier delay timer
  • When possible use IP configuration on phsyical interface over SVI

I did a separate post on Detecting Network Failure which goes into more detail
on detecting failure.

Why should physical interfaces be used over SVI? The following steps take place when
converging on a physical interface:

  1. Link Down
  2. Interface Down
  3. Routing Update

When using an SVI there are some additional steps however:

  1. Link Down
  2. Interface Down
  3. Autostate
  4. SVI Down
  5. Routing Update

When using an SVI when the link goes down the switch must check if there are any
other ports up with that VLAN configured. If it is the SVI won’t be brought down.
Even if there isn’t it takes time to go through all the interfaces before declaring
the SVI down. This can worsen convergence by a good 200 ms. If you do use an SVI
then make sure that it is point to point so that it’s not allowed on any other
links than the link connecting the two switches.

Recommendations for Spanning Tree

  • Don’t span VLANs across switches unless neccessary
  • Use RSTP or MST for best convergence
  • Even if you have no loops, STP is needed to protect against user side loops
  • STP can protect against misconfiguration or hardware failures creating loops

Layer 2 Hardening

Cisco recommends the following features for hardening layer 2 in a campus design:

Layer 2 hardening

I agree with most of this but there are some caveats.

One issue is with Root Guard. Why do we run Root Guard? To protect against another
switch dictating the bridging topology. I would set the root to a priority of 0 and the
secondary root to a priority of 4096. That should provide protection and if you don’t
trust your employees to not mess up the network that is an education problem or
management problem. Restrict user accounts in Tacacs what they can’t do to remove
potentially dangerous commands such as switchport trunk allowed vlan, no router bgp,
no router ospf and so on.

So what is the issue with Root Guard? The STP root will also be the HSRP primary device.
Because the network is designed with Equal Cost Multi Paths (ECMP) some traffic may
arrive at the standby HSRP router. This is the network without blocking links:

Root Guard step 1

No issues so far except that the crosslink is being used but it’s not a major deal.
But what happens if the links between the HSRP primary and standby fails?

Root Guard step 2

The access switches are sending superior BPDUs but the secondary distribution switch will
block the link due to Root Guard being implemented. This means that any traffic arriving
at the secondary distribution switch destined for the access layer switches will be
black holed. That is why I would not implement Root Guard towards the access layer.

Recommendations for Layer 3

Here are some recommendations for Layer 3:

  • Build triangles not squares for deterministic convergence
  • Use passive-interface default and only peer on links used for transit
  • Design the network with dual Layer 3 paths for resiliency
  • Summarize from the distribution to the core to cut down on flooding and Active queries
  • Tune CEF to avoid polarization of linkz

What happens if we design with squares?

Routing Square

If a device goes down the network has to rely on flooding of updates or LSAs before it
can converge. There is no secondary path that can be immediately installed.

But if the design is a triangle instead:

Routing Triangle

There are already dual paths so losing one won’t affect convergence and the other route
is already in the FIB so traffic can keep flowing.


There are many network designs out there. Learn the strengths and weaknesses of
different designs. Look at best practice designs from the vendors but don’t follow
them blindly. As I have shown sometimes recommendations will not work for all

Finding the right design depends on business needs, budget and what kind of applications
that are running. Read more on campus design in BRKCRS-2031 and also look at the Cisco
Validated Network Designs