Cisco ACI - 1

Why ACI:

Easy management

Scaleable

Security (follows whitelisting model)

No more spanning tree in the core

Devices replacement is super easy and time effective

Components:

Spine switches

Leaf switches

APICs (3 at least)

CLOS Design / Architecture

CLOS talks about connecting the interlayer devices with each other but not the intra layer devices.

APIC

Application policy infrastructure controller - Brain of the fabric

Works in a cluster - made up of at least 3 controllers. Scalability of the cluster is directly proportional to the needs of the ACI deployment and is scaled based on transaction rate requirements.  Any controller in the cluster can serve any user for any operation. *

Config is pushed from APIC to all the devices

Doesn't participate in the data plane

Connected to LEAF switches only

APICs are available as physical or virtual appliances

Physical APICs are 1 RU Cisco C-series servers with ACI code install and come in two different sizes : M (medium) and L (Large)

There are 3 generations of APIC UCS servers:

1st generation = Cisco UCS C220 M3 servers

2nd generation = Cisco UCS C220 M4 servers

3rd generation = Cisco UCS C220 M5 servers

*Number of APIC nodes in a cluster

3 node cluster = up to 80 leaf switches

4 node cluster = up to 200 leaf switches

5 or 6 node cluster = up to 400 leaf switches

7 node cluster = 400-500 leaf switches

For implementations requiring greater number of switches than above, it is advisable to go for multi-site architecture, using several APIC clusters.

Database is distributed between APICs in the form of a "shard", which is essentially a unit of data / subset of the database, or a group of database rows distributed across the nodes of the cluster to enable parallel processing and data protection in the event of a failure. Each APIC database has exactly 3 copies / replicas which is distributed across the cluster based on shard layouts defined for each cluster size.

Refer this link for further information on sharding in Cisco APIC = https://community.cisco.com/t5/application-centric-infrastructure/unraveling-the-concept-of-database-sharding-in-cisco-aci-apics/td-p/4972877

We can write the information only if we have more than half the APIC size active in the network.

For e.g. if we have 3 controllers then 2 APICs must be available all the time to perform the write operation. If 2 APICs go down, the state of remaining one APIC will fall to read only because of the sharding behavior which says that for a particular data set at least 2 copies must be present at a given time (this is called quorum).

Spine Switches

Used to exchange routing information between LEAF switches using ISIS protocol

Used to provide information to leafs about the endpoints via COOP (Council of Oracle Protocol)

Works as route reflector for all the leafs and distribute the external routes into fabric using the MP-BGP

Leaf Switches

Servers and external devices like switches and routers connect to Leaf switches

APIC will be connected to LEAFs only

All the security policies will be implemented on leaf switches

Types of Leafs

Border Leaf - connected to external domain servers

Compute leaf - connected to servers

Service Leaf - connected to devices like FW and LBs

Hardware overview

ACI runs on Nexus 9K switches

We have chassis switches like 9500 or fixed switches for example 9300.

A 9K switch can be either in NX-OS mode or ACI mode but not both at the same time.

Cisco ACI scalability guide - Check this out for switch specs

Nexus 93180YC-FX Leaf switch

54 ports

last 6 ports - fabric ports which connect to spine switches only

Fabric ports can be changed to downlink ports

All spine switch ports are in "routed" mode. The fabric ports of leaf are in "routed" mode.. the remaining downlink ports are in "trunk" mode.

Cisco IMC (CIMC) ports provide out-of-band access to the server hardware itself. This is for the APIC server itself.

Command to check interfaces on APIC CLI: cat /proc/net/bonding/bond0

Cisco ACI Multi-Pod solution = stretched-fabric

It connects multiple APIC pods via Cisco IPN i.e. Inter-Pod Network

IPN supports OSPF only between IPN and spine switches

Though each pod consists of its own spine and leaf, all the pods in a multi-pod setup share a single APIC cluster.

At the data plane, the multipod solution uses MP-BGP EVPN connectivity over the IPN between the spine switches from each pod for communication over VxLAN encapsulation.