In writing the post earlier about Chorus Fibre Lab, and because it's raining outside now, I thought I might go thru the thought process of building a small ISP, including the last mile of actually running cable to customers.
So lets see how this goes. I'm going to use Ubiquiti as my products, only cos I find their website pretty - I suspect that would be a bad choice for most actual cases.
And I'm not going to try to design for a multi-tenant system like Chorus has to - this is one company owning the lines, the exchange, and providing the network to a small town. And owning the town. None of this Resource Management Act nonsense, this is top-down communist planning, but with fibre. Think 80s Soviet Russia, with high-rise buildings with 64 apartments, and modern technology.
More on Pripyat.. if thats your jam
We have 8 apartment blocks, each with 4 floors, and each floor having 16 apartments, as well as a central exchange for our ISPs hosting. Call it an exchange, in old terms.
Customer Premises - the ONT
For the customer to plug something into, you need some equipment on site. This is the ONT - Optical Network Termination. It's a small box which has fibre in one side, and ethernet out the other. Each one has an ID, so the other end knows who's who and how the shared medium is shared.
Ubiquiti have these - the Fibre XGS-PON CPE. Nice (looking) little unit. We'll need one of these per apartment, for the customers to plug their router into.
Fibre back to the Exchange
In theory this could be just like a piece of Ethernet (point to point) or it could have multiple splitters, so its point to multipoint. GPON handles a single connection having many (usually around 64) clients on a single port with optical splitters.
On our case, we'll run a single fibre into each apartment block, and split it 4:1 (one for each floor) then 16:1 on each floor. Each floor has a long-ish single fibre from the riser, where the splitter is, to the ONT inside the apartment.
The single fibre into the building would not be a single strand. The cost isn't the cable - its putting the cable into the ground. So run 4 per apartment block cos why not. Redundancy. Hard to get new cable into the tundra in winter.
This is fairly simple when you think about it: you need a wire back to the exchange, to the OLT.
I say simple, but you're going to need people to run the cable, put splitters in appropriate places, drill holes... ok, less simple. But hey, we own the town in this experiment. Drill baby drill.
The OLT / Exchange
This is the other end of the fibre - the OLT or Optical Line Terminal. It combines the 64 ONT's into one port, and handles the multiplexing of light down the fibre - then multiply that by the number of ports.
Out the other side is 100Gbps (4 x 25Gbps) of... something. It depends what medium you put in your SFP cages, but likely fibre again, point to point.
We are still on layer 2 at this point, so it's all one big network, like you plugged 2048 computers into 8, 64 port switches, and then connected those 8 switches back to another single switch. The switch has the smarts and the map to know that ONT 86 is down port 8, and the port/optics handles chopping the light up so that ONT can see it.
In this case, you have 2048 ONT's hanging off this OLT. The building would have 8 fibres (well, bundles of 4 fibres) going out of the building, one to each apartment block. And one for external access, but thats next.
At this point we have 2048 ethernet ports hanging off a single 100Gbps ethernet port - well, 4 x 25 in aggregation. This of this as a huge ethernet switch over a very wide area.
Actually getting some internet going
Next step up the chain is where you could aggregate these and start running layer 3 - the IP in TCP/IP. Throw in a router, connect it's SFP ports (100Gbps total) to the OLT, and you have a router with potentially 2048 ethernet ports hanging off it (each ONT).
I can't find a Ubiquiti box which will deal with the 100G connection but lets go with this, the EdgeRouter Infinity. 80Gbps of routing over 8 ports. There's likely other products which could aggregate 4 x 25 into something more useful, but I can't find one.
So now we have something which can route IP packets to the endpoints. The upstream from here, I'm going to leave - lets say we have 20Gbps fibre into the town, so thats 2 more of the 10Gbps ports taken, so we have 2 more cages spare for expanstion, either another OLT or more inbound bandwidth.
We are now starting to get into ISP math on how much external bandwidth is needed for a given number of customers (hint: it's not the sum of customers X connection speed).
I'm pretty sure that this is the "border network gateway" (BNG) in ISP terms and appears as the first hop up the chain from the device connected to the ONT. You could also connect a customer router right into a router port over a single ethernet cable. Same result, but it'd be a total waste of capacity.
Other things on the network
We could set this up, with the router handling 210.54.60.0/24 (enough for 250-odd customers, so we'll need maybe a /22) and set each router up with a fixed IP. Or, we could put a DHCP server on the network, and have the DHCP server allocate each customer router (connected to the ONT) a static IP. Or somehow do PPPoE, tho we'd need somewhere to terminate that - thats router to BNG.
We could also put edge cache servers on our internal network - behind/next to the router - so Netflix et al come from a local cache, not down our 20Gbps line. And ban Fortnite, cos the load is too high.
If we were especially evil, we could enable CGNAT on the EdgeRouter (no idea if it does it, but you'd do it at that point), so everyone behind it is on a private, non-routable IP address, using a single (or just a few) external IPs. A lot like what you'd do in your own house.
Seams familiar now?
This starts to feel VERY familiar.
- An input internet connection going into a router (with or without NAT and a firewall)
- A router handling sending traffic to the various machines connected to ethernet ports, either routing traffic, or acting as as NAT / firewall to protect the machines behind it
- The ONT and OLT are really just bits which normally sit inside an ethernet switch, and you don't see. But with fibre. And much much bigger.
- Each machine on the ethernet has its own IP address, either private (and NAT/firewalled) or public (routed)
Seams like what I have at home! But on a very much wider scale.
But there you have it. A possible small-town fibre network. Doesn't seam quite so daunting.
