Another flippin’ year goes by with curtailed social interactions, travel and that raised baseline of anxiety but aside from all that, here’s what I’m thinking.

It’s been another tricky year for lots of businesses. Network assets get sweated for a bit longer because everyone’s a little uncertain, so you don’t spend money you don’t need to. For me, that meant the year began with the threat of redundancy. More on that later…

Wi-Fi continues not to be taken quite as seriously as it should be by most enterprises. As Keith Parsons commented in his London Wi-Fi Design Day talk, Wi-Fi doesn’t punish you for doing it wrong. That’s not strictly true as a really badly designed network can fall apart entirely under load, something I’ve seen a few times, but his point stands; namely you can throw a load of APs into a space with very little thought and… it works.

An interesting challenge I’ve encountered has been organisations ignoring the design, doing what they’ve always done, and then complaining they have issues. I’ve also encountered an organisation stating “Wi-Fi is not business critical” when clearly it is because as soon as there are issues it becomes the highest priority. In many circumstances Wi-Fi is the edge everyone uses. Not only is Wi-Fi as business critical as the wired network, in many cases it’s more so. Of course the wired edge supports the wireless edge, we need it all to work.

And on that subject, we continue to see a separation between wired and wireless in many network teams. In my mind this doesn’t make sense a lot of the time and I consider the wireless and wired edge to be the same domain. Plus we all know the wires are easier.

Wi-Fi6E has arrived and will totally solve all our problems… It’s something I’ve yet to play with, but lacking either AP or client that will have to wait. Wi-Fi6 (802.11ax) has arguably so far failed to deliver on the huge efficiency increase promised I think because legacy clients persist, and will do for many years to come, but also because the scheduling critical to OFDMA is probably not where it needs to be.

It was also really disappointing to see Ofcom release such a small amount of 6GHz bandwidth here in the UK.

This has been another year of firsts for me, working with high profile clients, learning new technologies (gaining Checkpoint certifications), embracing project management (I’m a certified Prince2 Practitioner)

It’s also particularly struck me how much my so-called “career” has been held back my ethics and, what could refer to as loyalty but, if I’m honest, is probably comfort. I don’t have a complaint about this, I am who I am, this is just an observation.

I’ve been approached for, and turned down extremely well paid roles working for organisations who’s core purpose is something I cannot support. I’m happy with this. If I’m not working to make the world better, I at least need to feel I’m not actively making it worse.

This is no judgement on anyone who does work in areas I disagree with btw. We all have to make our own decisions about what matters in life.

The loyalty/comfort is a trickier area. I declined a really interesting opportunity that meant moving overseas. Ultimately that was probably the right decision but, on reflection, it was taken too quickly. I work an interesting role for a great company, long may that continue, but when the feet start itching or an opportunity comes my way, I need to be ready to embrace the uncertainty. Starting the year with redundancies, and losing a great member of the team, served as a reminder that loyalty in employment is transactional.

I was recently invited to speak at the London Wi-Fi Design Day hosted by Ekahau and Open Reality. It was a fantastic day, great to catch up with people in person and some excellent talks (if I say so myself).

You can watch all the talks, from this and previous years, on this playlist. Talks from London 2021 start at number 41 as the list is ordered today.

My talk, some thoughts on Wi-Fi Design in Higher Education can be watched below.

I was particularly challenged by Peter Mackenzie’s talk on troubleshooting and the idea we all tend to jump to answers before we’ve asked enough questions. Hugely helpful and highly recommended.

The ClearPass Galleria template makes it easy to have a pretty, screen format responsive captive portal. It’s mostly a case of chuck some images in, set the colours, and then it looks great.

Some of the defaults are a bit odd though, especially the logo size. Recently I built a captive portal for a customer and their logo was king but in Galleria it was far too small.

Fortunately Aruba have made it easy to override CSS in the template. From Administration \ Plugin Manager, choose Configuration on the Galleria plugin in use.

Add your CSS overrides in the HTML HEAD section. This can be used to override any of the CSS in the template. For the logo you want to play around with this:

.nonav-logo img {
max-width: 100%;
max-height: 220px;
}

Another of my occasional non Wi-Fi related posts.

There’s often talk in political circles about “cutting red tape” and it’s almost always accepted as a good thing. After all, red tape is that irritating thing that stops us getting stuff done…. and it’s red, which is bad.

But I wonder if you’ve ever stopped and questioned what “cutting red tape” actually means. What is it you want to do, what or who is stopping you from doing it, and why?

A really good example from the world of corporate IT is change management. I don’t know anyone who likes change management. It’s stupid red tape that slows everything down, stalls projects, stops you from simply getting stuff done. It’s the very definition of red tape that we could happily do away with. Yet pretty much every corporate has some form of change control… So if this red tape is so bad, why do so many organisations embrace it?

Back when I ran my own IT department looking after the servers, desktops, and everything in between I could just do stuff. Occasionally when I did stuff everything broke and it was on me to fix it. I was answerable to other people, but they didn’t really understand what I did and generally they were just happy I’d made everything work, and wasn’t I clever.

Later, early on in a different role, I broke something but this time there were questions… managers wanted to know who had approved the change I’d made, the director wanted a report from the change manager about what lessons had been learned and why our roll back plan hadn’t worked (it didn’t exist). It all got very uncomfortable because, unbeknownst to me, there was some quite important red tape I’d just moved out of the way and crawled under that existed to make people like me think through changes more carefully.

We have red tape across our society and, yes, some of it is not helpful. Just as change management can be an unhelpful barrier, a pointless box-ticking exercise and a process that does little more than provide a handy scapegoat on which to dump the blame, red tape in the public sector can absorb time, effort and money, delivering very little.

However when it’s done right, it’s fantastic regulation that empowers people to do their jobs, supports individuals and teams through difficult decisions and prevents those who might take dangerous shortcuts for their own financial gain from doing so, or at least holds them accountable.

When someone talks about “cutting red tape” it’s important to understand what they mean. Is it removing unhelpful bureaucracy or is it removing important protections.

For example in the UK you can’t just do whatever you want with land you own. Planning permission is required for a great many things from modest house extensions up to large scale projects. Planning decides whether you can do it and then building regs serve to make sure it’s done properly. Again, nobody likes planning permission but allowing anyone to build anything without controls would be a complete disaster.

I also wonder who benefits financially from that particular red tape being removed.

Recently it’s been found many of the UK’s water companies are illegally emptying raw sewerage into watercourses, polluting our rivers, the sea around our coastline and our beaches. There are plenty of rules against this sort of thing, but the consequence for breaking these is a fine which, so the accusation goes, is simply absorbed into the cost of doing business. Sometimes that troublesome regulatory red tape is performing a really important task and getting rid of it leaves the way open for unscrupulous operators to do really damaging things. Perhaps, sometimes, rather than less red tape, we need a bit more.

A previous post waffled about setting up Radsec between an Aruba AP and ClearPass running in Azure. Having deployed this in a slightly different Azure environment I’ve learned some things worth sharing.

The first thing is ClearPass handles RadSec using RadSec Proxy. This receives the RadSec connection and proxies the RADIUS traffic to the ClearPass RADIUS server. One casualty of this approach is that, at the time of writing, Policy Manager sees these incoming connections as being from localhost.

There are circumstances where it’s useful to know which ClearPass cluster member has dealt with a request. For example if I don’t have a cluster VIP, and that’s not supported in Azure, it’s possible to build HA for a guest portal if you know which server handled the initial MAC-auth. With RadSec you don’t, so this can’t be done.

My Azure lab was very simple, just a ClearPass VM running in an Azure tenant with no Network Security Groups configured. In the production environment I’ve worked with ClearPass was placed in a NSG protected by Azure Firewall which performed source NAT. As a result incoming RadSec connections all have a source of the NSG firewall private range, rather than the true source.

The RadSec tab of the Network Device config in ClearPass lets you set an override source IP. Previously I put the NAT public IP of my client in here. In production I’ve used the internal /26 network range of the azure firewall. The config box takes a range, in this case its 172.20.0.0-62.

I’ve then used the option to validate the certificate, using the CN or SAN and entered the CN name of the cert issued to that client.

The final point worth noting is neither ClearPass nor the Aruba AP issue keepalives down the TLS tunnel. The Azure firewall drops idle connections after four minutes. This is not configurable. So you will see lots of up and down messages in the event viewer. When I was initially testing this in my lab setup this was an indication of a problem. In production it was just normal behaviour, which caused some confusion.

In a busy network with lots of authentication traffic you’ll probably see the tunnel stay active for much longer. I’m not entirely convinced all is well, but it’s working.

Here’s a little gem found by a colleague.

A customer has Aruba AOS 8 Gateways communicating with the Mobility Conductor via a fortinet site to site VPN.

The Fortigate units were upgraded and the IPsec traffic from the Gateways to the Mobility Conductor was being dropped. Nobody noticed for a month until the Gateways stopped working because the licenses couldn’t be validated.

My colleague was able to prove the traffic was getting to the Fortigate and was passing through the rules but didn’t arrive at the other end of the tunnel. This smelled to me like an ASIC or acceleration issue, and that’s exactly what it was.

Lots of head scratching later… the issue was solved with npu-offload disable.

The new Fortigates had this acceleration function enabled by default and, it turns out, you can’t pass the Aruba IPSec traffic across the IPSec site to site VPN with npu offload enabled. It might be possible to make this work by changing the IPSec parameters, although I’m not even sure it that’s possible.

As it is, this customer doesn’t need to accelerated performance, they just wanted it to work.

Congrats to Colin Irwin who figured this out.

Most of the time we’re using RADIUS over a private management network and don’t really care too much about security, but that changes when you put your authentication server in AWS or Azure and send authentication traffic over public internet. RADIUS is not secure and uses an MD5 hash for encryption that doesn’t meet modern standards. This is where RadSec comes in.

RadSec is effectively RADIUS over TLS with client and server certificates used to authenticate and encrypt the traffic. RadSec uses mutual authentication – at the setup of the tunnel both the client and the server need to successfully authenticate each other for everything to work. Worth noting RadSec only uses a single port – TCP:2083 – rather than separate ports for authentication, accounting and change of authorization (dynamic RADIUS).

To lab this I have ClearPass running in Azure. The authenticator is an AP on a private network, connecting to public internet via a NAT gateway and managed by Aruba Central.

For the server side this certificate authentication is probably familiar territory. You add a server certificate, signed by a CA the client will recognise, and that’s about it. You may of course need to add the CA root to your client’s certificate store. Here’s my RadSec certificate in ClearPass, signed by a private CA.

The network access device entry has the actual private IP address of the AP so I can identify it more easily but ClearPass will see an incoming connection from the NAT public IP. I’ve entered this into override IP under RadSec settings so ClearPass will accept the connection. I can also add certificate validation checks. This lets you separate multiple authenticators behind a single NAT IP. For example if you have both switches and APs on a site and need to handle these differently, placing them in different NAD groups or applying varying attributes. In this case I’m not performing any authorization checks.

Because it’s a private CA I’ve uploaded the root certificate that signed the server certificate (labca_root) into Central and applied this on the group security settings. The CA root is needed for the APs to authenticate the server. I’ve also uploaded a client certificate (radsec_client_san) that the APs use when contacting the server. The same CA was used for this so the root certificate has also been added to the ClearPass certificate trust list.

The way Aruba Central handles things, these certificates are used by all devices in that group. So all my APs will use the same client certificate. Best practice with Aruba Instant is to proxy RADIUS traffic via the Virtual Controller. This changes in AOS10, but it’s likely a single certificate used to authenticate all devices within a group is just fine. You can assign certificates per device using template groups but you’d really need a very good reason to want to do that. Don’t make this more complicated than it needs to be.

That’s where most of the documentation ends. But it didn’t work.

The one place I got stuck for an embarrassingly long time was getting the client certificate to work. I’m no expert on working with OpenSSL but to save you my pain, here’s what I eventually realised.

All certs, server and client, should have a subject alternate name with the IP address. This means things still work if the client can’t resolve the DNS name of the server, and it’s essential for the client certificate to have the IP address from which the client will connect included, so in this case it’s the NAT public IP. You can include multiple IPs for a certificate that’s being used across various sites. I don’t believe there’s a viable solution for not having static addressing. Dynamic DNS won’t help you here.

The client certificate needs to be just that, a client cert. Specifically it needs the X509v3 extended key usage variable of: TLS Web Server Authentication. In this output from OpenSSL you can also see the Subject Alternative Name appears where, trust me, the IPs are listed.

If you don’t have either the IP address your authenticator is connecting from in the certificate, either as the CN or in the SAN, or the TLS Web Server Authentication extension it won’t work and you’ll get unhelpful errors on the AP such as:

radsec_read_from_tls_socket:311: some error(6) encountered while reading

If, like me, you’re using ClearPass and, unlike me, have an Onboard CA you can generate the client certificate there. For my purposes I’m using OpenSSL to do this. There lots of tutorials in how to use it, but here’s what I did.

Create a config file with the necessary settings, which I saved as san.cnf

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
C = GB
ST = The Shire
L = York
O = Homelab Inc
OU = IT
CN = aps.homelab.home
[v3_req]
extendedKeyUsage = serverAuth, clientAuth, codeSigning, emailProtection
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = <ip1>
DNS.2 = <ip2>

Generating the CSR: openssl req -nodes -newkey rsa:2048 -keyout radsec_client.key -out radsec_client.csr -config san.cnf

Making the certificate – for this I’d already created my private CA root, which is what’s referenced as the root.pem and root.key: openssl x509 -req -in radsec_client.csr -CA root.pem -CAkey root.key -CAcreateserial -out radsec_client.crt -days 7200 -sha256 -extensions v3_req -extfile san.cnf

In order to import the client cert it often needs to be in pkcs12 format: openssl pkcs12 -export -out radsec_client.pfx -inkey radsec_client.key -in radsec_client.crt

There’s no requirement for the client and server certificates to use the same CA, in this case it allows easy rolling of certs that are valid for a long time, which is beneficial for this use case. Using OpenSSL for a simple private CA is really, easy, here’s a good run through.

As I said before, getting all this to work took an embarrassingly long time. In my defence this is mainly because when things don’t work there’s very little useful information logged, hence this post which may be useful to someone, not least me when I need to do this again in a few years.

One final thought on this. Certificates get forgotten and expire causing merry hell with infrastructure when they do. In this deployment, using ClearPass and Central managed APs it’s trivially easy to update the certificate so if you have to use a CA that doesn’t let you create long expiry dates that’s probably not the end of the world. However be sure to set a diary reminder so that work can be scheduled. It’s quick and easy to do but it’s still potentially disruptive.

As certificate expiry terms get ever shorter, there remain areas where mutual authentication is valuable but changing things, even by automation, is undesirable. I’d still advocate using a nice long expiry date for certificates internal to infrastructure. I admit the 20 year cert life span of my lab CA is, perhaps, a bit extreme

Here’s a video post about an interesting building construction that, probably, few will encounter so I thought it might be interesting. Also the perils of making assumptions about building construction when working on Wi-Fi designs.

Recently my friend and former 😢colleague Mac_wifi has called on others in the community to publish their CWNE application essays as a resource, benchmark, encouragement to others seeking to achieve the CWNE certification. I think this is a good idea. Nobody had asked me to write an essay in over 20 years and I wasn’t quite sure where to start. Hopefully a few of the current crop of CWNEs sharing their submission will help guide others.

Here’s a link to that blog post

I agree with the CWNP’s approach to their expert level. If you’re used to sitting multiple choice exams through Pearson Vue for your IT certs the need for an essay might seem a bit strange. Indeed many certification streams award the expert level automatically once enough exams have been passed. There’s nothing wrong with that approach but CWNP chose to take a different route for a good reason.

To become a CWNE, in addition to passing the exams, you need at least five years experience working in enterprise wireless networking, have people prepared to endorse you and write three essays to demonstrate that knowledge and experience.

What that means is a CWNE knows their stuff, has walked the walk and … well you know how the rest of that goes. It means a CWNE has experience that goes beyond the theoretical knowledge one can gather by studying for certification.

Some advice I’d offer anyone aiming for CWNE certification is to gather real world examples of solutions delivered and problems solved for your essays. In the rushing around of daily life it’s easy to fix a problem, move on to the next thing and then forget about the details. It’s also worth remembering your essays don’t need to be a technical tour de force of excellence, proving your capabilities as a Wi-Fi super human. We all know the real world is full of compromises. Name them and explain them. I consider an essay that knowingly describes a flawed implementation speaks more of experience than a textbook “perfect” network design, for example.

So those essays…

Essay 1 – Design
Essay 2 – Security
Essay 3 – Analysis

I’m always happy to talk further with anyone about this, if you want to get in touch you can reach out to me on twitter