If you’ve done any discoveries that look at existing processes, you’ve probably come across the term shadow system. It’s used to describe the unofficial systems people use to get their job done because the official one isn’t cutting it.
Maybe it’s a team using Google Drive instead of the official file sharing network because it lets them collaborate with external users. Maybe it’s a team filling out Word Document templates when a user calls and their main system is down or slow to respond. Maybe it’s a team tracking emergency repairs in spreadsheets because it gives them flexibility in how they schedule contractors to respond to them.
Why shadow systems are so common
Every time we do a discovery with a new team or a new organisation, we find shadow systems like these. They’re typically seen as a failing of the current systems. Something to workaround a shortfall in the designed system that isn’t fit for purpose; something to be fixed in the design of the new system – after all, isn’t that why we’re doing this work?
On the surface this seems reasonable – if people are working around problems in the system because it doesn’t meet their needs now, then the system will never have met that need. If we can design it better this time, we can avoid these shortfalls and stop using shadow systems!
But that assumes that the people and their processes are static – it implies that the team has been having to do this workaround since day one. People change, and processes adapt. The services we deliver aren’t fixed, static things. They’re reactive to the needs of users and to the constraints of the organisations delivering them.
In an ideal world, the systems teams use and digital services they support would adapt to those changing needs and constraints. Far more likely, however, is that the team make compromises and find creative workarounds to slow the inevitable rot of the service.
Designing for internal users
The GOV.UK Service Manual has some very brief words to say on the matter:
- good internal services reduce training costs and the amount of time spent dealing with errors. This gives people more time to provide a good service to the public
- hold internal services to the same standard you would a public-facing service
The guidance basically boils down to “treat internal staff as you would any other user”. It still assumes that a team is designing something for those users. The team might even use a co-design approach. But ultimately, the admin users end up relatively disempowered by this.
The expertise required to design and build services in line with the service manual is high. The amount you have to learn to even get started as a junior developer in a multidisciplinary team is vast. Admin teams are unable to make changes to their own tools, through lack of expertise and lack of access (often premised on that lack of expertise).
Those tools encode process, which means the teams are unable to change their processes. This prevents the very thing we preach for agile delivery teams, encouraging them to use regular retrospectives to reflect on how they worked and adapt their processes to make them more effective.
The pros and cons of building tools
Turning processes evolved in small teams from spreadsheets and emails into “proper” tools, designed and built by digital teams, formalises those processes. That makes it more robust, more repeatable. Hammering out edge cases in a way that software can handle them, rather than relying on a human, ensures consistency. These are often prerequisites for scaling up beyond the size of a small team.
We’re working with a team in Hackney to convert their spreadsheet-based process into a more formal digital tool. Integrating the tool with other services across the organisation allows the team’s impact to scale beyond their own service, so it makes good sense when you’ve got the strategy and means to enable it.
But as well as formalising the processes, you can end up calcifying them. Where, before, the team could have adapted by changing their spreadsheet, they no longer have the knowledge or access to change anything. Either they stick with doing it the official way, or they start finding workarounds outside of the system. As GDS put it recently when writing about managing spreadsheet data, “because they’re not in direct control of their own tools they gradually revert back to their spreadsheet-based workflows”.
Embracing flexible generalist tools
I want to take a closer look at those shadow systems, particularly those built on spreadsheets. A spreadsheet is a really powerful, flexible, generalist tool. It forces you to structure your data into rows and columns, labelling data and providing individual records. It allows you to make extra tables if you need to model more than one thing. It lets you write formulas to do calculations for automation or reporting. It lets you use conditional formatting to highlight problems in reports or records that need action.
It doesn’t really impose any limits or restrictions on what you want to do. And because it’s all written in a spreadsheet, one cell at a time, the people who use it can change the system at will. Although it has structure, the way a team uses a spreadsheet is ultimately managed by convention and agreement, not by strictly defined validation or workflows.
There’s a lot of buzz around teaching people to code, and around using low code platforms, but I think this is sometimes misdirected. It assumes that the problem is a skills gap, that people need more digital skills so they can build great tools and services. I don’t think that’s necessarily right or fair. You can field entire teams of people from your organisation who can already use spreadsheets and emails. Instead of focusing on teaching them new tools, what if we focused on how to use those existing tools more effectively? On how to build processes that are sustainable and consistent?
So, in summary, some things to think about when you’re designing admin systems:
- don’t rely on a once-and-for-all design to fix your problems – your legacy systems might not be doing a great job now, but that might well be down to their failure to adapt rather than their initial design. As you’re building it, actively consider how any new system will adapt to the changing needs of your users and your organisation
- spreadsheets can be the most appropriate tool for a small team. Try to hold off on formalising a team’s processes into software until the point you need to scale that team or that work
- custom software isn’t a build-it-once-and-done thing, it’s an operational cost – as well as the cost of maintaining and operating it, you need to allow for the cost of iterating it, so that it continues to be the right tool for years and doesn’t get replaced by shadow systems of its own