Autoclassification – where it works, and where it doesn’t (seem to yet).

I’ve been digging into auto classification lately.

It’s one of a number of very promising technologies that will help Records Managers clean up after users who don’t do what they should, and further reduce the burden of users operating in really complex situations.

It’s very promising. The technology works.

The main bar it had to jump for me is beating human accuracy.

The numbers I’ve seen show that people are around 65% accurate when classifying. The numbers I saw at a recent demonstration showed that the Auto-Classification algorithms confidence level on recommendations was at about 80%. This demonstration was from an Australian government agency of a system in active use – so this is real world accuracy.

While it’s an excellent tool. What I haven’t seen yet is its application to the problem of organising content in ways that are useful for people – making content “findable” rather than just well classified.

What I mean by this, is that it is excellent at applying a classification. “This is a development application”. 

Where there doesn’t seem to be an answer yet is in the area of understanding context, and relating one object to another. It seems to be good at saying “this is a development application”, but not at saying “this is the development application for 30 Reed Street Bristol, and these are all of the documents related to that development application” – which means that it can classify, and be useful for sentencing, but not file. 

It seems for the moment that we’re stuck with filing as the simplest way of showing the relationship between content.

While I think auto-classification is still a meaningful and useful improvement that can reduce compliance risk, I don’t think that reduction is as meaningful as it seems. 

My current read on the technology is that the best use is in manage in place scenarios, where we can tag the content with a classification and leave it the way a creator/user organised it to provide context. Without that context, I think we’re creating a problem for ourselves later. We’ll have better sentencing from a subject standpoint, but complying with instruments that require us to understand more context will be difficult. I think it has the potential to make it harder to comply with subpoenas and open government style requests (FOI, GIPAA, RTI, SAR etc.). 

On balance, I’m an advocate. I think that for lots of organisations it will take them from a place of poor compliance, to a place of meaningful compliance, and enable better lifecycle management. Until we have tools that can assess and understand context however, it should be adopted in addition to really good, usable file plans, not as a replacement of them.

The economics of Information Management and Records Management are different – so we should treat them differently.

The value of what we do in people’s hearts and minds is totally dependent on how they understand it.

Records management is about evidence of compliance.

Information management is about driving organisational performance by using information.

That sounds simple, and it is, but they have very different future values, and very different investment constraints.

Getting records management wrong can put you out of business quickly. It can also tie you up in audits, and get you on the wrong side of regulators that can make doing business hard.

The future value of getting records right is being able to stay in business (or retain the trust of the public). When we do Records projects, we’re either bringing ourselves into compliance, or reducing the costs associated with compliance.

Getting information management wrong can also put you out of business – but it’s going to do it slowly.

Poor information management puts you out of business slowly because you lose to competitors who are achieving superior value from their information.

Too often we put them together like they’re the same, and they’re not. 

The economics are totally different. 

We stop doing Records Management projects when we’re compliant, and we can’t meaningfully reduce the costs of compliance. The constraints are the risks of non-compliance, and costs of being compliant.

We stop doing Information management projects (and knowledge management projects) when the organisational performance gain will no longer be sufficient to pay for the project. The constraints are the available market, the availability of capital, and the ability of the organisation to absorb change.

Too often I think we put the disciplines together. There are large skill overlaps, and obviously every record is comprised of information, but they are different. If we don’t think about them differently, talk about them differently, and have a shared understanding with our executive and staff about the difference, we can’t expect that they’ll be valued differently.

So it’s either all compliance – and to be avoided, or all value delivering – and the average of both, so neither gets their full value in people’s hearts and minds.

Does your organisation have a measure of “records debt?”

There’s a commonly used concept in software development called “technical debt”.

It describes all the bad and dodgy (or maybe just sub-optimal) coding that was done in the past, that will need to be cleaned up later.

I think it’s a concept that we can usefully apply to Records management.

In record keeping, we need to measure two things –

  1. Direct costs – Cleanup costs and the costs of holding until clean up.
  2. Indirect costs incurred because records are no longer reliable.

Cleanup is the cleanest measure. Projects for cleanup work can be estimated with relative certainty as long as you know the information exists.

Indirect costs are obviously harder.

They include the costs of contract renewals missed or poorly enforced because their records are not managed. The generally higher costs of subpoena and information access responses, and generally reduced productivity due to increased search costs for specific records.

Records debt is a useful concept that we can take from software engineering, and a useful barometer that could be presented to executives on a regular basis.

One reason records ends up “the dumping ground” – and how we can avoid it.

Most people can’t define what a record is in the context of what they do.

If they can’t define it, they don’t know what it is.

What they’re left with, is the certain knowledge that there are penalties if they destroy a record, but little idea about what one is.

So they default to safe behaviour.

They keep everything.

And records becomes a dumping ground.

Great records starts with an organisational definition of what records are, that everyone in the organisation can understand and apply, and goes from there.

What happens when Record Keeping and Management is under funded.

Simply, the expense is only ever deferred, the work still has to be done, and doing it badly can be expensive at best, and catastrophic at worst.

The problem for funding record keeping is that the volume of records is directly related to the volume of work that your organisation engages in.

This means that as the volume goes up, the resources required by your records team to deal with the volume will also rise in direct proportion.

When a team is under-funded, the only option is to stop doing work. We can call it “reducing quality”, or use any number of other misnomers, but what we really mean is that we stop doing parts of it.

Parts that are often critical later.

Many of the runaway cost problems that we have with Freedom of Information, and discovery come from under-funding records and information management.

Case in point.

An organisation I work with is required to capture a statutory inspection form once a quarter, and undertake compliance actions if there is a problem noted on it. They receive about twenty five thousand per quarter.

A good quality process here means that the forms are all scanned, and filed with the records of the asset they relate to. This makes specific forms easy to find, and means that costs stay low when there is a problem that should have been picked up, an FOI request is raised, or a subpoena is issued.

The work the records team are expected to do though, has been rising faster than their headcount and funding.

The basic work of ensuring compliance in problem cases is still being done, but the records of inspection aren’t being scanned and associated with the asset anymore. They’re going in a box and being sent off to storage, because the time to do the extra work isn’t available, and something had to go.

The organisation still knows where the forms are are, but any future need for them will require that they’re all brought back from storage. Then it’s a case of searching for four specific, barely legible carbon copied forms among the hundred thousand that make up a year of forms.

That’s expensive.

And it’s likely to happen more than once.

That’s really expensive.

And then there are the Royal Commissions.

“Inadequate records and recordkeeping have contributed to delays in or failures to identify and respond to risks and incidents”

We should all be thinking twice before we under-fund records, or don’t fight to have it adequately funded.

Sentencing is the competency every organisation needs in a post-privacy legislation, breach reporting world.

Sentencing of records is the act of making and applying a decision about their long term handling.

Simply put, it’s about when to destroy them.

While most organisations are amazingly efficient keepers of information, very few outside of government are efficient destroyers of information.

When it’s done well, records that you capture are sentenced by the act of capturing them. This is called sentencing on creation. It means that the process will be largely automatic, and include destruction when the record reaches it’s end of life.

When it’s done badly, a vast trove of data and information will need to be classified and sentenced after the fact. It’s expensive, the chance for error is high, and the trove is what every hacker on earth is hoping they’ll find when they breach your organisation. Then there’s the chance that you’ll destroy information that legislation says you need to keep.

Sentencing is the discipline that every organisation needs to understand in a post privacy legislation, breach reporting world.

If your organisation has a great Records Manager, chances are that you’re already on your way there. If not – now is probably a good time to go and find one.

Records Managers – do you know how long a compliance inspection takes you?

18 months ago, I had a chat to a regulation expert in local government. 

He explained how he thought about the work that he did.

It came down to this:

  1. We know that it takes 6 hours to complete an audit.
  2. We know that we need to do a certain number each year.
  3. We’ve asked for a budget that lets us do slightly more than that.

3 months ago, his council won an award for effectiveness in the area that he regulates.

I thought that they’re questions worth asking about Records practice:

  1. How long does a compliance audit take?
  2. How much time does the remediation/training etc. take?
  3. How many are you likely to need to do?
  4. How much budget do you need to do that?

The quality of Records that your organisation is producing probably depends heavily on whether you’re getting the funding you need for audit.

The two ways to fail an audit, and how to avoid one of them.

There are only actually two reasons to fail an audit:

  1. You’re not doing the right things.
  2. You can’t prove you did the right things.

Doing the right things is obvious.

What is less obvious, is proving that the right things were done.

If you work in a regulated organisation, at some point the auditors are going to show up. When they do, they’re going to ask you for your records. Records are your evidence that you did the right things.

If you don’t have a records management program, chances are, you’re failing at capturing the right evidence, or you’re handling the audit process with a massive overcommitment of resources.

A records management program starts with a great records manager. A great records manager will review your regulations, work out what needs to be captured to provide evidence, and then execute a program to ensure that it is.

Then all you need to do is make sure your organisation does the right things.