Brian Kemp, Chris Carr, and a Records Management Perspective on that “Nothingburger” Server Wipe
Depending on what news source you read, you might believe that Georgia Attorney General Chris Carr abruptly quit Secretary of State Brian Kemp’s defense in the lawsuit both the SOS’s office and Kennesaw State University are facing over the security of Georgia’s election systems. In some stories, it sounded quite ominous, like Carr had dumped the case because he felt Kemp deserved to lose.
Of course the state deserves to lose the case, but that’s another story. Carr hasn’t waved the white flag. In truth, he’s saddled himself with the harder client. The AG’s office is citing a conflict in representing both the SOS and KSU, so it has outsourced the work of representing the SOS’s office to the law firm of former Governor Roy Barnes. This is not unusual for the AG’s office when there is a possibility that a conflict exists between two state entities. So, what happened?
The most likely cause of the conflict stemmed from a statement from Kemp last Friday, calling a server wipe carried out by KSU “inexcusable conduct or gross incompetence.” He has since reversed course and called the outrage over the wipe “a tasteless nothingburger.“ Clearly, with Kemp’s statement, there is a division there that would have caused the AG’s office to balk at representing both jointly.
Putting on my records management cap, I believe Kemp is right specifically about the data wipe, and it’s because the server was copied and the data retained before the server was re-purposed. Here are the University System of Georgia Records Retention Schedules that cover institutional research records. Note that data must be stored for three years or permanently, depending on whether this is a finalized report or not. There is nothing in the schedule stating that data must be retained in its original format.
Good thing. Servers crash all the time.
Archivists have pretty much settled on the fact that born-digital records (which this would be) have no one original. Rather, every copy is exactly the same, giving them all equal intrinsic value. This is a really, really geeky professional thing to discuss on GeorgiaPol.com, but I believe it’s crucial to understanding why I think the server wipe is a distraction from the larger case of whether or not our voting machines are trustworthy, which y’all may remember, I do not.
About Author
Holly Croft
Holly is an archivist at one of Georgia's institutions of higher learning. In a past life, she was a legislative assistant on Capitol Hill. She cares a lot about records management, open records laws, and privacy laws. Political persuasion? It's complicated. What's not complicated is that she's proudly equal parts Bulldog and Tar Heel.
Add a Comment
You must be logged in to post a comment.
I was among those figuring Carr was leaving a sinking ship. And that’s what he may well be doing, since the AG’s office presumably could’ve just as easily choose to represent the SoS office as KSU. Other than a sinking ship motive for choosing KSU over the SoS? Consider how Carr came to be AG.
It may not be cut and dried with respect to the wipe, either. Others aren’t necessarily returned electronic devices that have information that is under investigation.
As I just noted to Benevolous below, since there is migrated data, iI don’t understand why KSU didn’t turn that over when asked for the server. But then, I don’t know what they were supposed to turn over. If the court order specified the particular sever, then this mess would make a little more sense, I suppose.
I want to try to recap/summarize this, maybe for my own benefit.
– Georgia voting machines have no verifiable paper trail
– A security professional discovers that Georgia’s elections server is not secure (June)
– Mucho evidence of Russia trying to hack into various state election databases
– FBI investigates Georgia’s server
– A lawsuit is filed seeking data from the server specifically regarding June 20th runoff election (July 3)
– The server is wiped (July 7)
– Georgia standards are to keep records for 3 years, although Kemp says they followed “standard IT procedures”
– Memory space is cheap and plentiful
– No one has admitted to who initiated the memory wipe
– AG withdraws from case, no reason give
– It’s the Kemp campaign that says the AG withdrew due to a conflict of interest
Encourage Greg Bluestein to keep up with this story.
[email protected]
Here is the lawsuit:
https://www.documentcloud.org/documents/4118047-Gov-Uscourts-Gand-240678-1-2.html
Mostly correct, except that
1) Memory space is not cheap or plentiful if you follow trusted digital repository standards, which a records manager should. (KSU’s digital archivist would and does.)
2) Kemp has no clue about records management. He’s so incredibly bad at it that the governor TOOK HIS OVERSIGHT OF STATE RECORDS AWAY and blessedly gave it to the Board of Regents. But, he’s (accidentally, most likely) right about the server “wipe.”
3) The data was copied and saved, and the original server it was on was wiped. In archives and records management terms, this is data migration. Because the data was saved, KSU is in compliance with state records schedules that don’t specify that migration or emulation is prohibited. Records management is more concerned with evidential value than intrinsic value, but I don’t think either is affected here.
3) I cannot for the life of me understand why KSU didn’t turn over the migrated data when asked. Instead, we get of all of this nonsense about a wiped server. They brought this whole thing (macro and micro, honestly) on themselves, the dummies.
4) The voting machines are woefully bad. I think the state will lose its case – and should – on the fact that our data is insecure.
Oh, forgot!
Trusted Digital Repository- https://www.crl.edu/archiving-preservation/digital-archives/metrics-assessing-and-certifying/iso16363
Evidential value – https://www.archives.gov/research/alic/reference/archives-resources/appraisal-evidential-values.html
As some one who uses, and at times index and/or transcribes archival records as a part of a hobby, Thank God Clayton College and the BOR now controls the Georgia Archive. I did at lot of letter writing and explaining to my state reps on why Kemp should stop cutting their budget back and limiting access in 2011-2012. I think Buddy Carter become a congress man just to stop getting archive related letters and emails from me.
“– Mucho evidence of Russia trying to hack into various state election databases”
No evidence that they have actually succeeded. Quite the contrary: all available information from the feds and various states indicate that they failed.
“– Georgia standards are to keep records for 3 years, although Kemp says they followed “standard IT procedures””
Huh? Data migration most certainly fits that. I work in IT and data is moved from active storage to long term storage all the time. To put it another way, leaving critical data on active, highly used servers is bad practice. It increases the chances of the data getting corrupted due to server crashes, viruses, software problems or “user error” and it also lowers performance. Migrating old data that is no longer actively being used or accessed but must be kept to archival storage is not only very smart but not doing it is very poor practice, and moreover is mandated by government regulations in a lot of cases (i.e. when it is government data). Dell, HP, IBM and a ton of other companies make a mint off long term storage and there is a huge effort underway to move a lot of that into the cloud (i.e. Amazon S3 and Glacier but that creates a whole other set of concerns). There are government standards that specifically define standards for long term storage of data just like this.
“– Memory space is cheap and plentiful”
No, memory that can easily get hammered a dozen different ways is cheap and plentiful. Memory that meets the requirements for long term and secure storage is extremely expensive and is only provided by enterprise IT companies that are certified to archive government records (in this case).
“– A lawsuit is filed seeking data from the server specifically regarding June 20th runoff election (July 3)”
A lot of progressives seem to believe that the Russians support Republicans generally. They do not. They despise the Clintons due to their foreign policy decisions during the Clinton administration (not long after the fall of the Berlin wall) designed to hem in Russia’s influence over the former Soviet states and other areas that they wanted to get involved in. They like – or at least are willing to exploit or use – Trump for obvious reasons. The current alt right (and by this I mean Breitbart, Gateway Pundit etc and not Richard Spencer or neo-Nazis.) love for Russia began and because Putin aimed racist epithets at Obama around the time of the Sochi Olympics. That was when the first “wow I wish we had a strong leader like Putin instead of Obama running this country” stuff started. Putin has no interest in helping the GOP in general, especially not an establishment (and former suburban Virginia/chamber of commerce/Fulton county liberal) Republican become a back bencher and rubber stamp for the House speaker in Congress, which is Handel’s future.
Also, the polls consistently showed Ossoff losing anyway. (In contrast with the polls that showed Clinton winning and hence being the basis for the claims of Russian hacking.) And Ossoff himself admitted that his only path to victory was getting a general election Democrat turnout COMBINED with a primary/runoff/special election Republican turnout. Didn’t happen: thanks in no small part to the Democrats’ themselves nationalizing the race, the Republicans also turned out in general election numbers in a district where the GOP decidedly outnumbers the Democrats anyway. (As for the Democrats’ hopes that they would benefit from GOP dislike of Trump, they ignored that 1. Trump wasn’t on the ballot but a female Republican who had won statewide office was and oh yes 2. they may dislike Trump but they dislike Democrats more. Do not pretend to be aghast. Democrat primary voters in Democrat enclaves are no different or better.)
From Will Durant below:
“If the integrity of the server’s access is the question then the original hard drive array should never have been repurposed even if we are talking multiple terabyte drives.”
People who have worked in IT departments, ESPECIALLY for government and other highly bureaucratic agencies, disagree. There is a constant need for resources – because, you know, work needs to be done. But getting NEW resources? Please. Either you are on a tight budget, have to go through a lengthy procurement and provisioning process (involving multiple departments who don’t really communicate with each other … did you read the info from the Sony hack that exposed Sony’s ridiculous IT structure? Things. Just. Like. That. And remember: SONY IS PRIMARILY A TECH COMPANY.) So letting valuable resources sit around unused for long periods of time is crazy. You are constantly trying to reclaim and reuse resources whenever you can. Stuff like this is why first virtualization and now the cloud are so important: they provide access to new resources a lot more quickly. But a bunch of drives sitting around whose data that has already been backed up and is part of a system that won’t be used again for some time (elections aren’t a daily or even quarterly thing) are prime candidates for redeployment. If it was KSU, then wow they may have used those drives for fall registration or something who knows.
“– No one has admitted to who initiated the memory wipe”
Because 1. it is nothing to admit to and 2. they probably don’t know. Let me repeat: while “wiping hard drives” has the same connotation as shredding documents from all the crime shows and movies that we have watched, the truth is that both are very common practices. Space and resources are limited and even keeping inventory of that stuff is time consuming, and time is money. So those forms that you fill out whenever you go a government office or doctor’s office? They get scanned into the electronic records system. What happens after that? They get shredded and discarded. What happens to the electronic copies of those forms after their mandatory retention period in the live system passes? They get backed up to storage and the files are overwritten with new data. That’s why you have to pay – and in many cases wait – to get your records pulled from archival. Archive storage is a different system, almost always at a different site. But because formatting and reusing disks is a common they might legitimately have no idea who did it. The servers or storage or whatever may have been marked for reuse after the data was backed up and the person who did it might not have even known what was on it. Also there is likely a group of people with system administrator access and any one of them could have done it. Even if there is user account auditing it is really common for there to many any number of “admin” type accounts where multiple people know the password. While this might strike you as horrifying, the truth is that – again – if the data has been archived already and the drives are marked as eligible for redeployment it really isn’t a big deal. Running the system while it is live, managing the data while it is live, backing up the data = critical. Repurposing resources that are no longer being used and whose data has already been backed up … not so much. You could let a trainee or an intern do it. (In fact dealing with non-critical and offline data is precisely the sort of thing that you would want to use to break in new sysadmins).
As for why the data wasn’t turned over when asked … who asked? They aren’t obligated to turn it over to whoever asks for it and may not even be allowed to. There’s a lawsuit: if the plaintiffs win and there is a resulting court order, then they’ll have to turn it over. The same if the FBI, GBI or SOS wants it, or if there is some sort of audit. If/when that happens and they still can’t come up with the data, you will have something to go on. Until then, it is the sort of stuff that is common to conspiracy web sites – both left wing and right wing – and wishful thinking. Especially when it comes to a very socially liberal Democrat’s election chances in a suburban Georgia Republican enclave. Let’s talk up the chances of an RFRA supporting social conservative winning an election ANYWHERE in Clayton, Fulton or DeKalb before you start thinking that the opposite needs a Russian conspiracy to win in the 6th district.
First of all, I’m sure there are political motivations for some involved and observers, but my comments on this subject just have to do with election integrity.
.
“No evidence that they have actually succeeded. Quite the contrary: all available information from the feds and various states indicate that they failed.”
They would likely have succeeded if they had tried to access this server because it wasn’t secure!
.
I’m not trying to recount the Ossoff/Handel election. It’s just that there is a suspicious sequence of events here that need to be pursued.
.
Your points summarized:
– Russians were trying to hack: Didn’t succeed (speculation). Don’t worry.
– Server data was wiped: Was migrated (speculation), it’s all OK. (Was it migrated? Has anyone said that?)
– It’s OK they wiped the server that was under investigation by the FBI and also the subject of a pending lawsuit because memory is expensive.
– It’s OK they wiped the server because the lawsuit is politically motivated (so therefore whatever evidence was on there is unimportant?).
– Understanding who performed the memory wipe of a server that was under investigation by the FBI and subject of a lawsuit isn’t important because it’s just S.O.P. We always wipe servers right away that are subject to a lawsuit and part of an FBI investigation.
– It’s OK that they didn’t turn over the server data to the court because (apparently) they don’t have to.
.
And one more thing:
“also disclosed that two backup servers were wiped clean Aug. 9, just as the lawsuit moved to federal court.”
https://www.washingtonpost.com/national/apnewsbreak-georgia-election-server-wiped-after-suit-filed/2017/10/26/8fa8c578-ba36-11e7-9b93-b97043e57a22_story.html?utm_term=.cbc655c97fa6
I suppose that is OK too. Why not. S.O.P.
So let’s see; July 3 lawsuit, July 7 server wipe, Aug. 9 backups wipe. What bad luck! Just really an unfortunate coincidence that they keep deleting stuff just as a court might be interested.
Not going to speak to the rest of your post, evergreen, but one correction” You don’t have to win a lawsuit to be entitled to records. As soon as KSU got notice of the suit, they should have instituted proceedings to preserve the info. Even to intervene in S.O.P. if ncssry. I don’t know how they proceeded with the suit, but sometimes those document requests are filed with the suit. But I bet that they will certainly use some of the defenses you mentioned to avoid production of the records. The excuses will sound like “Oh, so-and-so deleted the data without knowing about the lawsuit yet” or “The data was deleted before the Plaintiffs formally asked for it.” Both of those excuses should fail. But it would all be moot if they turn over the migrated data. I bet there’s also an excuse why that can’t be done. There may be a strategy to stall and delay the legal proceedings, if they can, for a whole year, until after elections. This may get real interesting next summer.
Rumor mill up here at the archives confence is that the FBI took a forensic image of the server, and that KSU couldn’t fill the open records request because the FBI has all the copies. I can’t guarantee the accuracy, but it’s the word on the street.
So, if that is true, the 1st question is whether FBI asked for the wipe, or insisted KSU not retain a copy. That could be a reasonable exemption from retention policy.
The Plaintiffs will also no doubt ask the FBI for a copy, but may get a real nothingburger in response.
So KSU said last week that
“Following the notification from the FBI that no data was compromised and the investigation was closed, the server was returned to the University’s Information Technology Services group and securely stored,” the statement said.
The server was taken out of storage after a so-called “after-action” report concluded it could be erased and repurposed, KSU said.”
That doesn’t sound like the FBI asked them to wipe it all, they just didn’t care if they did. Of course it’s not the FBI’s problem that there was also a local lawsuit pending.
Then KSU also wiped both backups a month later.
The lawsuit was filed on July 3, which was a Monday (not a holiday?). The server was wiped that next Friday, the 7th. Wouldn’t they have known about the lawsuit by then?
Reliability of official info from KSU? It starts at the top.
AJC to KSU President Sam Olens on cheerleaders taking a knee:
AJC: Was there any pressure or demands from any individuals and organizations to change the policy regarding cheerleaders at sporting events?
Sam Olens: No.
Holly, you say ‘the server was copied and the data retained’ because that’s what state records management standards require. How can you be sure those protocols were followed? I haven’t read any news report or heard of any statement by a KSU official that would indicate the data has been saved.
The SoS office’s investigation report (headed ‘KSU Deletion of CES Server Data’) says the data hasn’t been lost because ‘indications are’ the FBI saved an image, and ‘all applicable steps’ have been taken by the SoS office to get the FBI to send them to send a copy. That doesn’t sound as if they know about an archived copy either.
You mention that you can’t understand why KSU didn’t immediately turn over the migrated data. Isn’t the easiest explanation that they don’t have it?
Here’s what KSU said last week: https://www.wabe.org/ksu-says-election-server-wiped-fbi-gave-clearance/
They gave the copies to the FBI, so that’s why the FBI has them.
Oops, this is what happens when I skim. The FBI made the copies, not KSU.
We aren’t talking about the NSA’s database of the entire world’s communications here. A record of every box checked, every ballot, along with all of the voter’s individual records should easily fit on a thumb drive. If the integrity of the server’s access is the question then the original hard drive array should never have been repurposed even if we are talking multiple terabyte drives. Brian Kemp’s integrity could easily fit on an 80 byte punch card.
Thumb drives are a terrible way to store data. They go bad within a year or two. I take your point, though: Data like this comes with serious security and management protocols. The Center for Election Security didn’t take that charge seriously, which is why it’s in trouble.
Since the FBI gave clearance for the migration and reimaging, it’s likely they had all the evidence they needed before it was done.
🙂
“Georgia Secretary of State Brian Kemp has started an internal investigation into the server wipeout.”
Let me predict the outcome. A low level employee misread a memo and has been terminated. New procedures have been put into place to ensure that this never happens again.
The investigation is over and the report has been issued.
Thanks for that link.
The investigation concluded that KSU no longer has the info, but “indications” are that the data hasn’t been lost, and it “appears” that KSU might be able to get it back.
The very last sentence in the report, the takeaway, is the conclusion that “the narrative asserted in the media that the data was nefariously deleted and is no longer available is completely false and without merit.”
The only thing that might be true in that statement is “nefariously”. The media was absolutely correct that KSU did delete the data without keeping a copy for itself, and that KSU may not be able to obtain a copy.
Translation for Trump supporters sought by Kemp: Fake news