I’ve been back at my job from my brief stint at the Recurse Center for over a month now and have largely settled back into a regular work week. For those of you who don’t know, I work at Port Zero, a small but efficient security consultancy from Berlin. While my business card says something about “Software Engineer”, these days I audit about as much code as I write for our clients. I figured that a considerable amount of my readers might not know what an audit entails and what to expect from it, especially if it’s focused on security. Let me tell you about it!
The process is a lie
First things first: unless your company is seeking accreditation in some restricted field of tech, chances are that there is either no standard process or that the people who audit you will, for various reasons, not adhere to it. Of course most of us with some experience will look at a standard set of checklists and see if we can spot anything there—e.g. usage of Python’s pickle module on a user-supplied datum—but oftentimes security holes aren’t that simple to find and require the interplay of various different components in your system. These are the times when automated analyses & tools often don’t cut it anymore and someone like me appears on the scene.
In the following I’ll talk a bit about my methods and tools. It’s not magic, just a trained eye and a healthy dose of masochism. The former is required to read through large amounts of code of varying quality and make a ton of notes over prolonged periods of time.1
These magic sausage fingers
I review code in a variety of programming languages, but the process changes surprisingly little on a micro-level. When I scan through lines of code and functions, all languages are created equal.2 The differences come with the architecture, or the interplay of components.
Secure programs are a subset of correct programs. As such, security bugs are a subset of bugs in general, and sometimes a “regular” bug can have security implications. This means that I, as an auditor, am constantly on the lookout for code smells. If something icky is going on, an attacker might be able to use some subtle behaviour of the code to take control of the app.
To illustrate that last point, let’s assume that you maintain a magic web application that herds unicorns. Users upload their unicorns, you assign them an ID number, and the unicorn gets to frolic through the meadows with its new friends. One of your colleagues thinks that 32 bits is too much wasted space for an ID, so they create their own ID class that creates a random number for each unicorn based on its mane color. Sadly they forgot to take uniqueness into account, so sometimes a unicorn gets assigned a number that was already taken. What sounds like a regular bug quickly turns into a security problem when, due to the lack of checks—you didn’t think ID creation could ever fail— your application crashes every time an erroneous ID is produced. If a unicorn-hating hacker or competitor ever finds out that this bug exists, they will send a whole bunch of unicorns with very similarly-looking manes to your service and constantly crash it. This is a very silly transcription of a simple security bug I found at one of my clients—these things happen, and we found it in time and fixed it. Bugs happen. Sometimes they’re benign, but as soon as they’re dependent on user input and reproducible it could turn into a security incident at any moment.
What I want to express with this needlessly long anecdote is that an auditor’s job is similar to that of an editor: identifying problems within the code and trying to suggest how to best get rid of them.
Let’s talk about methodologies. If you know me, you know I’m a relatively relaxed person, and this often translates into my thinking towards software; when I start out manually inspecting a codebase, my method will seem pretty random at first. The chief architect, head of engineering, or lead developer will probably already have explained the architecture of the code base to me and I’ll have a bird’s-eye overview. I’ll then try to poke around in the code, see how the components fit together, and make tons of notes that try to identify the relationship of the modules. I’ll not try to compile or run the code right away, because if I do it the other way around and get something wrong, I’ll know by the time I first run the application. I’d like to think these surprises happen less and less, but I really have no idea, as my data is largely anecdotal and fairly biased. Holding myself accountable is an art I practice, but will most likely never master.
I try to note any code smell, which in my world includes missing or wrongly-formatted documentation,3 because, as has been discussed time and time again by people much more reputable than I’ll ever be, those often lead to subtle bugs.
If my client has automated tests in place I’ll then look at those and try to find cases that were missed, because chances are that they were also missed in the component under scrutiny. A surprisingly large number of my clients has fairly comprehensive test suites; then again, if you invest in having your code audited, you’re probably not in the worst shape anyway.
Depending on the complexity and size of the project and the budget I’m operating with, I’ll also look at historic bugs and see whether I can sense a pattern. This influences the way I inspect current bugs, because if there is a consistent lack of, say, null checks or input sanitizing I’ll watch out for those in particular.
This process can take anywhere from a few hours to a few days, depending on the level of granularity, project, and budget. When this process is over, we move on to the reporting phase.
Blameless blames
Reporting audit results can take many shapes and forms. I usually prefer a combination of a comprehensive document and one or more presentations. If my clients are open to it, I will also have one or more meetings with the developers to try and “workshop” their coding. This helps get rid of bad habits instead of just duct-taping over problems and then going on as before.
Whatever the format is, though, I try to make it as blameless as possible. I have this bad habit of getting to attached to my own code and then feeling bad when someone criticizes it or finds a bug, and I don’t want anyone on the team to feel as if anything is “their fault” just because they wrote a particular piece of code that was buggy. These things happen to the best of us, and I don’t think I’ll ever encounter a bug-free piece of code in my life. We’re working in a continuum of worse and better, but I don’t think there are limits to either side of the spectrum.4
And so, it’s important to make a point of not blaming or fingerpointing and still “telling it as it is”. Our profession is already hard and stressful enough without someone screaming at us because we forgot that one of our parameters is nullable. I’m not perfect, and sometimes it will seem as if I’m being unjust in my assessments; but I try to minimize this source of friction as well as I can.
See you around!
This post didn’t contain as much information about my actual methods as I’d like it to, but I feel it’s too long as it is already. I might come back to this topic in a while and talk about a more specific aspect of my craft, but for now you’ll have to deal with me being a little vague here and there and not really giving you actionable items. The truth is that every customer is different, and a one-size-fits-all solution will fall short for all but one of them; the only thing you achieve by having a set of fixed rules handy for every gig is making your job as an auditor easier, not better.
Appreciating diversity in a business sense—I’m not commenting on the other meaning of that phrase here—has been a fairly successful model for me. I don’t want to waste my or my clients’ time by selling them an ideology. There are plenty of other people who will happily fill that role.
And with that I wish you a great day and hope to see you around soon!
Footnotes
1. You might think it’s more gratifying to look at bad code, since you can add more business value and address more needs that way. That’s not true for me: I enjoy well-written and architected code almost as much as good prose. I actually like my job. I don’t enjoy reading bad code all that much, and standing in front of the development team and talking them through a truck load of bugs they produced is not pleasurable for me—I’m not a monster.
2. This isn’t strictly true: there are some differences between languages that are memory-safe and those that aren’t, because the auditor has to watch out and think through another class of bugs when memory management comes into play. Of course there’s also the garbage collector, but that’s mostly a problem for the perfomance people.
3. Documentation is crucial especially when, during the initial development of an application, someone is writing a partial implementation of a particular component and then forgets to document that. If it isn’t even marked with a searchable marker—such as TODO or FIXME—, I consider this a bug. This marker should be fairly uniform. Of course it’s best to write a ticket detailing the missing functionality, but beggars can’t be choosers.
4. Arguably, defining “worse” and “better” themselves is a sisyphean task, as requirements change and there is never just one dimension. Is an unmaintainable and opaque piece of code that speeds up my application by an order of magnitude “worth it”? Is a pristine, concise, and proven solution to a problem if it means that the end user has to wait 15 seconds for a report to be generated, when we could generate it instantly if we didn’t care about race conditions? Such are the trade-offs we make, and we have to live with them. As always, documentation can at least codify intent, and enable you to tell future maintainers “I thought long and hard about these things, and this is the best I could come up with because of this or that condition”. It will stop them from cursing you and your kin to the seventh generation because they have to stare at your code in fear and trembling instead of getting the job done.