Where is your team’s center of gravity?

As more workers push for remote work, we’ve begun trying to find a common vocabulary to classify how well an organization supports its remote workers.  There’s the idea of “remote-friendly” vs. “remote-first” — how do your teams tools and processes enable your remote teammates to succeed?  The problem is that these terms are very subjective – it’s hard to assess yourself well with this standard.

I have worked on teams with various remote and distributed configurations, and there’s one factor that seems to determine whether remote employees can succeed — I’m calling it the “center of gravity.”  Where is your team’s center of gravity?  Let me show you a few examples.

The nominally “remote-friendly” team

If the majority of the team is in the office, the center of gravity is in the office.

You’ve decided to be remote-friendly, which means letting a few of your good people move back to their hometown, or closer to family, or just stop coming into the office.  But with the majority of the team still in the office, your center of gravity is in the office.  This sets you up for failure.  The majority of communication is going to happen in the office over a cubicle wall, or over coffee.  When you have a meeting, everyone in the office huddles in a conference room around one shared microphone while the remote employees struggle to hear, and can’t keep up.  This type of “remote-friendly” is anything but.

The majority-remote team

Another straightforward example: with most of the team working remotely, your center of gravity is now in remote-land.  Your default modes of communication move to remote-friendly tools. Instead of conversations over the cubicle, they’re happening in Slack.  Instead of the conference-room-plus-crummy-audio situation above, everyone calls in from their own desk, setting a level playing field. 

They may have lost the center of gravity, but people in the office aren’t put into a weaker position, relative to remote workers.  They have the same access to the same tools as the remote workers.  Everyone can contribute, everyone can be heard.

This works well, but a lot of companies think they can’t get any work done with most of their workers out of the office. There is, however, another configuration I’ve seen, and the results may surprise you.

The distributed team

This is a distributed team.  It has the same amount of remote workers as the first example (the in-office center of gravity) except now the in-office contingent is split up among several different offices.  Our visual metaphor breaks down a bit here, because the new “center” just depends on how we arrange the faces. So instead of taking the top-down view of the team, let’s draw that same team from any one worker’s point of view:

Even though the majority of people are in an office, from the perspective of any individual teammate, the majority of the team is somewhere else.  This forces the team to adopt remote-first tools and processes, even with a mostly in-office team.

This theory is based on my own observations, which are admittedly a small sample size.  Have you seen something different? Tell me about it.

Thanks to my colleague Mike Rooney for suggesting an improvement to this post.

Save time and reduce risk with Gradle’s includeGroup

With the recently released version 5.1, Gradle has added a great, subtle new feature that lets you specify which dependencies should be pulled from which repositories.  To explain what this is, let’s start with the default behavior. In your Gradle file, you probably have multiple repositories defined, like this:

When Gradle needs to find a dependency, it will search each of those repositories, in the order they are declared.  So when it goes to download, for example, [com.android.support.constraint:constraint-layout:1.1.3], it will first check the Google repo, which has that artifact, so it’s done.  But then let’s say you want RxJava: [io.reactivex.rxjava2:rxjava:2.1.9]. Gradle checks for it in the Google repo, but Google responds with a HTTP 404 error, so Gradle moves on to the next repository, which is JCenter.  And on and on, for each dependency in your build.

This can lead to a couple of problems:

  • There’s a performance problem.  Since you have to check each repository in order for each dependency, there are a lot of requests that return a 404, and you waste time and resources.  Wouldn’t it be nice if we could tell Gradle “oh, I know RxJava is on JCenter, so don’t bother checking the Google repository”?
  • If a repository that’s first in the list gives a bad response (like the time JCenter responded to Google artifacts with an HTTP 409 error). Gradle will give up, and not check other repositories.  This will break your build, and leads to a lot of advice like “make sure you list the Google repository first!”
  • You’re vulnerable to a spoofing attack.  If you have a dependency that’s in your last repository (fabric in the example above), but a malicious actor uploads a library with the same group and artifact names to JCenter, for example, Gradle will download the JAR from JCenter since it’s higher in the list, and you’ll never know the difference.  JCenter doesn’t do much to verify that you are who you say you are, so this type of attack is a real risk.

So how do we resolve this?  Gradle 5.1 adds a new API so you can specify which groups to include or exclude in a repository.  A quick note about what I mean by groups: it’s the bit before the first colon in a Gradle coordinate.

So in practice, your build.gradle might now look something like this – note that you can match groups exactly or with a regular expression:

Now, when Gradle goes to download constraint layout, it will match the regex on the google repo, and Gradle will never attempt to download it from another repository.  Likewise, even though the Google maven repository is listed first, Gradle won’t attempt to download RxJava from it, because it’s not listed in the include groups.

If you want to test this, try running a Gradle task with “–refresh-dependencies”, which will force Gradle to try to download all of your dependencies again.  If you get an error like this one then you know you still need to work on your configuration.

> Could not resolve all files for configuration ':app:debugCompileClasspath'.
   > Could not find io.reactivex.rxjava2:rxjava:2.2.2.

The important thing to know about includes and excludes is that the behavior is defined per repository.

  • If you list an include – Gradle will only try to download the included groups for this repository.
  • If you list an exclude – Gradle will try any groups except the excluded groups for this repository.
  • If you list includes and excludes – Gradle will download only groups that are included and not excluded.

There’s one non-obvious thing that I want to really emphasize, though:  If you specify includeGroups on repo but don’t specify any groups on a second repo, like the screenshot below, Gradle will still try to download com.google artifacts from both of these repositories.  The options you declare for one repository don’t affect other repositories.

This leads to my last, but perhaps most important piece of advice: whitelist every group.  The best way I can see to use this is to include every group in the appropriate repository, and make sure every listed repository has an includeGroup declared.  This will force Gradle to download each dependency from the right repository only.

What’s the point of Lockdown Mode in Android P?

Lockdown Mode is a new feature in Android P — it disables fingerprint login, forcing you to enter your PIN/passcode to unlock the device.  This is an important, but subtle, distinction. Ultimately, what’s the difference between your passcode and your fingerprint?

The difference comes down to legal precedents in the US.  Legally, the police can compel you to unlock a device using biometrics (e.g. face or fingerprint).  They can’t, however, force you to unlock the device using your passcode. This is due to the Fifth Amendment to the US constitution — you can’t be compelled to testify against yourself.  Courts have said that revealing your passcode is equivalent to providing testimony against yourself — but using your body to unlock the phone isn’t legally the same thing. I’m not going to try to explain the underlying legal theories in depth, but if you want to read more, check out these articles:

There are a lot more details and complications that I’m glossing over, but if you were curious about why Android bothered to create a new mode that forced one type of login while disabling another — I think this is why they did it.

Disclaimer: I’m not a lawyer and this isn’t legal advice — if police action is in your threat model, you need to talk to a real lawyer.

Yes, you should run ProGuard / R8 on open-source library dependencies

I have seen a few people argue that there’s no reason to let ProGuard run on the open-source libraries that they include in their app, because if the source is publicly available, there’s no point of obfuscating, right? Who are you fooling? Here’s an example from one library’s README file:

Their ProGuard rules file is just a big wildcard keep:


I think that’s exactly wrong, for two reasons — shrinking, and obfuscation.

Shrinking

The majority of the shrinking that ProGuard performs is by removing unused code from third-party libraries. You’re not writing a lot of code that you don’t use, but you’re probably not using all of the features in the third-party libraries that you include. If you use a

-keep com.example.** { *; }

-type rule on open source libraries, you won’t get the benefits from ProGuard’s shrinking. None of that unused code will be removed.

Obfuscation

I’m going to show this with an illustration. What does this class do?

The eagle-eyed among you may recognize a couple of clues and have a hunch, but I doubt you know everything that it’s doing. So let’s look at the same class, but without obfuscating the third-party libraries.


It’s a lot easier to figure out what this code does when the libraries aren’t obfuscated. Simply put, obfuscating the open-source code allows for better obfuscation of your own your code.
(I mention ProGuard by name, because it’s still the main tool for Android development, but these arguments apply just the same to R8.)

Quick-and-dirty flat-file storage

Not every app needs cloud storage with a SQLite cache.

I’m going to let you in on a little secret that helps me prototype apps quickly.  Sometimes, when nobody’s looking, I just write my data to disk. I don’t build a cloud backend.  I don’t build a SQLite database. I just serialize my data to JSON and write it to files on the disk.  It’s really easy.

There are definitely some drawbacks to this approach, so if your head is about to light on fire with anger towards me, skip to the Caveats section below.

The code

Here’s what it takes; it’s just a couple of lines of Kotlin.  Say I’ve got a FooConfiguration object that I want to persist to disk.  It’s complicated enough that I don’t want to put it in SharedPreferences, so we’re going to write it to a file.

Step 1: We have a couple of dependencies to bring in.

Step 2: we serialize the object to JSON and write it to disk.

Step 3: When we need to read that data, we read from disk and deserialize from JSON.

That’s it.  Persistent storage, done.

Caveats

Like I mentioned at the beginning, this is something I do to get an app idea off the ground quickly.  When you go to scale this up, you’re going to run into drawbacks.

  • Sync & conflicts – If you try to sync this storage over a cloud file store (e.g. Dropbox, Google Drive, etc), or build some import/export functionality, you’re quickly going to run into this question — How do I handle conflicts? This is a hard problem, and the best advice is to build your storage model such that somebody else handles that complexity.
  • Data model evolution – If you change your data model, you need to ensure that it’s backwards-compatible, or else you’ll lose data from previously-stored objects.
  • Security – storing your data to local disk may not make sense, depending on the sensitivity of the data.  This is highly dependent on your specific needs, but keep it in mind.
  • Local-only – The only copy of the user’s data is on their device.  If you want to build any kind of social app where users interact with each other’s data, this isn’t going to get the job done.

My point here is, this isn’t a perfect solution, and it may be downright bad for your app.  Don’t take my advice blindly.

Ship It

I have multiple apps in the Play Store using this method right now.  No, I’m not going to tell you which ones. My point is, if you’re working on an app but you’re getting stuck on building a backend, this is a way to punt on the complexity of your persistent storage.  Build it quick-and-dirty for now, so you can focus on the app; revisit a more scalable solution for storage when you’re ready.

 

Best Practices for Enabling ProGuard

I’ve written several blog posts recently about technical aspects of using ProGuard to obfuscate your code in Android apps.  Then, in March, I gave a talk at DroidCon Boston walking through an overview of How ProGuard Works. Through all of those, I tried to keep the content very matter-of-fact and technical.  I got a lot of technical questions in response, but I also got a lot of questions about the organizational and software engineering aspects — how to think about maintenance, and testing, and debugging, and communicating with your team.  So what follows are the less technical, but possibly more practical, guidelines I’ve found for working with ProGuard.

You’ll find that most of these tips are focused on the fact that obfuscation can expose problems that are only seen at runtime, and then only when a specific code path is executed.  This means that most of these rules are focused on making sure you’re likely to discover those problems.

As early as possible

If possible, start using ProGuard from day 1 of a new project.  I’ve gone through the experience of enabling ProGuard on a project that’s been worked on for years without it, and it’s more difficult.  You’ll get a warning about some code, and you’ll find that nobody on the team remembers that code.

Enabling ProGuard from day 1 ensures that issues are encountered immediately, when the relevant code is fresh in your mind.

As many builds as possible

Don’t enable ProGuard in your day-to-day dev builds*, it will increase your build times a little and cause you extra pain in debugging.  But for every other build — QA, beta, release — turn ProGuard on. You want as many testers as possible exercising the code as many ways as possible to ensure that you notice any ProGuard bugs early on.

*There is an exception to this, which is when you get a report of a bug that you suspect might be triggered by obfuscation, you want to enable ProGuard in your dev environment so you can easily reproduce the bug.  This means that you want to have a single, easy, flag in your build.gradle that you can flip to true to enable ProGuard on your debug builds. Have all of the configuration files ready for that build type — the same configuration files you’re using for the rest of your builds — you want the only effort on your part to be flipping the one flag.

The Phased Approach

If you’ve inherited a codebase that is already in production, you can’t enable ProGuard from day 1, but you can try to fake it a bit.  Start by enabling ProGuard in your dev builds and fixing the compile-time errors. Then, run a ProGuard-ed build and exercise the app as much as possible.  Walk through each feature as well as you can, and try to find problems. Next, enable ProGuard for your QA builds, and wait for a bit. This is the real key.

You’ll give your QA team time to work through the app and find problems.  You want to make sure they’ve had time to exercise the app before you move on to the next phase, which is a beta build, if you have one.  A beta should hopefully expose your obfuscated app to a broad, but still small, set of users. Let that work for a bit, then you can enable for your public builds.

Make your Keep rules as specific as possible

I’ve mentioned this in my blog post and in the talk, but it bears repeating.  There’s no value in using ProGuard if you’re going to kneecap it with overly-broad keep rules.

Document every keep rule

This is deceptively important.  It’s tempting to just add the keep rule you need to a config file and be done, but you’re hurting yourself in the long run.  A year from now, you’ll look at the rule and wonder why it was there. It might be referring to code that’s long gone, or a library that’s no longer used.  Or it could be critically important. You don’t know, because the rule itself doesn’t tell you the reason it was added. And, perhaps more importantly, it doesn’t tell you how to test whether it’s still necessary.

Add a comment.  Explain what was broken and why that rule fixed it.

Conclusion

That’s it.  Try it out.  If you’ve got some different ideas, based on your experience, I would love to hear it.

How ProGuard Works – Links

Thanks for watching my talk, How ProGuard Works, at Droidcon Boston or Android Summit. Here are some of the links that I mentioned in my slides, as well as links to some of my blog posts including the diagrams I used.

R8:

My posts:

Gradle snippet for forcing a particular version of ProGuard:

buildscript {
  ...
  configurations.all {
    resolutionStrategy {
      force 'net.sf.proguard:proguard-base:6.0.1'
    }
  }
}

Feeding ProGuard’s inputs: where are all of these rules coming from?

When ProGuard runs on your Android app, it’s not only using the rules that you’ve explicitly added, but rules from a couple of other sources as well.  Below are four sources to watch out for:

Rules explicitly added by the developer

These are the rules in a file that you add with `proguardFile` in your build.gradle.  If you’ve used ProGuard at all, you’re probably accustomed to adding configuration rules here.

build.gradle snippet showing the proguardFiles method

Default ProGuard file

A base set of widely-applicable rules get added by that “getDefaultProguardFile” line above, which you’ll see in the template build.gradle file.  It protects things like Parcelables, view getters and setters (to enable animations on view properties) and anything with the @Keep annotation.  You should walk through this file to see what ProGuard rules get added by default.  It’s also a great source of example rules when you’re trying to make your own.

snippet of example rules from the default Android ProGuard file

AAPT generated rules

When AAPT processes the resources in your app, it generates rules to protect, for example, View classes that are referenced from layout files, and Activities referenced in your manifest. Each rule comes with one or more comments saying where the class was referenced, so you can track down the offending resource if need be.  You can find these rules in:

build/intermediates/proguard-rules/{buildType}/aapt_rules.txt.
snippet of example ProGuard rules generated by AAPT
snippet of example ProGuard rules generated by AAPT

Note that this source isn’t explicitly added in your build.gradle, the Android Gradle Plugin takes care of including these rules for you.

Rules from libraries

If an Android library has ProGuard configs that apps should use when they include that library, the library can declare those rules with the consumerProguardFile method in its build.gradle.  For example, here’s a snippet from Butterknife’s build.gradle.

snippet of Butterknife’s build.gradle showing the consumerProguardFiles method

 

Bringing it all together

Rather than trying to track down every single source of ProGuard config that’s getting added to your app, you can look at them all together.  You can specify

-printconfiguration proguard-config.txt

in your rules file and ProGuard will print out all of the configuration rules to the specified file.  Note that they’re not in the same order that they were originally specified, so it can be hard to figure out where a particular rule is coming from.  

Reading ProGuard’s Outputs

When ProGuard processes an Android app, it generates a few output files to show what happened in each step. These files can be really helpful if you’re trying to figure out what ProGuard changed, or why your code ended up the way it did. But those files aren’t self-documenting, so I’m going to walk you through why each of those files is created and what it shows you.

These files are in the build directory, something like:

app/build/outputs/mapping/{buildType}/

Here’s a diagram I made with a really high-level overview of the steps ProGuard takes when it’s analyzing your app, because the output files line up nicely with these steps.  Refer back to this for some context around the following steps.

seeds.txt

The first things ProGuard does is to read all of your configuration files, and then read in all of the Java bytecode (.class files) to create what it calls the class pool.  ProGuard then looks through the class pool and prints to seeds.txt a list of every class and member that matches any of your keep rules.  This is useful for debugging if the keep rule you wrote actually matches the class you’re trying to keep.

If it’s a class that matches, there will be a line with just the fully-qualified name of the class.  For a member, it will be the fully-qualified class name, followed by a colon, followed by the member’s signature.

usage.txt

Knowing what code it has to keep, ProGuard then goes through the class pool and finds code that it doesn’t need to keep.  This is the shrinking phase, where ProGuard strips out unused code from the app.  As it’s doing this, it prints out unused code — code that’s being removed — to usage.txt.  Now this name seems backwards to me; I think it should be unused.txt or shrinkage.txt or something, but that’s just me.  

This is useful if you’re trying to figure out why a class doesn’t exist at runtime.  You can check whether it got removed here, or got renamed in the next step.

If an entire class is removed, you’ll get a line with the fully-qualified class name.  If only certain members of a class are removed, you get the class name followed by a colon, and then a line (indented with four spaces) for each member that was removed.

mapping.txt

The next thing ProGuard needs to do is obfuscate as much code as possible — that is, it’s going to rename classes and members to meaningless names like “a”, “b”, etc.  As it’s doing this, ProGuard prints the old name and new name for every class and member to mapping.txt.  Not all code is renamed, but all code is listed in mapping.txt.

This is the file you need if you’re trying to de-obfuscate a stacktrace.  It allows you to work backwards from the obfuscated names to the original names of code.

Each line is of the form “{old name} -> {new name}”.  You get a line for the class name, then a line for each member of the class.  Note that constructors are shown as “<init>()”.  

dump.txt

After ProGuard has done all of its magic (shrinking and obfuscating), it prints out one last file which is essentially a full listing of all the code after processing.  That is, everything that’s left in the class files, but in a less optimized format, so it’s a huge file.  I have a demo app that I use for testing ProGuard stuff, and the final app is about 1 MB, but the dump.txt is almost 18 MB.  It’s enormous. Here’s the output for a trivial class:

This can be really useful, though, if you want to see what’s in your class files but don’t want to decompile the .class or .dex files.

Archiving

One last note is that these files are important artifacts of your build — especially mapping.txt.  If this is a build you’re going to distribute (say on the Play Store, or even internally for testing), you’re going to need your mapping.txt to de-obfuscate stacktraces.