Menu

Nicer NPEs in Android L

KitKat + dalvikvm:

 java.lang.NullPointerException
  at com.roundsapp.HomeActivity.onCreate(HomeActivity.java:61)
  at android.app.Activity.performCreate(Activity.java:5312)

Android L + ART:

java.lang.NullPointerException: Attempt to invoke virtual method 'void android.app.ActionBar.setTitle(java.lang.CharSequence)' on a null object reference  
  at com.roundsapp.HomeActivity.onCreate(HomeActivity.java:61)
  at android.app.Activity.performCreate(Activity.java:5720)
  ...

I expect this to be extremely handy for those lines of code that involve chain several calls.

Why doesn't desktop Java have this?

Gson 2.3 and why you want it

When Gson binding fails, it gives you a crappy exception that tells you where in the document things went sideways:

com.google.gson.JsonSyntaxException: Expected a boolean but was NUMBER at line 1 column 4728  
    at com.google.gson.stream.JsonReader.nextBoolean()
    at com.google.gson.TypeAdapters$3.read()
    at com.google.gson.TypeAdapters$3.read()
    at com.google.gson.ReflectiveTypeAdapter$1.read()
    at com.google.gson.ReflectiveTypeAdapter$Adapter.read()
    at com.google.gson.ReflectiveTypeAdapter$Adapter.read()
    at com.google.gson.ReflectiveTypeAdapter$1.read()
    at com.google.gson.ReflectiveTypeAdapter$Adapter.read()
    ...

With Gson 2.3, you get a much nicer exception. It tells you the JSONPath to the failure. In this case, $.history[0].change_status.deletions:

com.google.gson.JsonSyntaxException: Expected a boolean but was NUMBER at line 1 column 4728 path $.history[0].change_status.deletions  
    at com.google.gson.stream.JsonReader.nextBoolean()
    at com.google.gson.TypeAdapters$3.read()
    at com.google.gson.TypeAdapters$3.read()
    at com.google.gson.ReflectiveTypeAdapter$1.read()
    at com.google.gson.ReflectiveTypeAdapter$Adapter.read()
    at com.google.gson.ReflectiveTypeAdapter$Adapter.read()
    at com.google.gson.ReflectiveTypeAdapter$1.read()
    at com.google.gson.ReflectiveTypeAdapter$Adapter.read()
    ...

Get Gson 2.3 from the

String Constants

Brent Simmons:

Here’s my secret, or maybe my superpower, or maybe just me being lucky — I’ve never had a bug related to using a string literal when I should have used a constant.

But I have had bugs when I used the wrong string constant.

I'm the same.

Holo vs. Material Design: Colors

Holo's color palette is opinionated, but limited. Five hues each in a primary and darker shade for a total of ten colors. (There's swatches for lighter tints, but these are rarely used.)

Material Design's color palette is the opposite: 19 hues, 16 of which have accent variants. With all tints including white and black, the palette is a neat 256 colors. With so many colors, Material Design is not a uniform, but a system. You might have six blue apps, with each one in a different shade of blue. The apps that use the system will be vibrant and colorful.

Both of my own apps (Shush & Rounds) use Holo's palette for a native look on Android 4.0. I didn't worry about colors: I just let the platform pick for me. Even Rounds’ icon uses the five Holo colors.

With Material Design in Android 5.0 (“The L Release”), I'm back in the driver’s seat. I'm okay with that. Though it might mean I'll need to redraw Rounds’ icon!

Comments should be sentences

Ned Batchelder:

Paying more attention to the comments will help you write better code. I can't tell you how many times I've written what I thought was a perfecty good function or line of code, then gone to write the comment or docstring, and realized a better way to do it, or even just a better name.

We follow this practice at Square and I'm very happy with it. Plus, if somebody forgets punctuation we get to joke, “You missed your period!” which is fun and silly.

Roboto, old and new

Today Google released a revised Roboto. Their design site says,

It is slightly wider and rounder, giving it greater clarity and making it more optimistic.

In Blue, Roboto from Android 4.4 (download .zip)
In Red, Roboto from “The L Release” (download .zip)

To my eyes they're quite a bit different. The new Roboto dots its i's and exclamation marks with circles; the old one with squares.

I'm not switching to FlatBuffers

On the Android Developers blog, Wouter van Oortmerssen writes:

Game developers, we've just released FlatBuffers, a C++ serialization library that allows you to read data without unpacking or allocating additional memory, as an open source project. [...] Use this library to store game data with less overhead than alternative solutions (e.g. Protocol Buffers or JSON).

The Java generated for FlatBuffers is ugly. For example, use the schema language to define an enum with flat buffers:

enum Color:byte { Red = 0, Green, Blue = 2 }  

That yields generated Java that isn't an enum!

public class Color {  
  public static final byte Red = 0;
  public static final byte Green = 1;
  public static final byte Blue = 2;
};

Worse, the format is not memory-safe. On Hacker News, Wouter writes:

Most readers of binary file formats can be made to read memory outside the buffer by corrupting the data, and FlatBuffers is no different.

That said, an option to bounds-check every offset would be possible, at a certain cost. Might be a nice optional feature to have.

Skipping bounds checks is what caused the Heartbleed bug. It's possible the Google servers using FlatBuffers may suffer the same problem!

FlatBuffers isn't ready. I'm going to stick with JSON

OkHttp ate MimeCraft

Last year, during Square's 7 days of open source, we announced Mime Craft, which makes it easy to build browser-style HTTP request bodies.

With this week's OkHttp 2 release candidate, we've promoted that MimeCraft code into OkHttp itself. You can encode a form request:

RequestBody formBody = new FormEncodingBuilder()  
  .add("search", "Jurassic Park")
  .build();
Request request = new Request.Builder()  
  .url("https://en.wikipedia.org/w/index.php")
  .post(formBody)
  .build();

Response response = client.newCall(request).execute();  

Or a multipart request:

RequestBody requestBody = new MultipartBuilder()  
  .type(MultipartBuilder.FORM)
  .addPart(
    Headers.of("Content-Disposition", "form-data; name=title"),
    RequestBody.create(null, "Logo"))
  .addPart(
    Headers.of("Content-Disposition", "form-data; name=image"),
    RequestBody.create(MEDIA_TYPE_PNG, new File("logo.png")))
  .build();

Request request = new Request.Builder()  
  .header("Authorization", "Client-ID " + IMGUR_CLIENT_ID)
  .url("https://api.imgur.com/3/image")
  .post(requestBody)
  .build();

Response response = client.newCall(request).execute();  

It's quite satisfying to build a multipart RequestBody from other RequestBody values. Yo dawg.

Complete examples are on the OkHttp recipes page.

Pinning SSL Certificates

I was planning two TLS code samples for OkHttp's recipes page. One to make development easier by disabling certificate checks and another to make deployment more secure by pinning TLS certificates.

Disabling Certificate Checks

It's a hassle to get TLS going on a local development server. Disabling certificate checking is unfortunate, but productive. The most important part about disabling security in development is remembering to re-enable it for production! And to test that yes, it really is secure in production.

Pinning Certificates

Disabling certificate checks trades generally-unnecessary security for developer productivity. Pinning certificates is the opposite: spend more time setting it up, and get even more security.

The extra security you gain is defense against rogue certificate authorities. For example, in 2011 DigiNotar signed fraudulent certificates for google.com, leaving web users vulnerable to man-in-the-middle attacks.

I wrote up a fun little program that captures the server's SSL certificate and generates copy-and-pasteable code with your pinned certificate:

  /**
   * Pinned certificate chain for google.com.
   * Expires Tue Aug 19 20:00:00 EDT 2014.
   */
  public static final SSLContext SSL_CONTEXT_GOOGLE_COM_EXPIRES_20140819 
      = new SslContextBuilder()
      // CN=*.google.com, O=Google Inc, L=Mountain View, ST=California
      .addCertificate(""
          + "M2DNSYlXEjlHxLfN23DIk7xnThH4RTk4bhhtWtBTyR9G5aOCBBkwgg"
          + "MIIGgTCCBWmgAwIBAgIIoSeucwDQYJKoZIhvcNAwSTELMAkGA1UEBh"
          + ...
          + "FdDynG494MhWpmLdY5xzOWqb7+xmPdo3947O2Mg

Javadocs and Code Samples

OkHttp has a bunch of docs:

The recipes are new. The contrast between recipes and Javadocs is interesting:

Javadocs show you how to call a method: what parameters are legal and what return values to expect.

And the recipe shows you why.

OkHttp 2.0 RC1

Adrian, Jake and I have been working on OkHttp 2.0 quite actively for nearly a year, and we're finally ready to share its new API with the world:

OkHttpClient client = new OkHttpClient();  
Request request = new Request.Builder()  
    .url("https://square.com/")
    .build();
Response response = client.newCall(request).execute();  
System.out.println(response.body().string());  

The 2.0 API leverages fluent builders and immutability to make HTTP easy. It can be used synchronously (above) or asynchronously:

OkHttpClient client = new OkHttpClient();  
Request request = new Request.Builder()  
    .url("https://square.com")
    .build();
client.newCall(request).enqueue(new Callback() {  
  @Override public void onResponse(Response response)
      throws IOException {
    System.out.println(response.body().string());
  }

  @Override public void onFailure(Request r, Throwable e) {
    ...
  }
});

It shares the same sophisticated backend as our HttpURLConnection API which moved to a new okhttp-urlconnection artifact.

OkHttp 2.0 is not backwards-compatible. The changelog describes what's changed and what's gone. To make upgrading easier, we're also releasing OkHttp 1.6 which is the 1.5 code plus some new 2.0 APIs. Use 1.6 to transition to 2.x APIs.

Today we're publishing a release-candidate with the goal of a final release in June. Between now and then we're going

The JSON benchmarks are wrong

Claiming “We’re the fastest!” is a great way to grow a project’s popularity, particularly for mechanical tasks like JSON parsing. This month there's been plenty of comparison between Jackson and Boon.

Boon author Rick Hightower:

“Boon JSON parser and the new Groovy 2.3 JSON parser are faster than Jackson.”

I contribute to Gson which occasionally gets mentioned. But Gson lags behind Jackson and Boon in benchmarks.

But benchmarks don't tell the whole story.

They measure best-case or common-case scenarios. But both Jackson and Boon are vulnerable to hash denial of service attacks, and behave extremely poorly in the worst case.

I have a 1.2 MiB JSON file with 100,000 keys that all share a common hash code. Gson parses this file in 150 milliseconds. But Boon and Jackson take 100x longer: over 15 seconds to parse the same file. Hash collision performance is exponential, so doubling the file size quadruples parse time for Boon and Jackson.

An attacker could tie up all of your servers with just a few megabytes of malicious data. If you use current releases of Boon or Jackson, you’re vulnerable to DoS attacks.

Unfortunately, the benchmark race focuses on top

Finding, Killing Flaky Tests

Flaky tests are the worst. They break the continuous build, the presubmit checks and confidence in the test suite.

Fixing flaky tests is unpleasant because you need a way to reliably reproduce something unreliable. And a way to confirm that you've fixed it. I prefer brute force:

  @Test public void pushPromiseStream() {
    // This test is flaky.
    ...
  }

@Test public void pushPromiseStream1000x() {
    // This test fails quite reliably!
    for (int i = 0; i < 1000; i++) {
      System.out.println(i);
      SpdyConnectionTest test = new SpdyConnectionTest();
      test.setUp();
      test.pushPromiseStream();
      test.tearDown();
    }
  }

The looping test finds the problem. I fix it. Then I can delete the looping test which has served its purpose.

There's probably better ways to do this, but this approach works for me.

Embed binary data in your tests with Okio

Here's some code that uses Okio to serialize an object to a base64-encoded string:

String toBase64(Object object) throws Exception {  
  Buffer buffer = new Buffer();
  ObjectOutputStream out = new ObjectOutputStream(buffer.outputStream());
  out.writeObject(object);
  out.close();
  return buffer.readByteString().base64();
}

And this code will deserialize that string back into an object:

<T> T fromBase64(String base64) throws Exception {  
  Buffer buffer = new Buffer();
  buffer.write(ByteString.decodeBase64(base64));
  ObjectInputStream in = new ObjectInputStream(buffer.inputStream());
  Object result = in.readObject();
  in.close();
  return (T) result;
}

With these two methods you can take a serialization snapshot of an object running under one version of your code and confirm that it'll deserialize under another.

For example, you can System.out a BloomFilter created with Guava v16:

@Test public void printSerializedObject() throws Exception {
  BloomFilter<Integer> bloomFilter = ...;
  bloomFilter.put(5);
  System.out.println(toBase64(bloomFilter));
}

Copy and paste the printed base64 into another test and confirm that it deserializes correctly under Guava v17:

@Test public void deserialize() throws Exception {
  String goldenValue = "rO0ABXNyAC1jb20uZ29vZ2xlLmNvbW1vbi5oYXNoLkJsb29tRmlsd"
      + "GVyJFNlcmlhbEZvcm0AAAAAAAAAAQIABEkAEG51bUhhc2hGdW5jdGlvbnNbAARkYXRhd"
      + "AACW0pMAAZmdW5uZWx0AB9MY29tL2dvb2dsZS9jb21tb24vaGFzaC9GdW5uZWw7TAAIc"
      + "3RyYXRlZ3l0AC1MY29tL2dvb2dsZS9jb21tb24vaGFzaC9CbG9vbUZpbHRlciRTdHJhd"
      + "GVneTt4cAAAAAV1cgACW0p4IAS1ErF1kwIAAHhwAAAAAYAAAAAAAAkkfnIAIm9raW8uQ"
      + "mxvb21GaWx0ZXJUZXN0JEludGVnZXJGdW5uZWwAAAAAAAAAABIAAHhyAA5qYXZhLmxhb"
      + "mcuRW51bQAAAAAAAAAAEgAAeHB0AAhJTlNUQU5DRX5yACxjb20uZ29vZ2xlLmNvbW1vb"
      + "i5oYXNoLkJsb29tRmlsdGVyU3RyYXRlZ2llcwAAAAAAAAAAEgAAeHEAfgAIdAARTVVST"
      + "VVSMTI4X01JVFpfNjQ=";
  BloomFilter<Integer> bloomFilter = fromBase64(goldenValue);
  assertTrue(bloomFilter.mightContain(5));
}

With base64 it's easy to embed binary data right in the test.

This

Sense 6

I've been on stock Android since before the T-Mobile G1 was released. The new HTC One (M8) is the first time I've experienced the joys/horrors of a third-party skin in my day-to-day phone. I was anxious, especially after seeing Touchwiz on my wife's Note 3.

Sense 6's built-in apps are both more attractive and more usable than their peers on Nexus KitKat. I'm thrilled to avoid the broken time picker in the nexus Clock app and the unusable radial-configuration of the default Camera app. HTC's quick settings is customizable so I could give Wi-Fi hotspot easy access.

Sense 6 is Android refined. I highly recommend it.

2014-04-14

Okio Timeouts and Deadlines

It was six weeks from when I teased an Okio code sample, to releasing the project publicly. In that time I went back and forth over and over on how to do timeouts.

The lightbulb came when I discovered that we were addressing two different problems:

  • Detecting network partitions. That way we can recover from common I/O problems like bad WiFi connections, crashed modems, and unresponsive servers.
  • Limit how much time is spent on a task. We might want to preload images without wasting too much battery. 

In the first scenario, any connectivity is good connectivity. When we receive just a single byte we know that the remote peer is alive and reachable. If we don't see any activity on the other end for a long time, we can assume the worst and terminate the connection.

The second scenario applies even if the network is operating perfectly. If we exceed the time budgeted for the task we cancel the task.

Okio implements both of these strategies, referred to in the API as timeouts and deadlines respectively. You can even use Okio to set up write timeouts for blocking sockets:

    Socket socket = new Socket("square.com", 90);

    BufferedSink sink = Okio

Basic HTTP/1.1 with Okio

In a post on The Corner on Wednesday I claimed that OkHttp is one of the most challenging projects I've worked on. Here's the proof that I was lying: an HTTP request in 12 lines of code.

    Socket socket = new Socket("square.com", 80);

    BufferedSink sink = Okio.buffer(Okio.sink(socket));
    sink.timeout().timeout(5, TimeUnit.SECONDS);
    sink.writeUtf8("GET / HTTP/1.1\r\n");
    sink.writeUtf8("Host: square.com\r\n");
    sink.writeUtf8("\r\n");
    sink.flush();

    BufferedSource source = Okio.buffer(Okio.source(socket));
    source.timeout().timeout(5, TimeUnit.SECONDS);
    for (String line; (line = source.readUtf8LineStrict()) != null; ) {
      System.out.println(line);
    }

One neat method in this code sample is readUtf8LineStrict(). This is like BufferedReader.readLine(), except it throws an EOFException if the input is exhausted before the \n is encountered. That way the parser won't silently ignore truncated input.

There's another method, BufferedSource.readUtf8Line(), that behaves exactly like BufferedReader.readLine(). That's the one you want to use for human-edited text. Unlike computers, humans can't be trusted to include trailing newlines, and they shouldn't be punished for omitting them.

Get Okio on GitHub.

10 things you can do with Okio's ByteString

    // Decode base64.
    ByteString a = ByteString.decodeBase64("bWFwbGUgc3lydXA=");

    // Encode base64.
    String b = a.base64();

    // Decode hex.
    ByteString c = ByteString.decodeHex("776166666c6573");

    // Encode hex.
    String d = c.hex();

    // Read from an input stream.
    ByteString e = ByteString.read(new FileInputStream("foo.txt"), 50);

    // Write to an output stream.
    a.write(new FileOutputStream("bar.bin"));

    // Encode UTF-8.
    ByteString f = ByteString.encodeUtf8("petit déjeuner");

    // Decode UTF-8.
    String g = a.utf8();

    // Use a byte string as a Map key (uses equals() and hashCode().)
    Map<ByteString, String> map = new LinkedHashMap<>();
    map.put(a, "decoded base64");
    map.put(c, "decoded hex");
    map.put(e, "read an input stream");

    // Copy the bytes into an array.
    byte[] h = a.toByteArray();

The entire jar is less than 100 KiB so you shouldn't hesitate to include it wherever you need it.

Get Okio on GitHub.

Satisfying improvements

My post on The Corner today:

“The most satisfying improvements are the ones that simultaneously speed up the code, simplify the implementation, and improve the API. In this post I'll describe an ugly situation that led to one of these solutions.”

The computer you wear

  • Smart watch: the computer you wear
  • Smart phone: the computer in your pocket
  • Tablet: the computer for your coffee table
  • Laptop: the computer you take to work
  • Desktop: the stationary computer

I don't use my tablet because the niche between phone and laptop isn't big enough. I think the niche for the smart watch is similarly small.

I guess we'll find out!

OkBuffer Preview

I spent my weekend hacking on OkBuffer, an unbounded byte buffer that reads at the front and writes at the back. Unlike NIO buffers, you never need to flip this thing!

Edit Bookmarklet

Makes any webpage instantly editable. Use it to redact private information before taking a screenshot. Or to preview what new text will look like in a UI.

Or for pranks.

javascript:document.body.contentEditable = 'true'; document.designMode='on'; void 0  

3 highlights and 1 lowlight of Google's Java Style Guide

Google employs thousands of Java programmers that collectively maintain many millions of lines of Java code. So they need a comprehensive style guide to quickly resolve any dispute. It's a brilliant document that succinctly and unambiguously describes how to write stylish Java code .

Highlight: UTF-8 encoding

Googlers aren't limited to ASCII in their source files. It's 2014 after all.

    String unitAbbrev = "μs";

Highlight: But what is a constant, exactly?

The guide explains the difference between static fields and constants, ending the LOGGER vs. logger lumberjack fight.

Highlight: Expect the exceptional

Testing what happens when your code throws? No need for much ceremony; just catch expected and carry on.

    try {
      emptyStack.pop();
      fail();
    } catch (NoSuchElementException expected) {
    }

Lowlight: Mandatory braces

You must equip all of your control flow with curly braces. The following is taboo:

    if (empty() == 0) throw new EOFException();
    if (byteCount == 0) return 0;

At Google, Java programmers must spread each check over three lines:

    if (empty() == 0) {
      throw new EOFException();
    }
    if (byteCount == 0) {
      return 0;
    }

All that extra scrolling will make your wrists hurt!

Overall it's a great doc. Satisfying this guide is certainly sufficient for the open source projects I maintain at Square.

The Tick

To a human the computer's clock is fluid, constantly running through imperceptibly-brief milliseconds. But to the CPU, clock changes are rare events. You can run lots and lots of code between clock ticks.

High-precision timers tick more frequently, but they don't solve the problem. How many distinct timestamps should I observe when running this code?

    long[] timestamps = new long[1000];
    for (int i = 0; i < timestamps.length; i++) {
      timestamps[i] = System.nanoTime();
    }
    for (long timestamp : timestamps) {
      System.out.println(timestamp);
    }

If you guessed 1000, you guessed wrong. I saw 75 distinct timestamps on my MacBook Pro. The output stair-steps, staying on one value for several observations and then jumping to another. Here's a sample of the output:

  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026068000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026069000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000
  1381031468026070000

On this particular Mac and JVM, the source of System.nanoTime() has microsecond precision. That's typical, and the method's Javadoc doesn't hide it:

"This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change."

The

Waterloo's new startup neighbourhood has a parking problem

Waterloo's new startup neighbourhood has a parking problem. Anthony Reinhart argues that it's time to break with car culture:

"Free and cheap parking – and huge parking lots, often with empty spaces – were everywhere, even downtown. People drove everywhere, even just a few blocks, often alone in their cars. And still they felt there wasn’t enough parking if they had to park more than a minute’s walk from their destination."

Driving to work takes me 12 minutes, biking is 20 minutes and the bus is at least 36 minutes. I should bike more.

View from the ‘Loo: The problem with parking — Communitech