Who cares?

Terminology: "cross-device"

Questions about "mobile":

  • What about tablets?
  • What about other kinds of devices like TV/Cars/etc?
  • What's the opposite of mobile?

Yesterday

Today

Tomorrow

What are the issues?

  • Variety of form factors.
  • CPUs and networks are slow.
  • New kinds of input (multi-touch).
  • Development with devices.

Supporting multiple devices

Extreme 1: One version to rule them all

Extreme 2: A version for each device

And many many more...

  • HTC One
  • Galaxy Nexus
  • HTC Rezound
  • iPhone
  • Blackberry Bold
  • Blackberry Curve
  • iPad
  • Amazon Kindle
  • Sony W810i
  • Nexus S
  • Lumia 900
  • ... (* browsers)

Seek middle ground

The more versions you create, the better each of them can be from a UX and performance perspective, but the more overall effort it will require.

You want to be between one version and many many versions.

  • Cross-device means both platforms and form factors.
  • Which one can we save on?

Platforms

Human-interface guidelines (HIG) and UI frameworks are great!

Android HIG.

  • Frameworks make it easy.
  • HIGs make it consistent.

Goal: pit of success!

Beware emulating platform guidelines on web

Problems of emulating native UI on the web:

  • Hard to implement.
  • Perpetually slightly off.
  • Doesn't look right on other platforms.

iOS back button implemented using mobile web.

Detailed study trying to do this for iOS: read the article on cheeaun.com.

Platform differences

  • Visual: Different look and feel.
  • Layout: Android and iOS place navigation bar in different places.

Foursquare for Android and iOS with different places of navbar.

Form factor differences

  • Variable usage patterns (one vs two handed).
  • Significant variation in screen real estate.

Devices from small phones to large tablets.

Compromise: phone tablet desktop

Devices of the same form factor in groups.

  • Approach 1: form-factor tweaks to a single version.
  • Approach 2: form-factor specific versions.

Approach 1: One adaptive version

Media query limitations

  • Shared DOM, so making big changes between form factors is hard:

Different kinds of layouts for devices.

  • How to load additional form-factor specific functionality?

Media queries in JavaScript

window.matchMedia lets you evaluate arbitrary media queries:

var result = window.matchMedia('(min-width: 500px)');
result.matches // True iff width > 500px.

You can also listen for changes to match:

function onOrientationChange(mql) {
 var isLandscape = mql.matches;
}
var mql = window.matchMedia("(orientation: landscape)");
mql.addListener(onOrientationChange);

Approach 2: Separate versions

function detectFormFactor() {
  var device = DESKTOP;
  if (hasTouch()) {
    device = isSmall() ? PHONE : TABLET;
  }
  return device;
}

function hasTouch() {
  return Modernizr.touch; // Soon: (hover: 0) and (pointer: coarse)
}
function isSmall() {
  return window.matchMedia('(min-device-width: ???)').matches;
}

(hover: 0) and (pointer: coarse)

Distinguishing tablets from phones

Draw the line at 650px

Devices of varying resolutions squared to take landscape mode into account.

Correctly specifying multiple versions

<head>
  <link rel="alternate" href="http://foo.com" id="desktop"
      media="only screen and (hover: 1) and (pointer: fine)">
  <link rel="alternate" href="http://m.foo.com" id="phone"
      media="only screen and (max-device-width: 650px)">
  <link rel="alternate" href="http://tablet.foo.com" id="tablet"
      media="only screen and (min-device-width: 651px)">
</head>

Read these Google Webmaster smartphone guidelines.

Device.js: formalizing this approach

Use it to automatically redirect to the correct version based on <link rel="alternate"> declarations.

Built-in version forcing, and intelligent redirection.

In action!

For details, visit github.com/borismus/device.js.

On the server

All we have is the User-Agent string.

Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19

...and the user agent string was a complete mess, and near useless, and everyone pretended to be everyone else, and confusion abounded." - Aaron Andersen (How the User Agent got its String)

DevTools Demo.

Device databases

Multiple versions summary

  • Start with device.js.
  • Measure performance overhead (mobile redirects can be expensive: 100-1000ms)
  • If necessary, switch to a server-side UA solution.

Model view controller

Model view controller.

Custom views → code reuse

Model view controller with custom views.

Building a great UI

Single page sites

  • Limited zooming.
  • Fixed headers/footers.
  • Smooth transitions.

Basic viewport

<meta name="viewport" content="...">. At minimum, set width=device-width.

← No viewport
Viewport →

Prevent zooming

Goal: serve page at the right size, eliminating the need to pinch-zoom.

Use initial-scale=1, minimum-scale=1 and maximum-scale=1:

<head>
  <meta name="viewport" 
     content="width=device-width,initial-scale=1,minimum-scale=1,maximum-scale=1">
</head>

Avoid user-scalable=no, since it's not well supported.

Use commas, not semicolons;

Jank-free UIs

Limit of human perception: 60 FPS

Bottom line: utilize hardware acceleration as much as possible.

Example: use position: fixed to keep things fixed to viewport.

Fixed header.

Scrolling sub-elements

For in-element scrolling, overflow: auto;

Tablet GMail for Android.

-webkit-overflow-scrolling: touch; forces hardware acceleration.

In action!

Fast transformations

Goal: animations at 60 FPS.

This is slow: position: absolute; left: 100px; top: 100px;.

This is usually hardware accelerated → fast:
transform: translate3d(100px, 100px, 0);

Others: rotate, scale...

Advanced transforms

3D transformations, and perspective!

matrix3d(m00, m01, m02, m03,
         m10, m11, m12, m13,
         m20, m21, m22, m23,
         m30, m31, m31, m33)

CSS interpolates for you: transition: transform 1s ease-in-out;.

Note: transition, transform still need vendor prefixes!

In action!

Touch input

Touch != mouse

  • No hover state
  • Multiple points
  • Less precise input
  • Fingers aren't (x, y), but also have shape
  • Future: Pressure? Haptic feedback? Hover?

Touch building blocks

Touch events

events: touchstart, touchmove, touchend

properties: touches, targetTouches, changedTouches

There's a specification!

Overriding the browser

Browser has built-in touch behavior (scrolling, tab switching).

How to override?

var el = document.querySelector('#my .selector');
el.addEventListener('touchmove', function(event) {
  event.preventDefault();
});

In IE:

#my .selector {
  -ms-touch-action: none;
}

Some things can't be overridden, such as bevel swipe to switch tabs.

Touch performance

Beware click delays!

The click and mouse* events are delayed by 300ms.

Use touchend for faster buttons.

Advanced touch performance

Multi-touch events come in faster than 60 FPS. Decouple input and draw:

var touches = [];
canvas.addEventListener('touchmove', function(event) {
  touches = event.touches;
}, false);

function draw() {
  // Touch rendering code goes here...
}

// Setup a 60 FPS timer.
var timer = setInterval(function() {
  draw();
}, 1000/60);

Request Animation Frame

In Chrome, use window.webkitRequestAnimationFrame instead (see shim).

function draw(time) {
  // Touch rendering code goes here...
  requestId = window.requestAnimationFrame(draw);
}
function start() {
  requestId = window.requestAnimationFrame(draw);
}
function stop() {
  window.cancelAnimationFrame(requestId);
}
start();

Demo!

Problem: Mouse and touch?

What happens if you want to support both?

el.addEventListener('touchstart', function(e) {
  var touch = e.changedTouches[0];
  onDown(touch.pageX, touch.pageY);
});
el.addEventListener('mousedown', function(e) {
  onDown(e.pageX, e.pageY);
});
function onDown(x, y) { /* ... */ }

Unnecessary boilerplate!

Problem: Gestures are difficult to implement

Unfortunately gesture* events aren't widely implemented.

Robust pinch-zoom based on touch* events:

/**
 * Gesture recognizer for compound multi-touch transformations.
 *
 * 1. pinch/zoom/scale gesture.
 * 2. rotate gesture.
 */

function TransformRecognizer(element) {
  // Reference positions for the start of the transformation.
  this.referencePair = null;

  // Bind touch event handlers to this element.
  element.addEventListener('touchstart', this.touchStartHandler.bind(this));
  element.addEventListener('touchmove', this.touchMoveHandler.bind(this));
  element.addEventListener('touchend', this.touchEndHandler.bind(this));
  this.element = element;

  // Object of callbacks this function provides.
  this.callbacks = {
    rotate: null,
    scale: null
  };

  // Define gesture states.
  this.Gestures = {
    NONE: 0,
    ROTATE: 1,
    SCALE: 2
  };
  // Define thresholds for gestures.
  this.Thresholds = {
    SCALE: 0.2, // percentage difference.
    ROTATION: 5  // degrees.
  };
  // The current gesture of this transformation.
  this.currentGesture = this.Gestures.NONE;
}

/**
 * Touch event handlers.
 */
TransformRecognizer.prototype.touchStartHandler = function(e) {
  var touches = e.touches;
  for (var i = 0; i < touches.length; i++) {
    this.log('identifier: ' + touches[i].identifier);
  }
  // If there are now exactly 2 touches, this is the initial position.
  if (touches.length == 2) {
    // Save these two points as the reference.
    this.referencePair = new TouchPair(touches);
  }
};

TransformRecognizer.prototype.touchMoveHandler = function(e) {
  // Prevent default behavior of scrolling.
  e.preventDefault();
  console.log('current gesture', this.currentGesture);
  var touches = e.touches;
  // Check if there are exactly 2 fingers touching this element.
  if (touches.length == 2) {
    // Get the current touches as a TouchPair.
    var currentPair = new TouchPair(touches);
    // Compute angle and scale differences WRT reference position.
    var angle = currentPair.angleSince(this.referencePair);
    var scale = currentPair.scaleSince(this.referencePair);

    // Check if we're already in a gesture locked state.
    if (this.currentGesture == this.Gestures.NONE) {
      if (angle > this.Thresholds.ROTATION ||
         -angle > this.Thresholds.ROTATION) {
        // If rotated enough, start a rotation.
        this.currentGesture = this.Gestures.ROTATE;
      } else if (scale > 1 + this.Thresholds.SCALE ||
                 scale < 1 - this.Thresholds.SCALE) {
        // Otherwise if scaled enough, start a scaling gesture.
        this.currentGesture = this.Gestures.SCALE;
      }
    }
    var center = currentPair.center()
    // Handle known gestures.
    if (this.currentGesture == this.Gestures.ROTATE) {
      // If we're already rotating, callback with the rotation amount.
      this.callbacks.rotate({
        rotation: angle,
        x: center.x,
        y: center.y
      });
    }
    if (this.currentGesture == this.Gestures.SCALE) {
      // If already scaling, callback with scale amount.
      this.callbacks.scale({
        scale: scale,
        x: center.x,
        y: center.y
      });
    }
  }
};

TransformRecognizer.prototype.touchEndHandler = function(e) {
  var touches = e.touches;
  // If there are less than 2 fingers, reset current gesture.
  if (touches.length < 2) {
    this.currentGesture = this.Gestures.NONE;
  }
};

/**
 * Registers a callback to fire when a pinch occurs.
 */
TransformRecognizer.prototype.onScale = function(callback) {
  this.callbacks.scale = callback;
};

/**
 * Registers a callback to fire when a rotate occurs.
 */
TransformRecognizer.prototype.onRotate = function(callback) {
  this.callbacks.rotate = callback;
};

TransformRecognizer.prototype.log = function(msg) {
  this.element.innerHTML += msg + '
';
};

/**
 * Represents a pair of fingers touching the screen.
 */
function TouchPair(touchList) {
  // Grab the first two touches from the list.
  this.t1 = new Touch(touchList[0].pageX, touchList[0].pageY);
  this.t2 = new Touch(touchList[1].pageX, touchList[1].pageY);
}

/**
 * Given a reference position, calculate how much rotation happened.
 */
TouchPair.prototype.angleSince = function(referencePair) {
  // TODO: handle the edge case of going between 0 and 360.
  // eg. the difference between 355 and 0 is 5.
  return this.angle() - referencePair.angle();
};

/**
 * Given a reference position, calculate the scale multiplier.
 */
TouchPair.prototype.scaleSince = function(referencePair) {
  return this.span() / referencePair.span();
};

/**
 * Calculate the center of this transformation.
 */
TouchPair.prototype.center = function() {
  var x = (this.t1.x + this.t2.x) / 2
  var y = (this.t1.y + this.t2.y) / 2
  return new Touch(x, y);
};

/**
 * Calculate the distance between the two touch points.
 */
TouchPair.prototype.span = function() {
  var dx = this.t1.x - this.t2.x;
  var dy = this.t1.y - this.t2.y;
  return Math.sqrt(dx*dx + dy*dy);
};

/**
 * Calculate the angle (in degrees, 0 < a < 360) between the touch points.
 */
TouchPair.prototype.angle = function() {
  var dx = this.t1.x - this.t2.x;
  var dy = this.t1.y - this.t2.y;
  return Math.atan2(dy, dx) * 180 / Math.PI;
};

function Touch(x, y) {
  this.x = x;
  this.y = y;
}

MSPointer events

Idea: consolidated input model.

Microsoft pointer events diagram.

Problems:

  • Not well-specified yet.
  • Another input mechanism to deal with (now mouse*, touch*, and MSPointer*)

Pointer.js

Idea: consolidate input model across browsers (pointerdown, pointermove, pointerup).

var el = document.querySelector(mySelector);
el.addEventListener('pointerdown', function(event) {
  // ...
});

Basically, abstraction layer around mouse*, touch* and MSPointer* events.

Get original event source was (touch or mouse) via event.pointerType.

In action!

Gestures in Pointer.js

Pointer.js also provides gesture events built on top of pointer events.

var el = document.querySelector(mySelector);
el.addEventListener('gesturescale', function(e) {
  console.log('scale: ' + e.scale);
});

For details, visit github.com/borismus/pointer.js.

Mobile web development

Chrome DevTools ♥ mobile

Easiest path: emulate mobile on desktop.

  • Emulate screen size.
  • Override Chrome's User-Agent string.
  • Emulate (single) touch events.

Emulate multi-touch events on mac

Remote debugging!

Ultimately, you need to test on the device. More info.

Remote debugging.

Thanks for coming!