Pebble Video Player

2017-10-27 20:54:20 Views: 1833

Over the past few days, as a form of procrastination, I've implemented a video player for the (obsolete) Pebble smartwatch.

After posting this video to reddit, I had a number of people calling for code and/or a technical write-up. My response was, invariably, "Yes!" - if I release my code, then somebody might fix it!

As a 'write up', I'll document the stages that my prototype went through - including where it is now, which is beyond my original Proof of Concept video.

Iteration 1 - How fast can I force images on to the Pebble?

With games such as Tiny Bird for the Pebble in mind, I knew that the intrinsic Pebble refresh rate would not be the problem - it'd be physically transferring the images over bluetooth from the phone.

To get started, I cloned the 'pebble-faces' git repository. It's an excellent example of loading PNG images from a server on to the Pebble without a companion app. The process is:

  1. Pebble sends AppMessage to Phone with request URL
  2. Javascript on phone grabs image data from URL
  3. Phone sends image data back to Pebble in as large chunks as possible
  4. Pebble displays image

This process works great for individual images, but to try and stream a video I'd need something faster.

First, I refactored the Pebble code so it only requested an image once - this would act as a 'start stream' command. Then, I changed the JS so instead of loading 1 image then returning, it immediately started loading the next frame when it was done transmitting it to the Pebble (we'll see later, this is a place where some serious optimisation can be done). In this iteration, steps 3 and 4 are basically the same. 

Once I did all that, and used a handy script to automatically convert a video to the required images, I posted the above video as an initial Proof of Concept. The frame rates frequently dropped below 1FPS, with significant stuttering and lagging - I knew it could be better!

Iteration 2 - Buffering

Pebbles don't have much memory, but they do have some, and I needed to use it. What's more, in v1 the app only loads the next frame from the server after the current one is already displayed on the Pebble - this is inexplicably inefficient.

Inspired by my university's Programming in C course, I made the (inevitably bad) decision to implement a doubly-linked list on the Pebble to store a frame buffer of bitmaps loaded from the phone. 

struct llnode {
  GBitmap *bmp;
  struct llnode *nxt; //Next frame
  struct llnode *prv; //Previous frame
typedef struct llnode ll;

struct deq {
  unsigned int len;
  ll *fst; //First Element
  ll *lst; //Last Element
} *fbuff;

To my surprise, this worked! And my FPS (by this point I had implemented an FPS counter) went up from ~1 to around ~3, and up to ~6 on black and white content! That's incredible pay-offs for a relatively simple fix, but there was still more to do.

Next, I did a similar thing on the JS side: why wait for the transmission to be completed before downloading the next frame? Instead, I gave the JS a buffer of downloaded frames too, and let it send them off continuously - effectively splitting in to two different execution threads (does the Pebble JS interpreter use multithreading?). This got me to the 7-8 FPS on black and white content, which I believe is the limit without some fundamentally different tricks.

Next, register the middle button as play/pause, limit the FPS to something achievable consistently, output some videos in that FPS, and you have something not half bad:

This is the current state of affairs.

Iteration 3 - Inter-frame compression?

A fundamentally different trick - it should be possible to only transmit changes between some keyframes, utilising PNG transparency. This is what I've been (kind of) working on now, between school work. I rewrote the watchapp so it uses the layer_set_update_proc instead of a generic bitmap_layer, which should hopefully make it easier to keep previous images even after they've been removed from memory (or maybe not?).

I've also been struggling to generate the PNG images I'd need - if anyone can help with this, please do!

What's Next?

There's a bunch of things to consider: what's the final end goal of the project? Youtube viewer? GIF viewer? If this was Pebble heyday, I would've made it like tertiary text, very easy to include into other projects that want to use it. Whatever it is, it'll probably need a neater pebble interface - including play/pause icons and a menu to select content.

In addition, I already have tech debt: my linked-list frame buffer is implemented badly and seems to be leaking memory. I need a fixed-size buffer (~10 frames is probably enough) to prevent heap fragmentation.

I'd like to open this project up to whoever's interested!

Project Source

On Github