AutoPhone is a brilliant platform for running automated tests on physical mobile devices.
:bc maintains an AutoPhone instance running startup performance tests (aka “S1/S2 tests” or “throbber start/stop tests”) on a small farm of test phones; those tests run against all of our Firefox for Android builds and results are reported to PhoneDash, available for viewing at http://phonedash.mozilla.org/.
I have used phonedash.mozilla.org for a long time now, and reported regressions in bugs and in my monthly “Performance Check-up” posts, but I have never looked under the covers or tried to use AutoPhone myself — until this week.
All things considered, it is surprisingly easy to set up your own AutoPhone instance and run your own tests. You might want to do this to reproduce phonedash.mozilla.org results on your own computer, or to check for regressions on a feature before check-in.
Here’s what I did to run my own AutoPhone instance running S1/S2 tests against mozilla-inbound builds:
git clone https://github.com/mozilla/autophone
pip install -r requirements.txt
Install PhoneDash, to store and view results:
git clone https://github.com/markrcote/phonedash
Create a phonedash settings file, phonedash/server/settings.cfg with content:
python server.py <ip address of your computer>
It will log status messages to the console. Watch that for any errors, and to get a better understanding of what’s happening.
Prepare your device:
Connect your Android phone or tablet to your computer by USB. Multiple devices may be connected. Each device must be rooted. Check that you can see your devices with adb devices — and note the serial number(s) (see devices.ini below).
Configure your device:
cp devices.ini.example devices.ini
Edit devices.ini, changing the serial numbers to your device serial numbers and the device names to something meaningful to you. Here’s my simple devices.ini for one device I called “gbrown”:
cp autophone.ini.example autophone.ini
Edit autophone.ini to make it your own. Most of the defaults are fine; here is mine:
#clear_cache = False
#ipaddr = …
#port = 28001
#cachefile = autophone_cache.json
#logfile = autophone.log
loglevel = DEBUG
test_path = tests/manifest.ini
#emailcfg = email.ini
enable_pulse = True
enable_unittests = False
#cache_dir = builds
#override_build_dir = None
repos = mozilla-inbound
#buildtypes = opt
#build_cache_port = 28008
verbose = True
#build_cache_size = 20
#build_cache_expires = 7
#device_ready_retry_wait = 20
#device_ready_retry_attempts = 3
#device_battery_min = 90
#device_battery_max = 95
#phone_retry_limit = 2
#phone_retry_wait = 15
#phone_max_reboots = 3
#phone_ping_interval = 15
#phone_command_queue_timeout = 10
#phone_command_queue_timeout = 1
#phone_crash_window = 30
#phone_crash_limit = 5
python autophone.py -h provides help on options, which are analogues of the autophone.ini settings.
Configure your tests:
Notice that autophone.ini has a test path of tests/manifest.ini. By default, tests/manifest.ini is configured for S1/S2 tests — it points to configs/s1s2_settings.ini. We need to set up that file:
cp s1s2_settings.ini.example s1s2_settings.ini
Edit s1s2_settings.ini to make it your own. Here’s mine:
#source = files/
#dest = /mnt/sdcard/tests/autophone/s1test/
#profile = /data/local/tmp/profile
# test locations can be empty to specify a local
# path on the device or can be a url to specify
# a web server.
remote = http://192.168.0.82:8080/
blank = blank.html
twitter = twitter.html
iterations = 2
resulturl = http://192.168.0.82:8080/api/s1s2/
Be sure to set the resulturl to match your PhoneDash instance.
If running local tests, copy your test files (like blank.html above) to the files directory. If runnng remote tests, be sure that your test files are served from the resulturl (if using PhoneDash, copy to the html directory).
python autophone.py –config autophone.ini
With these settings, autophone will listen for new builds on mozilla-inbound, and start tests on your device(s) for each one. You should start to see your device reboot, then Firefox will be installed and startup tests will run. As more builds complete on mozilla-inbound, more tests will run.
autophone.py will print some diagnostics to the console, but much more detail is available in autophone.log — watch that to see what’s happening.
Check your phonedash instance for results — visit http://<ip address of your computer>:8080. At first this won’t have any data, but as autophone runs tests, you’ll start to see results. Here’s my instance after a few hours: