Using Apples new TestFlight Beta Testing

... which is not to be confused with the old TestFlight (which Apple bought and called TestFlight).

At work, we do a pilot release of our app before we go live. This is usually part way thru regression testing, and obviously before we release to the general public. Up to now, we have used TestFlight (the old one) to manage the iOS side of this.

It's worked fine - well, it's worked fairly well, it's never been an overly reliable service - but we are constantly running into the 100 device limit, after we load up all of our test devices, a few developer devices, the business users devices for the pilot, and of course, a lot of people have got new devices since the last roll over....

Enter stage right the new, improved TestFlight Beta Testing.

So whats different?

• It's integrated into iTunes Connect now.
• It's supported by Apple, for better or worse.
• It has a proper client, and proper integration into iOS8 (only).
• In some cases, it requires approval / review before release. (not as bad as it sounds)

The steps to use it are pretty simple, and some of them can be automated, at least for internal testing.

They do introduce some new terms tho.

• Internal testers are people who are on your Developer Account who need to test the app. Think people on your direct team, your developers, testers, and maybe product managers. You have have 25 of these and each person can have 10 devices.
• External testers are normal people who have an Apple ID (the one they use for the AppStore, not iCloud, tho they are often the same). Up to 1000 of these, and no limit on devices, but a single person can't beta more than 10 apps at a time. They do not have to be part of your organisation.

So, how do you do it? Fairly easily.

First, make an AppStore build and upload it as normal. I suggest you set the Build Number to something which you can change each time (I used 3.4 for the version number, and 3.4.1, 3.4.2 etc for the build numbers). It will appear in iTunes Connect in the Prerelease > Builds section. You can then turn on TestFlight Beta Testing and submit it for review.

For me, review took about 12 hours. If you are just releasing it to Internal Testers, you do not require review, and there is a command line tool to update (and I'd assume, release) it for internal use. I suspect we'll be using this as soon as we move to Xcode 6.

Once it's approved, you can then go into external testers (or internal) and add some people^[1]. You need their name and email address for this. Once you have them in there, they get an email which tells them to go get the TestFlight app from the appstore, and after they install it, the same link will take them to the app in TestFlight and let them install or update it. It mostly works like the Appstore from there.

NOTE: If they use the Gmail app and have Google Chrome installed, it will take them to the AppStore every time. Solution is to take the URL it has in Chrome, and paste it into Safari - then it works right.

After they have downloaded your app and installed it, you can see who's installed what in the External Testers list. I've found this to be somewhat inaccurate (eg, in this case, AJ had also installed it), but a good indication.

The user, however, can stop being part of the beta from within the TestFlight app, and get to a few bits of your website (urls provided), but it would be nice if there was away to jump directly to an email to the developer. Tap, get a new message with a predefined subject and some beta metadata in the body. Maybe in version 2.

If you have a new build to push up after that, you can just upload it as normal (with a new build number!), look for it in Builds, and submit it for review again. It asks you if you have made major changes, and if you say no, then it's ready to go and be tested.

It's absolutely an improvement over the old system, but it's still got some kinks to work out - mostly timing issues, and things not updating. Still better than the old system, except for one thing: It's iOS8 only for the testers. I think they could have made the effort to have it on iOS7, which would cover 99% of devices out there, and made the break for broken UDID-based testing once and for all.

So far, we've run into a few problems notifying people - eg, I have 5 testers, I add a new build, tell it to give it to everyone - and only 3 of the 5 get it. No way to "reenable" the other ones, I have to remove and re-add them (which spits out 2 emails). They can't even go into the app and manually refresh - it just shows the old version. Frustrating at best. Consider it a nice Alpha.