Enhanced Statistics for Android Pie and Later

Android continues to throw wrenches into how GSam Battery Monitor gathers statistics.  That said, the changes are for the most part good from a security perspective, and Google has made it possible to continue to access much of the data - it just requires some re-work.

Standard vs Enhanced Stats

Standard:  For standard statistics (what you get if you don't run any ADB commands), GSam tracks things like bluetooth, phone, network, and screen usage using standard Android interfaces.  These all require GSam to be running at all times as a service in the background.  GSam is pretty efficient, so should have negligible impact on the battery.

Enhanced:  In enhanced mode, the user needs to provide GSam with access to the BATTERY_STATS permission.  This allows GSam to access a wealth of data that Android collects.  In KitKat, Android removed the ability for apps to request access to this permission.  However in Lollipop and beyond, they partially restored the ability by allowing the user to explicitly grant an app access via ADB.  This requires the user to connect their phone to a computer, install ADB, and run a command.  Doing so is worth the effort in terms of the stats you get, however I don't expect most users to do this!

Enhanced Changes in Android P

As of GSam version 3.39, GSam will use a combination of Standard and Enhanced mode to gather statistics.  That is, for system wide stats, it uses the Standard mode, and for App specific stats (visible in the App Sucker screens), GSam will use the Enhanced mode.

The reason for this is complex.  

1. Google requires all apps to have a Target SDK version of n or n-1 in order to publish an update to the Play Store.  In other words, if the latest version of Android is Q, you must have a target SDK version of P or later.  This means that Version 3.39 of GSam requires updating the Target SDK to P.  Doing so has consequences.
2. Android has started restricting access to non-public interfaces.  Unfortunately, GSam relies heavily on non-public interfaces and a technique called reflection.  Android allowed older apps to continue to access the APIs so as not to break things, but any updated app must conform to the new rules.  This all makes good sense - it just breaks GSam :).

Fortunately, Android is providing a similar level of access to battery statistics for applications via the HealthStats API.  So GSam 3.39 is written to take advantage of these formal interfaces for access to all app specific statistics.  These still require access to the BATTERY_STATS permission, but otherwise provide (almost) the same level of detail (see below for limitations and workaround).

Limitations For Enhanced Stats in P and Later (see below for workaround)

System Wide Stats:

1. Screen Brightness Levels:  We don't track this due to the performance impact tracking it would have (the brightness changes too often).
2. Phone Radio Signal Levels:  We don't track this for the same reason we don't track screen brightness.

Application Stats (under App Usage / App Sucker):

1. Multiple Profiles - such as personal vs work profile:  Android has the ability to have multiple profiles.  Often your work apps are running in a work profile separately from those apps in your personal profile.  You'll likely notice you may have to install the same app in both profiles.  For security reasons, GSam can't gather information on applications running outside of the profile in which it's running for security reasons.  So - if you need details on the app usage in your work profile, you'll need to install another instance of GSam in your work profile.  Yes, this is a pretty big limitation if you have work profiles (fortunately most users don't) - see below for a workaround.
2. CPU Stepping:  The power that the CPU uses can't be determined quite as accurately.  Previously, GSam would look at how much time the app spent running at different CPU frequencies (which all draw different levels of power).  Now, we have to collapse all the frequencies into one bucket and average them out.
3. Kernel Wakelocks:  There is no longer a way to gather kernel wakelocks.  

Workaround - Restore Full Enhanced Stats

Fortunately, it IS possible to restore fully enhanced stats, even on P and Q by allowing GSam to access what Android has deemed hidden apis.  It requires running another ADB command, and more importantly, is not specific to GSam.  In short, it not only allows GSam to access hidden APIs, but allows ANY app to access the hidden APIs.

Is this a security concern?  Yes - sort of.  That said, older apps can still access these APIs, but it is something Google is trying to ween people from.   From a security perspective, it allows GSam to access statistics on apps that are outside of its profile.

On Android Q and later, run the following ADB command:

adb shell settings put global hidden_api_policy 1

On Android P, run the following ADB command:

adb shell settings put global hidden_api_policy_p_apps 1

Finally, restart GSam (force close it).