android problem - analyzing mobile data usage

dhdurgee · 08-16-2016, 02:53 PM

I have a Jelly Bean 4.1.2 based phone, specifically an LG Intuition VS-950, which I am using with the Page Plus standard plan. As the data rate of $0.10/mb on that plan is ridiculous I have it toggled off and use a Karma Go for mobile data access at a much better data rate. Unfortunately I recently encountered a case where an app or two ran through a lot of my prepaid data. I know one of the culprits and have removed it, but I would like to confirm I have the situation addressed and perhaps seek reimbursement.

If I go into Settings->Mobile Data I see the data usage history. I would like to be able to break this down a bit further, as I only use the Karma Go data part of the time. I would like to be able to isolate the data usage only to the Karma Go for the period of interest. Much of the time my phone is connected either to the office WiFi or a public hotspot, so only the usage while connected to the Karma Go is of interest to me. I have the MAC address of the Karma Go and the two SSIDs used with it, so as long as one of those is part of the data usage history I should be able to isolate it.

Is there an app out there that can do this for me? If not, can I use adb to pull the data of interest over to my laptop for further analysis?

Dave

dhdurgee · 08-19-2016, 06:15 PM

I did a little more searching and from what I can tell the data I am looking for is stored on my phone in /data/system/netstats/ and I downloaded the files in the directory to my laptop. The files are named dev.<startepoch>-<endepoch>, uid.<startepoch>-<endepoch>, uid_tag.<startepoch>-<endepoch> and xt.<startepoch>-<endepoch> respectively with the latest lacking the <endepoch> and are binary files of some sort. How can I work with these files to retrieve the data I am looking for.

Dave

dhdurgee · 08-22-2016, 10:57 AM

I did yet a bit more searching and it appears that the data is in NetworkStatsCollection format. I found a link to the open source here:

https://android.googlesource.com/pla...ollection.java

How can I extract this data so that I can analyze it with other tools?

Dave

nhedgehog · 08-22-2016, 01:48 PM

I'm quite sure that if you were asking at xdadevelopers it would bear more fruit.
I'm not an expert but willing to have a look if you provide the aformentioned logfiles.

dhdurgee · 08-22-2016, 02:26 PM

Quote:

Originally Posted by nhedgehog

I'm quite sure that if you were asking at xdadevelopers it would bear more fruit.
I'm not an expert but willing to have a look if you provide the aformentioned logfiles.

Thanks for your offer. I have posted at xda-developers and received very little response. Attached please find a set of the logfiles for your inspection.

Dave

nhedgehog · 08-22-2016, 02:57 PM

Couldn't read anything out of the files. But I think a solution to your quest could be foune here:
https://source.android.com/devices/t.../netstats.html

dhdurgee · 08-22-2016, 04:48 PM

Quote:

Originally Posted by nhedgehog

Couldn't read anything out of the files. But I think a solution to your quest could be foune here:
https://source.android.com/devices/t.../netstats.html

You are correct, that does decode the data into human readable format and could be used to track what I wanted if it were in the current data. I need to figure out how to get it to dump history data that is still on file fter the current boot.

This at least is a start. Thanks again.

Dave

dhdurgee · 08-23-2016, 05:38 PM

I was able to retrieve the data of interest with the adb sysdump netstats with a few additional options and isolate the usage over the SSIDs of interest. The only issue I have at this point is in interpreting the uids associated with the data. Are uid=-1 and uid=-4 some sort of "special" accounts? How do I get the apps associated with other uids, such as 10004 and 10011 which are the other big users in the time period of interest?

Dave

nhedgehog · 08-24-2016, 03:36 AM

Glad it worked. To your question: I think I saw an app somewhere at Google Play, associating the appnames tothe uid. Can't remember how it was called but it must be somewhere.

dhdurgee · 08-24-2016, 10:08 AM

Quote:

Originally Posted by nhedgehog

Glad it worked. To your question: I think I saw an app somewhere at Google Play, associating the appnames tothe uid. Can't remember how it was called but it must be somewhere.

I found some documentation on this and was able to confirm the information is in the file /data/system/packages.xml on my phone. So I now have the data decoded to the limits of its capture. Some information appears to not be captured, in particular what traffic traversed my OpenVPN tunnel when it was in use. I see usage for the OpenVPN app as an agregate, but the details of which apps were using the tunnel are unavailable.

Dave

08-16-2016, 02:53 PM	#1
dhdurgee Guru Posts: 846 Karma: 2525050 Join Date: Jun 2010 Device: K3W, PW4	android problem - analyzing mobile data usage I have a Jelly Bean 4.1.2 based phone, specifically an LG Intuition VS-950, which I am using with the Page Plus standard plan. As the data rate of $0.10/mb on that plan is ridiculous I have it toggled off and use a Karma Go for mobile data access at a much better data rate. Unfortunately I recently encountered a case where an app or two ran through a lot of my prepaid data. I know one of the culprits and have removed it, but I would like to confirm I have the situation addressed and perhaps seek reimbursement. If I go into Settings->Mobile Data I see the data usage history. I would like to be able to break this down a bit further, as I only use the Karma Go data part of the time. I would like to be able to isolate the data usage only to the Karma Go for the period of interest. Much of the time my phone is connected either to the office WiFi or a public hotspot, so only the usage while connected to the Karma Go is of interest to me. I have the MAC address of the Karma Go and the two SSIDs used with it, so as long as one of those is part of the data usage history I should be able to isolate it. Is there an app out there that can do this for me? If not, can I use adb to pull the data of interest over to my laptop for further analysis? Dave

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Storing plugin data for multi-machine usage	kiwidude	Development	13	06-18-2013 08:13 AM
Touch Nook Simple Touch Battery Usage Data	frankelr	Barnes & Noble NOOK	4	01-27-2012 11:41 AM
Series data usage, and GUI manipulations?	DSchaper	Calibre	1	01-17-2011 09:10 PM
One third ignoring risks of unprotected mobile usage	Colin Dunstan	Lounge	1	11-15-2005 06:52 PM

08-19-2016, 06:15 PM	#2
dhdurgee Guru Posts: 846 Karma: 2525050 Join Date: Jun 2010 Device: K3W, PW4	I did a little more searching and from what I can tell the data I am looking for is stored on my phone in /data/system/netstats/ and I downloaded the files in the directory to my laptop. The files are named dev.<startepoch>-<endepoch>, uid.<startepoch>-<endepoch>, uid_tag.<startepoch>-<endepoch> and xt.<startepoch>-<endepoch> respectively with the latest lacking the <endepoch> and are binary files of some sort. How can I work with these files to retrieve the data I am looking for. Dave

08-22-2016, 10:57 AM	#3
dhdurgee Guru Posts: 846 Karma: 2525050 Join Date: Jun 2010 Device: K3W, PW4	I did yet a bit more searching and it appears that the data is in NetworkStatsCollection format. I found a link to the open source here: https://android.googlesource.com/pla...ollection.java How can I extract this data so that I can analyze it with other tools? Dave

08-22-2016, 01:48 PM	#4
nhedgehog Guru Posts: 771 Karma: 625816 Join Date: Sep 2013 Device: EnergySistemEreaderPro, Nook STG, Pocketbook 622, Bookeen Cybooks ...	I'm quite sure that if you were asking at xdadevelopers it would bear more fruit. I'm not an expert but willing to have a look if you provide the aformentioned logfiles.

08-22-2016, 02:57 PM	#6
nhedgehog Guru Posts: 771 Karma: 625816 Join Date: Sep 2013 Device: EnergySistemEreaderPro, Nook STG, Pocketbook 622, Bookeen Cybooks ...	Couldn't read anything out of the files. But I think a solution to your quest could be foune here: https://source.android.com/devices/t.../netstats.html

08-23-2016, 05:38 PM	#8
dhdurgee Guru Posts: 846 Karma: 2525050 Join Date: Jun 2010 Device: K3W, PW4	I was able to retrieve the data of interest with the adb sysdump netstats with a few additional options and isolate the usage over the SSIDs of interest. The only issue I have at this point is in interpreting the uids associated with the data. Are uid=-1 and uid=-4 some sort of "special" accounts? How do I get the apps associated with other uids, such as 10004 and 10011 which are the other big users in the time period of interest? Dave

08-24-2016, 03:36 AM	#9
nhedgehog Guru Posts: 771 Karma: 625816 Join Date: Sep 2013 Device: EnergySistemEreaderPro, Nook STG, Pocketbook 622, Bookeen Cybooks ...	Glad it worked. To your question: I think I saw an app somewhere at Google Play, associating the appnames tothe uid. Can't remember how it was called but it must be somewhere.