How to analyze and get statistics for Shoutcast streaming logfiles using the open-source logfile statistical analysis tool AWStats.

This document is all about using AWStats to provide logfile usage and analysis for Shoutcast streaming. It's last update is with:

• aswtats v7.3 (build 20140126)
• Shoutcast logfile v1.9.8

Background

Everyone wants it; nobody has it. Streaming Stats for their Shoutcast logfiles, that is. Thus this document on what I've found out. In short, AWStats as a streaming logfile analyzer will provide some info, but not really everything a streamer wants to know.

A local non-profit non-commercial community-funded and -run radio station I work with had been wanting better analysis of their streaming stats, which is provided via a Shoutcast server. The server creates 2 sets of logs - connections and w3c style logfiles. Unfortunately, seemingly few tools exist, but I did find thousands of requests, especially for an open-source solution. I found the quite outdated Webalizer Streaming Version (circa 2002, which also required compiling). The two commercially available solutions are CasterStats and Sawmill, both somewhat expensive.

So, lacking funding and not finding anything, I shelved the project. Until I decided to revisit this in late 2014 with a vengeance. 

I finally ran across some documentation on using AWStats with some adjustments and such that looked promising. I also found some similar info for Webalizer and analog, both quite outdated, but haven't tried those yet. Only AWStats has been kept updated.

Installation/Configuration

The setup I used is one that runs a daily cron job to download the latest stream logfile, adds the method, then parses it into the database to ready for viewing.

Download AWStats and unzip into your webroot directory somewhere. (I deleted the docs directory, but kept the remaining as somewhat useful).

In the cgi-bin directory, copy awstats.model.conf to awstats.conf and configure it to work for your website. Key settings for this solution are:

• LogFile="../logs/nonespecified.log" (has to be set, but you will feed the logfile to it via command line)
• LogType=S
• LogFormat="%host %other %time2 %url %code %ua %bytesd %extra1 %extra2 %method"
• SiteDomain="your.domain.com"
• HostAliases="127.0.0.1"
• DNSLookup="1" (if you want IPs changed to FQDNs)
• DirData="/path/to/db" (doesn't need to be in webroot, but does need to be accessible by awstats script)
• DirIcons="../icon"

Note: in other configuration settings I turned off useless analyzing: Referrer, Robots, Search Engine, Keywords, Filetypes, Worms, and set browser detection to allphones and OS detection to 2 (highest). I also turned off the useless reports: Authenticated Users, Robots, Worm, Email, Session, Pages, Filetype, Filesizes, Screensize, Origin (Referrer), Keyphrase, Keywords, Misc, Errors, Cluster. I adjusted the number of lines shown on various charts.

You'll also need your logfile(s) available somewhere (doesn't need to be in webroot).

The commands I used to update the dB: /path/to/awstats/cgi-bin/awstats.pl -config=/path/to/awstats/cgi-bin/awstats.conf -LogFile=/path/to/logs/sc_serv_w3c.log -update 

Sources: 

• AWStats Documentation 
• awstats requires a METHOD in the logfiles, thus you need to add that for it to do its work. I found a post in the Winamp forums for such a perl script called sc_parse.pl.
• Common Logfile formats (from analog)
• AWStats Extensions and How To - also better location detection instructions
• Installing a public instance of AWStats (how I added a 2nd instance to a server that already had awstats installed for it.)
• Country and State detection plugin (replace geoipfree with GeoIP) (Download database Updates)
• Understanding AWStats Statistics 

So, I downloadedd the latest AWStats (stable), which now has a setting for streaming logs (FileType). However, AWStats will not read the raw Shoutcast W3C files because there is no method there and you will get an error. You need to use the sc_parse.pl script (mentioned above) to add the GET method to your logfile. Then it will read and add your logs to the database.

After much testing, configuration, 

I ended up adding the GeoIP plugins for country, city, ASN (their freebies) and the timezone plugin (to get stats to match our timezone). These required perl modules to be installed: GeoIP::PureIP and 

I tried using the ExtraSection capabilities, but couldn't get it to do what I wanted - durations.

Conclusions

Unfortunately, AWStats can be somewhat lacking used as a streaming stats toolset. The stats show when folks INITIATED their listening session, not something that shows duration of listener, a number which exists in the logs. AWStats does their duration based upon when the next page visit occurred (within a 60-minute window). We can't view accumulated listening hours on individual days, to better see which shows are well listened to.

In the end, you'll want a better toolsetup like the CasterStats or Sawmill solutions which seem to offer up better statistics on what streaming providers will actually want to know. But you can get some solid info otherwise. Almost makes one want to spend some time on developing AWStats to do that better, perhaps as a plug-in.