Once Upon a Beta: My NSSA Experience

I had the privilege of being one of the beta testers for NextStage Sentiment Analysis (NSSA) over recent weeks, and this is a total rewrite of my original report of my experience. Doing sentiment analysis on your own writing can be quite a revelation – it can definitely put you in your place. First of all, let me tell you a couple short things about me. I have worked in web analytics for about ten years. I view NextStage’s work as a kind of brave new world (that has such people in it!) that – even with careful and patient guidance – feels just beyond the edge of what I can comprehend. I will tell you honestly that, unlike (probably) other beta testers, I have not experienced a need for sentiment analysis in my job. What I am saying is that my testing approach and my desire to be involved came simply out of analytical curiosity. I didn’t come with a pile of case studies, or anything like that. I am going to do my best here to tell you about my experience. I invite you to ask me any questions that you think I may be able to help with, and I understand if you’d feel more comfortable ignoring this kind of layman-style review.
I am going to start by telling you why this is a total rewrite, even though you’ve probably guessed. I ran sentiment analysis on my first draft and it flagged high for some things I felt were fairly negative, and indicated I would be doing any readers a disservice. I will start with what was not surprising, and explain it to you. I flagged very low for confidence (the scale originally conceived as the BS meter). This is not surprising – as I described above, I am no expert on sentiment analysis (by a LONG shot) and I don’t think I spent enough time being clear about that in my original draft. And despite Joseph Carrabis’s efforts, I always feel a bit of an interloper here in this world of neuroscientific analysis. However, I grasp what my scientific value to the beta effort is: analyst. So I hereby am attempting to provide you with analysis, which should go better. I am sure my confidence will still be low, because that’s my nature, but it shouldn’t be quite so abysmal.
What surprised (and cowed) me was that I also scored very highly for “Retribution” and “Troll”. I see why I scored high on Troll. My tone in the first write was flippant, in a failed effort to project a bit of confidence. I made a few jokes inside, mostly of the variety that would only be funny to me, referencing Dr Seuss and throwing in statements informing you that you could ignore that, etc. Retribution really knocked me over though, and I’ve spent a few days pondering it (without reaching out for help, just as a test). I’ve decided that it was two factors: because of the research I did for my review and my references back to web analytics (which were largely flippant and probably didn’t help my Troll score either). But let me talk about that separately. I hope you’ll stick with me here!
If you’ve read the previous postings on sentiment analysis (I put a list of the ones I found most useful at the end of this posting), you know that in NextStage’s view you need multiple dimensions of data to determine sentiment truly. In preparation for writing this review article, I read a few articles on sentiment analysis. I didn’t do this in a particularly scientific way, I just sort of “turned up” my sensitivity to the term in my regular reading. I took this approach because I wanted to get a feel for what my peers (web analysts) thought sentiment analysis was. Or where it was failing, what was needed, etc. Every article I read talked about scoring a statement (usually a Tweet) as positive, negative, or neutral. Sometimes neutral was omitted. Because I did this “research” after doing my beta testing, you can imagine I had some preconceived notions about why this was inadequate. I likened these people to those that use web analytics only for Hits! (pretty cruel, really) Now, you (or heck, even Joseph) may argue that that can’t cause the retribution flag. That was the best I could come up with after pondering.
Which brings me to a very key point: I had the benefit of Joseph’s help through the beta process. The NSSA results (in current form – I understand plans are to work on this) are really not information that the average analyst can walk up and interpret. I don’t tell you this to trouble you; I am just telling you that the data is nuanced. My first beta interpretation was wrong in so many ways that it could easily fill another post. And, even though I’ve described reports and values individually, they are actually interdependent. For instance, not only was my confidence low, but I scored low on trust and affinity. That’s because I have a very limited idea of who follows Joseph – I know you’re not all web analysts. There’s a good chance some of you think web analytics is total bunk pseudo-analysis! I do not expect that number to improve on this review. I am struggling to give you a clear picture of how the various values interweave, but they truly do, which is a key point. Just like you cannot rely on a single isolated metric or KPI in web analytics, you cannot rely on a single metric in sentiment analysis.
So, to summarize before this gets so long as to be unreadable: the analysis is eerily accurate and eye-opening to say the least. If you write or read, you will probably find yourself in need or at least in want of this tool at some point in your life. I find no evidence out there that there is a comparable tool at your disposal, so when you find yourself needing to know author sentiment (including your own sentiment) you will come back to NSSA.

Thank you for reading,
Jen
Twitter: @jdaysy
Skype: cmjenday

P.S. As promised, the four postings that I leaned heavily on during the beta:

• http://www.bizmediascience.com/2009/06/sentiment_analysis_anyone_part.html
• http://www.bizmediascience.com/2009/06/canoeing_with_stephane_sentime.html
• http://triquatrotritecale.hungrypeasant.com/?p=25
• http://triquatrotritecale.hungrypeasant.com/?p=40