Bad Behaviour, Poor Attitude….
Some of you will recall a poll being published last month by the Sunday Times, conducted by “Behaviours and Attitudes”, a polling company with little record of measuring party support. This poll was significantly out of step with other companies at the time and so I didn’t devote time to analysing it, but the Sunday Times have used them again this weekend, with findings that the people are, apparently overwhelmingly, in favour of cuts instead of tax rises.
Given the use of polling to try to persuade politicians, I think it’s worth having a look at how reliable B&A are.
As a company, they’ve not been used, generally, for party polling, but to measure attitudes. One example covered here attempted to measure our attitudes to sex and sin. Interesting stuff, and certainly good for filling column inches, but not something that can be tested really – it’s not like there will be a referendum on the issues tested. Good clean fun, but nothing that establishes the quality of the polling in the first place.
As I’ve pointed out beforehand, most of the big 3 (MRBI, RedC & Lansdown Millward/Brown) have a particularly decent record in calling General Elections. They’ve been doing it long enough, and they have records that speak for themselves. In the last GE, all measured FF, FG, LP, SF, GP and OTH quite well, and as I point out here in the final polls of the GE 2011, of the 18 party totals, only one (FF in the L& M/B) was outside the margin of error (by 0.4%) and when you’re assuming a 3% margin of error to 95% probability, that’s about what you’d expect if polling was being done as close as perfect as possible (those margins are due to random statistical variation, not sample or method error). It’s worth noting that MRBI were closest (for the second GE in a row), Millward Brown were 2nd and RedC were third. (Unless you could my predictions which were slightly closer than MRBI, ahem!)
The B&A poll on 4th September had the following party totals;
FG 44%
LP 12%
FF 15%
SF 13%
GP 2%
OTH 12%.
Obviously, if this had been a finding of a major company poll, this would have been big news, showing sudden swings from LP and to FG. The most recent poll had been an MRBI one (who were closest in the previous 2 General Elections). This showed FG at 38% and Labour at 18%. Assuming they haven’t suddenly become pants at polling, this suggested that, if B&A figures were true, Labour had lost some 6% over the summer (more properly between 3-9%), and FG had gained a similar amount.
So maybe they had? If so, you’d expect it to show up in the other company results. However, no joy for them there. MRBI have since come out with a poll that has FG at 35% (i.e. 3% down on July), and Labour 17% (down just 1%, unlike B&A well within the margin of error). Millard Brown (who were second closest to the result in February) also have released a poll since the B&A one, and this actually showed Labour up 1% to 20%, and FG down 2% to 40%. RedC, who came 3rd in February, do have Labour down in their latest poll, but within the margin of error, to 16%, and FG down a whopping 8% to 33%.
In other words, all the other polling companies have B&A’s latest results well out on FG and LP, not just in the totals, but in the trends they were suggesting. Notably, their findings suggest an electorate that are more right wing than that found by the other pollsters, which of course, would not make them unattractive to a paper in the News International stable.
This isn’t to suggest that they aren’t a real company, that their polls are fraudulent, or even to suggest that there is bias at work. It’s just to point out that, as polling companies go, their figures lack the credibility that more serious pollsters have earned over the years. I don’t know why the Sunday Times have chosen them, certainly they don’t have the brand recognition or credibility that the big 3 have, and can only hope it’s because they are cheaper.
Judge them on their results, that’s what I always say….
Irish Polling Report 
I think your criticism of B&A is somewhat exaggerated, dotski.
The 3 previous FG poll results were 41%, 42% and 38% – 44% is not that out of line, especially as the next poll afterwards put FG at 40%. Also, it is not unreasonable for the main government party to increase in support over the summer, when there was some relatively good news for the government.
In reality, it is the Millard Brown IMS poll of 19 September that looks a little out of step – it puts FF at 10% and Labour at 20%, when the general trend seems to be FF consistently at 15/16% and Labour dropping down to a similar level – although possibly FF were hurt by the mess they made over the presidential election and this poll caught some FF voters temporarily switching to Labour.
I think that the most you can really say about B&A, after 2 polls, is that they may be slightly underestimating the Labour vote.
Ciaran
October 9, 2011 at 8:52 pm
FG could be argued as close to margin of error (although still an outlyer), but Labour figure is wildly outside it, far too much to be statistical variation. It’s either very rogue, methodical error, or the big 3 wrong on Labour. Given their track record, would have to be one of first 2, IMO
Dotski
October 10, 2011 at 9:56 am
[…] Sunday Times Poll. That is if its any way accurate…. Before we go any further Dotski over at IrishPollingreport doesn’t highly rate these pollsters and outlines why. That said they’re not Quantam Research State of Parties: FG 32 (+2) LAB 10 (-1) FF 16 (-4) […]
That “Behaviours and Attitudes” Poll….. and the Fine Gael dream contest. « The Cedar Lounge Revolution
February 25, 2012 at 11:44 pm
[…] […]
further increased poll ratings for SF - Page 76
February 26, 2012 at 6:13 pm
There is NOT just ONE margin of error for a poll – even though all polling companies report just one. The margin of error differs for each party, so what we have reported is the margin of error for a party that is about 50 percent. The margin of error for smaller parties is smaller.
So for the recent B&A poll, showing Sinn Féin to be on 25% – this is based on 650 respondents giving an opinion. If we calculate the standard error of the mean, and double it we get more-or-less the confidence interval, so +/- 3.4%.
The questions that B&A ask are pretty standard, and wouldn’t obviously bias the results. The sampling method is pretty standard as well, so this shouldn’t be a problem. B&A don’t use political weighting to weight the respondents to the last election, as many other pollsters have. But this is reasonably controversial, so they might not be necessarily criticised for this. Nor do they weight on probability to vote, as Red-C do. But again, we might not care at this stage when there is no election to predict. If we just want a measure of Irish people’s opinions, this should be ok.
The thing to remember is that we are a few years away from an election, so people might say things now that they would not do in the heat of an election campaign. Campaigns focus people’s minds and they are more likely to say what they would really do if there were an election tomorrow, because in fact there’s an election next week and they are thinking about this sort of thing.
So if I were in SF, I’d be pleased but not convinced. If I were in FF or Labour, I’d tell my people to relax – but I’d hopefully have a plan.
Eoin O'Malley
February 28, 2012 at 3:51 pm
Sorry Eoin, just saw that now. Yes of course, you’re right about the moe referring to a party at 50%, and there are a number of calculators you can use to check it for parties at different support levels, but as you show yourself it doesn’t deviate too much for the main parties (and would double the time I’d take to write up each blog entry!)
B&A are clearly way out compared to all the other companies, and the other companies are as accurate as one would expect, and so the question really isn’t whether they are out, it’s why. I understnad that they don’t do any weighting for liklihood to vote, RedC do this explicitly, and I understand other companies have proxies that effectively do this, and so that is possibly their problem. Unfortunately, it does result in polls that mislead people.
Dotski
March 3, 2012 at 11:14 pm