In show #150 of FIR last Thursday, Shel and I talked about Toby Bloomberg’s experience with Jupiter Research concerning their recent research report about corporate blogging.
This is the report that says 35% of large companies (in the US) plan to institute corporate blogs this year, and nearly 70% of all site operators will have implemented corporate blogs by the end of 2006.
I’d mentally-dismissed the research when I read the press release. Nice backgrounder with yet more big numbers about the blogosphere with the usual PR quotes. It didn’t seem wholly credible to me based on reading the press release, but I certainly wasn’t inclined to cough up $750 for the report. I saw a yawning gap between my perceived value of what the report might contain and the cost of acquiring it.
What makes the whole thing interesting, though, is Toby’s experience with Jupiter. She’d asked them for more information so she could get some facts to write about that research. But their PR agency just gave her the brush-off because Toby has a blog that’s connected with her consulting business, and they suggested she buy a copy.
So that’s what Toby blogged about.
The new twist in Toby’s tale is the experience of Fard Johnmar at HealthCareVox MarketingVox who did lay out the $750 to Jupiter. This is what it got him:
– A four-page summary of the research results
– A 30-minute interview (which I have not yet conducted) with the analyst responsible for putting together the report
Plus he had to sign a non-disclosure agreement so he can’t publicly comment in any detail about the research findings. Furthermore, the PR agency told him:
JupiterResearch is not revealing any more information about the survey to any member of the media. We do not play favorites. If we answered your questions, we’d have to answer everyone’s. It does not matter if you write for the Wall Street Journal or a blog. We are not revealing any more information about the survey’s methodology.
Bizarre. So Jupiter’s PR agency expects you to buy this “research report” yet they refuse to answer any questions about the research methodology. In which case, why should anyone trust anything in this resarch if no one will answer questions about how it was conducted? Looking at the agency’s response, you have to ask: Is Jupiter not confident enough about their methods and their research?
No wonder Fard’s conclusion is this:
I have two pieces of advice for readers:
– Don’t buy this report
– Don’t accept the results of this survey
Enough said about the research.
Now what about the PR agency? Well, it’s clearly a firm whose people lack some of the PR 101 basics on relationship-building, let alone an understanding of the area the report’s about.
Neville –
This experience has yet another complexity. I have to assume that Jupiter’s PR agency was the conveyor of their client’s direction – not to provide the research methodology. (By the way, I think the reference to the WSJ came into play because Constantin Basturea forwarded the post to Carl Bialik of the WSJ with the hopes that he could get a journalist copy of the report.)
So the agency gets slammed by bloggers for following their client’s orders. Although that type of negative fall out might have always been an industry hazard; however, as PR firms take on the responsibility of ‘blogger relations’ the agency can no longer hide behind the scenes with only their clients front and center. The ease of publishing consumer generated content coupled with viral linking changes the PR game even further. As you pointed out the PR agency is as much at play here as is their client. Two companies’ reputations could be at stake. (One commenter on the Diva Marketing post said she would be “advising my clients to look for market research elsewhere.”)
Although ‘don’t kill the messenger’ might be a subtitle of this story, what does a PR professional do when a client provides a mandate that might create a blog storm that could effect both client and agency?
Additional complexity indeed, Toby, I agree, assuming that the agency were just following orders. It would begger belief that they’d be taking the stand they have taken without their client’s explicit direction.
It’s a tricky one as you point out. Without knowing anything at all about the agency or their relationship with their client, it’s hard to analyze the picture in any further depth. For instance, all I know is what I’ve read in your posts and that of Fard Johnmar.
Still, what you’ve said has given me pause for further thought. I’ve jumped to a conclusion that may be unfair to the agency without some further knowledge. Killing the messenger indeed.
So what I’d like to know is this – is the agency’s position something the client insisted they adopt? Was it the agency’s recommendation to the client? Or what?
Probably very difficult indeed for answers to these questions to come out publicly. A pity that Greg Dowling, the Jupiter analyst who is the author of the report, doesn’t blog (he’s not listed in the listing of Jupiter bloggers) as he might be able to throw some light on the overall situation.
When I first read of the incident, my reaction was that the agency was merely following the direction of the client. Many of us have had the unenviable position of having to follow poor client direction. We advise, but sometimes our clients feel that they have to follow a different course of action (lawyers anyone?).
In this case, however, I think that the agency does have to shoulder some blame. I would hope that the agency would have at least anticipated some of the questions from the media. Number one question would have to be: How did you conduct the research? As someone who has managed a fair amount of research, that is the basis of the believability of any research project. If I can’t provide that basic information *to protect my reputation* as well as that of my client, I am compelled to walk away.
Our responsibility as public relations pros is to provide accurate information for our clients to various publics. If that information is not accurate and we know it, we ruin our reputation and that of our industry. In this case, this agency might have provided accurate information, but they have done so in a way that it makes the information look suspect. They should have better advised their clients or offered a referral to another agency.
A good assessment, John, thanks.
I’ve just taken a look at the Jupiter case study on the website of the PR agency, Peter Arnold Associates.
Talk about proactive outreach, streamline requests and other such PR jargon. Thta doesn’t appear to gel with what’s happening in the experiences of two bloggers at least.
I really would love to know the answers to the questions I posed.
Well then Neville, based on that case you indicated and John’s remarks and that both client and agency claim to be leaders in the social media field, *both* should have known better.
I find it interesting that neither company has joined in any discussion. Perhaps they’re assuming the attention spam of bloggers is nano seconds and this will be forgotten as soon as the next controversial issue comes up to bat. Several ‘lessons learned’ for PR professionals from this situation. Can’t help but wonder what JupiterResearch and Peter Arnold Assoc. took away …
All:
Thanks for your comments. I agree with everyone, especially John. It is a company’s responsibility to provide information about a survey’s methodology so that the public can evaluate whether the results are believable. My major problem is that the report itself — the one I was directed to no less — provides very little information about the methodology/demographics of the survey. So, did the PR firm know that the report had so few details? Why is JupiterResearch publishing reports that do not provide basic statistical information? Is that information only available to clients?
I don’t know, but I’ve certainly learned a lot from this incident — about what NOT to do.
Fard
Toby, maybe they do see it purely as “attention spam” :)
More good questions, Fard. Add those to the ones I asked plus the points John raises re the agency’s anticipating the type of questions that might arise regarding the research – the very questions you and Toby have asked.
It’s the weekend and maybe the agency and their client don’t pay attention to online discussions at this time. Yet lots of people are talking about this, some with a great deal of skepticism.
If client/agency really aren’t paying attention, as opposed to just ignorning the chatter, then Jupiter’s claim that they are “a leading authority on the impact of the Internet and emerging consumer technologies on business” (from the press release) doesn’t seem credible at all.
Neville:
Thanks for your comment. I’ll be sure to try to squeeze in as many questions as I can during the conversation with the analyst. My major hope is that the company allows me to go public with their response. I’m a big believer in providing people/companies to have their say. Although I’m skeptical of the study results, I can be convinced that they are valid — if I am provided with the right information.
It struck me this morning that we have talked around this issue and have not really discussed the results. If I’m Jupiter, that’s the point where I recognize the complete failure of this study (or the release of its information).
Let’s pretend that they took a different approach and announced their methodology — methodology that carries no statistical validity or reliability (I have no idea if this is the case, we are pretending). So, how do I announce it? I add something in the release to the effect of “while this study is not conclusive it…,” “we are going to be furthering our study in this area…,” blah, blah, blah.
If Jupiter did that, what are we discussing? The results. Perhaps we’ll debate their methodology, but if Jupiter is upfront with their methodology and its shortcomings, it’s no more than a discussion. Jupiter gets coverage by the major outlets who will mention that Jupiter is conducting more research in the area. Suddenly, Jupiter is leading the way in learning about blogging, etc. What a great position!
By this major blunder, Jupiter not only took two steps back, but they missed the opportunity to take two steps forward.
Thanks, Fard and John.
Shel and I will be discussing all of this in today’s edition of the FIR podcast.
Have any of the people calling this “a major blunder” or the like actually bothered to call up Jupiter? The names of their analysts are on the web site, their number isn’t hard to find (in fact, if you dig into their blogs, you can find several analysts’ cell phone numbers).
If you’re accusing someone of something, then you should really give them a chance to comment – and not through their PR agency.
Ian –
Upon receiving my request for information about the methodology behind the study, the agency sent me an email which said, ““Let me check in with someone on the research team at JupiterResearch to find answers to your questions. I’ll be back in touch as soon as I hear.†So although I did not contact Jupiter directly, I have to assume that their agency did refer my email to the analysts who were involved with the research.
Jupiter was aware of my request and concerns about their findings chose not to provide any additional information. When Peter Arnold informed me that Jupiter had decided not not provide additional information I emailed the agency telling them that I intended to post Jupiter’s findings, based on the data that was available to me, along with my concerns about the lack of methodology information. That was the last communication I received.
Based on Jupiter’s involvement in social media, I also have to believe that someone at the firm and/or the agency is following this conversation and either Jupiter or Peter Arnold Associates has the opportunity to jump-in at any time to correct misconceptions.
So .. how far does a blogger have to go to provide accurate information to her readers? Was it my responsibility at that point to spend additional time tracking down emails/phone numbers when it was obvious to me that JupiterResearch didn’t want any further discussion? Keep in mind, I’m not an investigator journalist and no publication/media outlet is paying me to write this story. If the WSJ or the NYT want to hire me to go further .. happy to do that.
In response to Ian…
I believe that Jupiter is directly in the blog monitoring business…but even if not, if they are unaware of the conversation that has been generated by Toby’s post, then they should turn in their social media membership card and magic blog de-coder ring. Their silence on this is deafening.
After reading the quote below from the Jupiter President, I would say that Jupiter, albeit inadvertently, engaged prospective customers in an active dialogue and elicited feedback…as long as they believe that negative attention is better than no attention we can assume that they are pleased with the conversation.
“By engaging prospective customers in active dialogue, companies can showcase their expertise and domain knowledge, creating a forum for communication of their strategies and visions,” said David Schatsky, President of JupiterKagan. “In doing so, companies can generate buzz around their products or services, while eliciting feedback and collaboration from product evangelists.”
The “I am not an investigative journalist” excuse really doesn’t wash, Toby. The accusations made are serious ones which – thanks the “echo chamber” effect – can have serious consequences for Jupiter’s business. I think the five minutes that it would have taken to find the email address of someone there – rather than at the PR company – is the least that is due.
I’m not saying you’re wrong. It might be a lousy piece of research – even the best research companies have a dud every now and then. I’m saying that if you are going to publish with inflammatory headlines like “Jupiter Research passes round the Kool-Aid”, leading to stories like “Don’t expect answers from Jupiter Research” you have a responsibility to chase down the story. At the moment, the best story you have is “Jupiter Research’s PR company doesn’t get it” – which, as you’ll agree, isn’t quite the same as the story is spun here and on your site.
It seems like you’re expecting the priviledges that Jupiter extends to accredited journalists – getting more info about the research, getting to talk to analysts – without wanting to do the same level of work.
As an aside, while you might not think you’re doing investigative reporting, clearly some of your readers do: take a look at the first comment on your site about the Jupiter story.
Ian –
Yes, the concerns about the lack of information regarding the research methodolgy should not be taken lightly.
I would be happy contact a JupiterResearch analyst. Would you please forward me the phone number or email address: tobyb1 at gmail dot com?
You might find this interesting. In trying to find Alan Meckler’s contact information I came across this post that Mr. Meckler wrote on his blog about a poor customer service experience with FTD flowers. Following are Mr. Meckler’s conclusions and recommendations to his readers.
http://weblogs.jupitermedia.com/meckler/archives/008141.html
“Perhaps I will order flowers online in the future, but I would never use FTD.com. I will let readers know if the flowers are ever delivered. Based on my experience with customer service over at FTD.com, it will be difficult to get my money back — the response will be an automatic: “sorry, but we are very busy due to the holiday” or some such silly excuse.
In the meantime I hope Google and other search engines pick up this post so that searchers can be warned about FTD.com.”
Sure, Mr. Meckler spoke directly to the company .. a customer service rep. Should he have contacted someone more senior before he posted his remarks that could have serious consequences for FTD? Perhaps. However, what the post tells me is that JupiterResearch is no stranger to using blogs for expressing personal opinions and is well versed in social media strategies. Again, my message was relayed to the JupiterResearch team – even if indirectly – throught the channels that the firm established.
Toby – yes, I think Alan SHOULD have thought before he posted. That’s the danger of the blogosphere.
I’ve emailed you some more info (or will straight after this post!)
Ian,
Jupiter chose to release the report through their PR agency and the PR agency was the designated contact. Toby contacted them and was told that they would check with Jupiter and respond…which they did by saying that they were not going to provide support for their study. Furthermore, Fard Johnmar bought the report for $750 and still was not able to find out the necessary information about the Jupiter methodology. I think the ball is clearly in Jupiter’s court at this point. However, I think that it was Jupiter’s responsibility from the very beginning to release methodology and other pertinent documentation with their findings. That is just best practices. Their report is confusing at best and questionable at worst; being clear on methodology and sample is their responsibility and by not providing it either with the press release or when requested by someone seeking clarification they put their credibility at risk. I think Toby did a more than adequate job attempting to clarify things that should have been made clear by Jupiter to begin with…in other words, their press release was the problem and it seems that maybe their research is a bigger problem. You are taking shots at the wrong messenger in my opinion.
Marianne,
“Fard Johnmar bought the report for $750 and still was not able to find out the necessary information about the Jupiter methodology.”
Fard admitted that he hadn’t actually had taken up the 30 minute interview with the offending analyst, at which he could – as a client – have asked about the methodology in depth. So Fard’s comments are, at best, based on incomplete information. Of course he “was not able to find out the necessary information”: He didn’t ask the necessary questions!
Jupiter did NOT release the report through its PR agency. It’s PR agency released a press release about it’s release. There is a big difference. A press release is not the report, and will not contain all the details of the report. Why should it? You want the details of the report, including its methodology – pay up the cash.
In saying “their press release was the problem and it seems that maybe their research is a bigger problem” you’re making the typical echo chamber assumptions, and making a serious accusation about the quality of Jupiter’s research based on precisely zero evidence. Can you justify what you’re writing, based on the evidence at hand? I doubt it.
Ian:
Thank you for your close attention to this issue and for your many comments over the past few days. The fact that the JupiterResearch report is receiving so much attention within the blogosphere indicates that bloggers take research in this area seriously and are more than willing to debate it.
In your comments, you make a number of points. I’d like to respond to two of your major assertions below.
1. The JupiterResearch Incident Is Another Example Of Bloggers Acting Irresponsibly
You argue that bloggers have a responsibility to exercise all due diligence before posting negative commentary about a company. You have asserted that Toby did not do enough to get JupiterResearch’s side of the story before posting her original article on this subject. If she had gone around the PR agency, she may have received answers to her questions.
You make a good point about the responsibility of bloggers to check their facts before going public with a story. However, as someone who has been involved in numerous public relations campaigns on a number of contentious issues, I am well aware that companies hire public relations counsel to interface with the media and direct inquires to the appropriate company spokesperson(s). The media have come to expect this and nine times out of ten, if the company wants to comment on an issue they will direct their PR counsel to schedule an interview.
My point is this: By going to the PR agency, Toby was interfacing with the company directly. In fact, based on the content of their e-mail, Peter Arnold Associates contacted JupiterResearch about Toby’s questions. The PR firm relayed the company’s response to Toby.
When I contacted the PR firm, I was told the same thing: JupiterResearch (not the PR agency) is not releasing any more information about the study methodology. Anyone else in this position would have made the conclusion that Peter Arnold’s response was that of the company. Simply put, JupiterResearch did not want to respond to her inquiry.
As you well know, this happens to mainstream media all of the time.
Overall, I believe Toby acted very responsibly. She did not post before contacting JupiterResearch (through their PR agency) about the research. She could have simply posted her opinion on the research without contacting the company. She did not, which raises her level of professionalism in my eyes.
Remember: Companies set up information sharing channels for a reason. Toby followed the channel established for the media and received a corporate response. I don’t think Toby would have received a different one if she had gone around the PR firm.
2. I Asked The Wrong Questions When Reviewing The Report
As I mentioned in my original post on this issue, I decided to purchase the report because of my intense interest in this issue. I also purchased it because the PR agency directed me to the report, which they implied would provide me with answers to the questions I was asking about the study’s methodology.
Overall, I am very confident that I asked the right questions about the report. For example, the National Council on Public Polls (NCPP), has a very good article, “20 Questions Every Journalist Should Ask About A Poll.” Key questions they suggest journalists ask include:
– How many people were interviewed for the survey?
– How were these people chosen?
– When was the poll conducted?
– How were the interviews conducted?
These questions get at the heart of how a study is conducted and whether the results can be accepted. It is standard operating procedure to provide this information. For example, Harris Interactive, which produces a number of online and client-directed surveys, releases detailed information about the survey’s methodology in every press release it issues about its polls. For example, see this press release.
In Conclusion
As someone who is trained in statistical methods and has interpreted and reported a number of surveys, I find it odd that JupiterResearch did not reveal their survey methodology in the press release they issued about the report. I find it even odder that they did not provide detailed information about the research methodology in the report I purchased. I expected to see this information and still wonder why it was not provided. I don’t think I should have to speak with an analyst from the firm to answer basic questions regarding how the research was conducted.
Every survey has limitations. I believe that they should be highlighted so that people will view the research as credible and reliable. (In fact, I am currently planning to launch an opinion poll in partnership with another firm that — while it has a number of limitations — I believe will provide some useful information. I plan to ensure that anyone reading the research report will be fully aware of these limitations and know that we are planning follow-up data to attempt to bolster the data and identify trends.)
Thank you again Ian for your comments. As an aside, I am currently speaking with someone at JupiterResearch who is providing me with information about the survey methodology. As I have mentioned a number of times, I plan to provide JupiterResearch with the opportunity to respond to my criticism of the report and set the record straight about their research practices.
Overall, I think we acted very responsibly.
Ian,
I think that I am justified in saying that there is really no sense in quibbling over every word that I used in my comment…of course they released a press release ABOUT the report; and within that press release there was information announcing the FINDINGS from the report. A press release that was merely announcing the report would have done just that…perhaps “New Report Available From Jupiter Research on Corporate Blogging” period. So by releasing the information that was at odds with other available data points without any supporting methodology, Toby began inquiring about the methodology; she was not seeking the report. So, with all due respect, I believe I am justified in my statement that the press relesase was problematic in the information that it provided and in the information that it did not provide; I don’t believe I have made a “serious accusation” about Jupiter’s report; I am quite aware that I have not seen it which is why I chose the word “maybe.”
Fard,
I agree with you regarding the release of the methodology both in the press release and the report that you purchased…it is standard operating procedure to provide this information; context provides meaning. To paraphrase Mark Twain, the difference between the right word and the almost right word is like the difference between lightening and a lightening bug. The difference between 6.8% and 70% could be the difference between a Fortune 500 corporation and a large corporation. It helps to know the right word.
Marianne
Fard,
First of all, thanks for the great response with lots of good points. It’s exactly this kind of dialog that the blogosphere is good at when it’s at its best.
I’m not going to respond at great length, because I’d like to let you points have time to sink in a little. But a couple of things pop into mind.
First of all, I don’t think that what Toby herself did was irresponsible. As I said above, I think Toby did the right thing up to a point, and apart from the headline what she wrote wasn’t inflammatory. However, the effect of the echo chamber is such that what she said was bound to be spun in directions that she couldn’t justify: that Jupiter doesn’t get bloggers, that it does lousy research, and so on – none of which is justified from her original post.
Does Toby bare some responsibility for what others have interpreted her story to mean? It seems harsh to say yes. After all, “you own your own words” (to borrow the old WELL slogan) but you don’t own others. The problem with that view is simply that others will put you in roles that you’re not actually taking. Toby was posting as a blogger, yet the first comment on her story thanks her for “investigative journalism”. I’d suggest that once you start to be put in that role, you start to have to do what investigative journalists do – and to her credit, I believe that Toby is taking that responsibility seriously.
And from your comments, so are you – it’ll be interesting to see what you come up with.
Marianne, I understand your “maybe” :) The problem is that in the echo chamber, “maybes” are often the first thing that are dropped as a meme moves on. I’m actually going to do some proper work when I get chance tracking how stories evolve as they are passed around the blogosphere – it’ll be interesting to see how it works.
blog et entreprise : un métro de retard ?…
les blogues laissent ils indifférents les entreprises ? non si on en croient une étude (mise en cause d’ailleurs ici ) du célèbre cabinet jupiterResearch À les croire 34% des grandes entreprises américaines disposeraient d’un blogue corporatif,…
Not providing the research methodology before one’s 30 minute interview is nonsense. Without that information the 30 minutes could easily be aboout research methodology only depending on one’s interests in evaluating the report.
As a highly trained qualitative researcher (PhD, OSU, 2000) I’m deeply disturbed by this whole affair. I’d like to get into more commercially relevant research myself so seeing a major research firm undermine the credibility of such research is quite disturbing.
It’s been interesting to read the comments about the agency and I’ve gotten much more interested in the relationship of research and pr. But doesn’t any company have to take responsibility for the services they outsource? I don’t see how Jupiter Research could get off the hook at this point in time.
Because most of my blogging is industry oriented I am regularly faced with the call or not question. In this case, that’s not an important issue based on the extensive contact with the appointed representatives of Jupiter Research.
On bloggers and echo chambers, the reality is that the web communication is simply heightening the dynamics that already existed. If you talk to researchers who deal with the press or study what happens in the process of research findings going mass media, you’ll find that the nuances disappear very quickly no matter how well you state things.
Deep research always produces a lot of maybes and the media, politicians, etc. don’t handle that very well and remove the nuances that they don’t understand or can’t fit into a soundbite. In the process they typically misrepresent the research.
It’s a mess and it’s sad to see a research firm contributing to the situation in this manner.
Peace
[…] My own commentary on this story a few weeks ago attracted quite a few opinions from readers of this blog on Jupiter’s behaviour and that of their PR agency. A particularly thoughtful view on that latter point came from John Mims: […] I think that the agency does have to shoulder some blame. I would hope that the agency would have at least anticipated some of the questions from the media. Number one question would have to be: How did you conduct the research? As someone who has managed a fair amount of research, that is the basis of the believability of any research project. If I can’t provide that basic information *to protect my reputation* as well as that of my client, I am compelled to walk away. Our responsibility as public relations pros is to provide accurate information for our clients to various publics. If that information is not accurate and we know it, we ruin our reputation and that of our industry. In this case, this agency might have provided accurate information, but they have done so in a way that it makes the information look suspect. They should have better advised their clients or offered a referral to another agency. […]
This has been a terrific discussion, everyone, thank you. I’ve posted an update with a summary of the pros/cons expressed in this discussion:
http://nevillehobson.com/2006/07/12/the-twisted-jupiter-tale/
I’d say the story is far from over, though – take a look at Toby’s latest post.
[…] This is where I first got turned on to the story. […]
[…] The kerfuffle about Jupiter Research and their corporate blogging report (see the gory details here and here) has reached one conclusion with Fard Johnmar’s post yesterday in which he reports on the result of his telephone discussion with the Jupiter analyst concerned. […]
[…] I hope this doesn’t turn out to be like the Jupiter farce last year. […]