Credipedia - A Wiki Credibility Spectrum

[randomidea]

Hi everyone,

I haven't been to Manpollo for years, and it looks like participation has teetered out since my last visit. However, whereas I have gone on to start a new university degree and what not, several things about Manpollo and Greg's achievements have stuck in my mind. Most prominent has been the credibility spectrum, which, to me, is one of the most important aspects of Greg's arguments and also one of the most disputed.

That is why a friend of mine and I want to start a wiki project, whose aim it will be to create some sort of framework for people to post publications and assess their credibility.

This surely is a daunting task, and there are many obstacles. The most obvious one is how we make such a framework in the first place. It has to be accessible and easy to use, yet robust and, shall I say, self-protecting. The most I can say on this at the moment is that we will likely use a comprehensive Likert scale. Another obstacle is to account for our own biases while building the framework. We could just as well unknowingly exclude groups that have different biases from our own -- which, if the project is to be open and fair, is not ideal. On the other hand, we do indeed want to exclude ideologically driven motives -- or, more to the point, to bring those motives to the light of day so that they may be assessed by others. One other obstacle, which by no means is the final one, would be to prevent "source-bombing." It is feasible that ideologically driven groups could urge followers to assess something a certain way, so that the source appears biased or unbiased when the opposite is true.

It is also obvious that this project will delve into many other fields than just our own (Politics, Economics and Philosophy), and not only for the fact that the publications themselves will be from different fields. Not to mention the website itself will require coding well beyond my skill level. There is little doubt that we will eventually have to deal with different applied sciences, each with its own methods, as well as psychology, law, medicine, and the list goes on.... While the framework, as I envision it, should be able to encompass all of these fields, since it is not (only) the content that is to be assessed but the likelihood that the publication itself is credible (which could partially be done without at all reading the publication), we have to make sure that one field is not put at a disadvantage due to our framework, and we'll have to be able to ask ourselves whether one field is inherently more credible than another (consensus? method?).

The main feature of the framework, I think, will be to create some sort of bias recognition test for the users. This website, in effect, will be for lay people, and yet there are lay people and then there are lay people. Someone who scores highly on a bias recognition test would probably be more trustworthy than someone who scores lowly. The best thing, of course, would be if actual trained professionals, specialists in the respective fields, could join the framework, but the likelihood of that happening is low, and anyway it would be difficult to determine whether these people are indeed trained professionals. A comprehensive bias recognition test -- one that would be hard to manipulate and cheat -- is probably the best bet for weeding out users who would purposefully or otherwise bring their bias into the assessment. This does not mean, however, that other users would be excluded -- just that they would likely hold less weight than those who have passed such a test.

The last point would be to make people aware that while this is a tool for helping to decide whether a publication is credible or not, it is by no means the word of God (although we would like to see it becoming the closest thing!).

Anyway, I have rambled for long enough to give you a picture of what it is we are trying to do. At this moment I am not asking for people who want to join the project, because so far I want to keep it within the area of Hamburg University so that I can meet with people face-to-face. But I would greatly appreciate ideas and suggestions, and even tips for finding similar projects and whatnot. So far as I can tell there is no single, standard, comprehensive framework for assessing credibility across fields, even for journalists, who should be trained to do this. While this is disappointing, it is not surprising. I have no doubt that each field has its own methods of assessing credibility, and so it would be great to find those methods.

In the future I hope it will be possible for anyone to join the project and become a "credibility expert." If you want to follow and help along now, then use this thread, or follow us on diaspora by following #credipedia

Take care,

Andrew

[randomidea]

I guess this place really is dead, since I don't think this is such a bad idea....

[BaerbelW]

Originally Posted by randomidea

I guess this place really is dead, since I don't think this is such a bad idea....

Hi Andrew,

yes Manpollo is mostly "hibernating" at the moment and only perhaps a handful of people check in every once in a while and/or post something. I did see your post the other day, but I'm fairly heavily involved with Skeptical Science and unfortunately just don't have the bandwidth to get involved with a project of the scope you envision.

I think that your idea has potential but only if - and I guess it's a big IF - the content-creation cannot be gamed by "special interests".

Cheers
Baerbel

[wibble]

Originally Posted by randomidea

Someone who scores highly on a bias recognition test would probably be more trustworthy than someone who scores lowly.

That sounds interesting. I'd very much like to test myself to see how highly I score on bias recognition. I would be most grateful if you could point me to any such tests.

Originally Posted by BaerbelW

Hi Andrew,

I think that your idea has potential but only if - and I guess it's a big IF - the content-creation cannot be gamed by "special interests".

Ditto. There's too much gaming of the system going on already, the last thing needed is more noise...

... so I should shut up