DailyDirt: Judging The Quality of Science
from the urls-we-dig-up dept
The Scientific Method is often taught as a linear process that proceeds from hypothesis to theory. In practice, science — like any other human endeavor — can be much more haphazard. Science isn’t perfect, but it has some built in processes for error correction to help weed out mistakes and outright fraud. There are some well-known issues with the traditional peer review mechanism, but science is adapting and coming up with new ways to improve its results and conclusions. Here are just a few links on judging the quality of science.
- The Meta-Research Innovation Center at Stanford is planning to develop a meta-research institution to research… research. The new lab will be called METRICS for short, and it’ll study factors like irreproducibility and publication bias. [url]
- Jimmy Wales has decided that no pseudo-science will be accepted for publication on Wikipedia. Wales says, “If you can get your work published in respectable scientific journals — that is to say, if you can produce evidence through replicable scientific experiments, then Wikipedia will cover it appropriately.” [url]
- As more scientific research is done more globally, is the diversity of the researchers influencing the quality of the results? Some analysis suggests that papers with geographically diverse authors are better, in the metric of getting more citations. [url]
If you’d like to read more awesome and interesting stuff, check out this unrelated (but not entirely random!) Techdirt post via StumbleUpon.
Filed Under: jimmy wales, meta-research, peer review, publication bias, reproducibility, science, wikipedia
Comments on “DailyDirt: Judging The Quality of Science”
Science or Pseudo-Science - How can we tell
The problem Jimmy Wales has here is that it can be quite difficult to tell the difference between science and pseudo-science. There are enough discussions taking place right now about the subject.
The biggest problem we have on both sides of the argument is, of course, people. Now that I have more time to look into various subjects, it is interesting to see the number of main stream areas (in science) that are actually problematic. Particularly, when a small group sees one of the problems and starts to discuss this and are then labelled as “kooks” because they do not agree with the mainstream ideas. Yet they are raising legitimate concerns about the prevailing models.
Of course there are many different models out there which fail even the first experimental tests. But there are still others which oppose the standard models and have suitable match up with experimental results and still are considered pseudo-science because they go against the mainstream.
I know of various experiments that have been proposed which have been given no support at all. These experiments would categorically dispose of the alternative model if they failed. Yet as far as I can tell, no such experiments are being worked on because it would be career suicide to partake of these experiments. It quite often appears that those controlling the purse strings are the ones who would lose most with their reputations if they allow the experiments to go through.
Re: Science or Pseudo-Science - How can we tell
One obvious way is: do the proponents of the field practise basic tenets of critical thinking? Do they conduct rigorous controlled studies? In the medical field, for example, it is very important to account for the placebo effect. Are their results reproducible? Just asking these basic questions is enough to rule out 90% of the bullshit.
Yes, science is fallible, because it was created by fallible humans, after all. But, rather than relying on some imaginary Higher Authority to fix the problem, science, builds on an interesting characteristic of that fallibility: that we are better at spotting other people?s mistakes than our own.
This is the essence of critical thinking. Not only is it useful to help filter out mistakes, it also works against deliberate fraudsters as well.
Perhaps that?s why the ?alternative medicine? crowd?and all the other religionists?hate it so.
Re: Re: Science or Pseudo-Science - How can we tell
Let’s apply this to astrophysics. Black holes, neutron stars, dark matter and dark energy. None of these entities have actually been observed – they have been proposed based on the mathematical models developed and are believed to exist. But they have not actually been observed as testable entities.
Yes I know there will be those of you who will say that they have been observed. But I would suggest you check out the actual language used and it boils down to “most probable object/entity”. If you pin these scientists to the wall they will say that they believe the entity in question is one of the kinds of objects. Dark energy and dark matter have most definitely NOT been observed. Every experiment that has been devised to find them has had a negative result.
True, they rely on their Higher Authority to determine the “truth”.
There are interesting discussions going on at the moment about the difference between “science” and “scientism”. Whatever your view, there are some very critical thinkers who are raising their voices about the rise of “scientism” and the harm it is causing to the scientific method and investigation of the world around us.
I have had my discussions with various “subject matter” experts over many years about different models. There have been various attitudes expressed when asking questions that oppose the “models” believed in by these experts. Everything from “good questions and here is my take on the answers” to “trust me, it’s too complicated for you” to “you’re an idiot to be asking any question against the model – it’s proven fact”.
Which religionists? – atheists, buddhists, moslems, hindi, communists, fascists, humanists, naturists, christians, capitalists, new agers, alienists, nihilists, pantheists, animists, secularists, etc.
Re: Re: Re: None of these entities have actually been observed
Do you believe in bacteria and viruses? Atoms and molecules? Have they actually been observed?
Re: Re: Re:2 None of these entities have actually been observed
Bacteria most definitely have been. Any science classroom microscope is powerful enough to see them.
Re: Re: Re:3 Any science classroom microscope is powerful enough to see them.
What is your understanding of how a microscope works? Have you seen the principles of one with your own eyes?
Re: Re: Re: Science or Pseudo-Science - How can we tell
“Black holes, neutron stars, dark matter and dark energy… have been proposed based on the mathematical models developed and are believed to exist. But they have not actually been observed as testable entities.”
I don’t know what you mean by “observed as testable entities”. Our theories of these entities make predictions, which we can and do test. If you don’t want to believe in them, fine, don’t. But I have a sneaking suspicion that the “models which go against the mainstream” — which you mention but don’t specify — don’t stack up so well.
Re: Re: Re:2 Science or Pseudo-Science - How can we tell
The biggest problem with such entities as Black Holes and Neutron Stars is that when you include all model phenomenon for these entities, they can’t exist in our universe.
Black holes – include time dilation effects and they will never form
Neutron Stars – include strong force interactions, a neutron star would have to be considered as a single nuclear entity and by the rest of nuclear physics cannot exist. Gravity is not sufficiently strong enough to override strong, weak and electromagnetic forces at the atomic and sub-atomic level. Remember that neutrons themselves are unstable particles unless found in very specific environments (which require protons and electrons).
They only exist if you ignore the full range of effects and forces as predicated by the various models in use.
I don’t deny that there are strange entities out there but I do seriously doubt that they are what the current ideas are saying.
Re: Re: Re:3 when you include all model phenomenon for these entities, they can't exist in our universe.
But the physical models–you know, the ones that describe our actual universe–say they do. Einstein?s General Theory of Relativity famously predicted the existence of black holes long before anything resembling them was observed.
How do you reconcile that? Perhaps your (mis)understanding of physics is the problem.
Re: Re: Re:4 when you include all model phenomenon for these entities, they can't exist in our universe.
Do you understand what a model is?
It is a simplification (an approximation) of the actual entity or physical environment. Many things are ignored to obtain the model. The model is used for specific investigations. So when you change what parts of the actual entity or environment you are investigating, then a different model will generally be needed (so you can ignore areas you are not investigating).
I have also seen a review of said work that says the model being by him used was based on a single body within the universe, not multi-bodies that exist in our universe.
Again, I’ll reiterate – models are simplifications and approximations of the real world. Go outside the boundaries of the model and the models will give you incorrect results (incorrect approximations to the actual answer).
Example: Newton’s Laws of Motion (including gravity) are very good approximations (a good model) at low speeds and low gravity but at high speeds and/or high gravity corrections have to be made to the model to give the answers which approximate the actual observed results.
Example: Electronic circuit analysis at DC or low frequency AC uses simplifications and approximations of the different types of circuit elements. Change the conditions to high frequency, the model must change to give you the results.
Example: Static and Dynamic analysis of structures. Assumes in many cases that we are dealing with rigid bodies and for the level of forces and the frequencies in question, we can do this. But change things and we need to change the models to non-rigid bodies.
Example: Mathematics uses infinites and infinitesimals to provide a way of solving calculus problems. There are different techniques to remove infinites from the results including the old L’Hopital’s Rule. The mathematics is then used in the various models for the discreet world.
I understand what models are and how they are used and I don’t assume that the model in question is relevant outside its original domain.
Re: Re: Re:3 Science or Pseudo-Science - How can we tell
You’re wrong about neutron stars; show your calculations if you like. I’ll hold off on black holes until I’m satisfied you know what your words mean.
Re: Re: Re:4 Science or Pseudo-Science - How can we tell
Show me any experimental evidence that a multi-neutron entity (where the number of protons is very much less than the number of neutrons) exists and is stable and I’ll give credence to an object like a neutron star existing. To do their calculations they use quantum mechanics to ensure that the energy states are unique. This occurs within very specific conditions such as nuclei and near absolute zero.
Re: Re: Re:5 Re:Show me any experimental evidence that a multi-neutron entity (where the number of protons is very much less than the number of neutrons) exists and is stable
Would everything above about calcium on the periodic table do? All the larger atomic nuclei have significantly more neutrons than protons.
Re: Re: Re:6 Re:Show me any experimental evidence that a multi-neutron entity (where the number of protons is very much less than the number of neutrons) exists and is stable
No.
I want to see a stable entity that has a ratio of at least 100 neutrons to one proton.
The last stable element is Pb (Lead), though this is still considered very slightly radioactive. The neutron to proton ratio is less than 2:1.
Neutron stars are considered to considered to consist of neutrons with a very small level of impurity by protons, hence I want to see a stable entity that has at least 100 neutrons for every proton.
Re: Re: Re:7 Neutron stars are considered to considered to consist of neutrons
You do realize neutron stars are held together entirely by gravity, don?t you?
Re: Re: Re:8 Neutron stars are considered to considered to consist of neutrons
The point I am making is that gravity is completely insignificant at the level of neutrons beside neutrons and that the strong and weak and electromagnetic force interactions should be the most significant forces. Hence, my question is why are these forces being ignored for a force that is so weak. The stability of neutrons is low, take them out of stable nuclei and that neutron will decay in just under 15 minutes. Inside unstable nuclei, we then have beta decay processes.
Since combinations of neutrons in a neutron stars would not be in stable nuclei, we should expect to see significant decay, hence proton/electron formation. What is the effect of the strong/weak/electromagnetic force on the stability of the entity in question? What happens to any electric currents that may form in any magnetic fields that may be created?
The model posits that the strong, weak and electromagnetic forces are non-existent and that the only consideration is gravity. This is not demonstrated as a factual reality. We see all sorts of interesting phenomena in ordinary radioactive materials, so why would we not see such in the posited neutron star?
My whole point is that treating an entity that is effectively only neutrons (as per the model) and as such is not “ordinary” and is only affected by gravity (as per the model) is so simplistic as to be effectively nonsensical. Those who support this model would have to demonstrate a number of things:-
1). Entities consisting of either only neutrons or having “impurity” protons of less than 1% exist.
2). That such entities are only affected by gravity alone and that the strong, weak and electromagnetic forces are irrelevant to such entities.
Now it may be that someone somewhere has done both things as well as any other necessary provisos for neutron stars to exist as the model states. If this has been done, I would have expected it to have been heralded from the rooftops. Have you heard of such news?
Re: Re: Re:9 The point I am making is that gravity is completely insignificant at the level of neutrons beside neutrons
So what else do you think has crushed all those atoms down into neutronium?
Re: Re: Re:10 The point I am making is that gravity is completely insignificant at the level of neutrons beside neutrons
I am not saying neutronium exists, you are. I have simply pointed out that gravity is not sufficient against the other forces at the subatomic level to allow neutrons (only) to exist as a stable entity (as demonstrated by the inability of anyone today to produce any stable entity that consists solely of neutrons). Hence there is no actual basis for neutron stars to exist. It is based on the outcomes of a theoretical model which may in fact be the result of not examining the basic assumptions of that model to see if in fact those assumptions are even reasonable.
You appear to have an inability to follow a discussion in any logical form but appear only capable of approaching the discussion from a point of view that is based in total belief that you are right and anyone that has an alternative view is wrong. This is demonstrated by
You do not even question if neutronium is even possible.
This is usually the argument that is made against religionists of any kind. So, the question is – are you a “true believer”?
Re: Re: Re:11 I am not saying neutronium exists
You were already talking about ?neutrons??that?s what ?neutronium? is. So now you?re saying there aren?t these neutrons?
Re: Re: Re:2 Science or Pseudo-Science - How can we tell
I suspect the real issue behind that comment was the Climate Change/Anthropogenic Global Warming “debate”. (At least, in my experience, climate change debates are when this meme is most likely to pop up.)
Re: Re: Re:3 Science or Pseudo-Science - How can we tell
Oh yeah, I forgot to mention — the “sophisticated” section of the anti-evolution crowd kind of likes this argument, too.
Both the climate-change denialists and the anti-evolutionists despise scientific modelling.
Re: Re: Re: Science or Pseudo-Science - How can we tell
If by “observed” you mean directly detected and measured, then black holes have been observed: http://news.discovery.com/space/galaxies/spinning-black-hole-nustar-x-rays-130227.htm
Re: Re: Re:2 Science or Pseudo-Science - How can we tell
Read the article carefully, they have observed something. But they have not observed the black hole, they are using the observations as an indirect support of their model.
Re: Re: Re:3 Read the article carefully,
Sounds like you have not actually observed them admitting their evidence for black holes is only indirect, you are merely using your observations as indirect support for your model of their thinking.
Re: Re: Re:4 Read the article carefully,
When the scientists proposing these entities are pinned to the wall, they will use words such as “I believe”, “We think”, “We have not observed but we understand”, etc.
Now if such terms are used in other fields, then it becomes acceptable to have a dissenting view.
After 20 to 30 years of reading and trying to follow the mathematics involved in various models that deal with both sub-atomic and galactic/universe, I want to know what are the basic assumptions that they start their models with.
An interesting phenomenon that occurs in any model development is that certain things are ignored when first developing the model, which may be included later.
If these elements are included at the beginning, the end model can be quite different and the outcomes expected from the model quite at odds with the original development.
Re: Re: Science or Pseudo-Science - How can we tell
Nobody else brought up religion, so why did you?
I mean, can we not discuss science without the “at least it’s better than religion” argument popping up? Seems like it happens every single time. And most of the time it’s in the form of exactly this sort of sneering, throwaway remark.
It’s just tiring.
Re: Re: Re: Nobody else brought up religion
Any kind of belief that ignores fact is religion. I include ?alternative medicine? as a form of religion, for this reason.
Re: because it would be career suicide to partake of these experiments
I don?t think that?s a big problem. For a start, it?s an argument that would mean nothing to some budding young researcher, just starting out, who has no career reputation to speak of.
We saw this with the slow (decades-long) mainstream adoption of quantum physics, for example. I forget which physicist it was who said ?science progresses, one funeral at a time?…
As for funding, look at all the money being freely bestowed on out-and-out crackpot ideas. Surely something with even the remotest sliver of a chance of success would be just as likely to get some rich backer.
Re: Re: because it would be career suicide to partake of these experiments
I have seen enough discussions that would strongly suggest otherwise. The generally espoused comment is that you have to toe the line of what is acceptable or you won’t get funding/tenure/etc. Cross the line of looking at something the bossman doesn’t like and you are in for it. This happens in all areas wether it is business, research, design and development or any other area.
Look into the political affiliations and you will see the reasoning for the backing. What I am talking about is funding from within the halls of science (universities/research centres/etc). Areas that will cause a problem for the purse strings or go against its belief system will not get backing. Again I have seen this is various areas in business, universities, etc.
Re: Re: Re: What I am talking about is funding from within the halls of science
There are no ?halls of science?. It doesn?t matter where the funding comes from, or who discovers the truth. If they can prove it?s the truth, it?s still the truth.
Re: Re: Re:2 What I am talking about is funding from within the halls of science
Religion and Philosophy deal with the Truth. Science deals with understanding the universe about us by building simple (as in simpler than the reality about us) testable models that allow us to work with the universe around us. One of the fundamental aspects of science and the scientific method is falsification.
One of the more devastating attacks against any theory is to say it is not even wrong. So tell me how science deals with the truth? The problem here is that testability requires finding ways to prove it wrong not prove it right (not prove it true).
Unfortunately today, there is a mindset that makes the assumption that science can prove things true. This mindset turns science and the scientific method into a religion instead of it remaining a tool for investigation of the universe around us.
Re: Re: Re:3 Religion and Philosophy deal with the Truth.
If that were true, why are there so many different religions and philosophies? After all, there is only one reality, one truth.
Re: Re: Re:4 Religion and Philosophy deal with the Truth.
You are trying to have your cake and eat it too.
Yes there is only one reality. However, as you are very finite and very small (like everyone else on the planet), you are unable to comprehend reality in any very meaningful way, let alone in its fullest extent. That is why we have unanswerable questions. That is why there are different views about the universe and the meaning of life.
The amount of knowledge and understanding we now have is insignificant to the amount of knowledge and understanding that is available.
My own perspective of what I see of the universe leads me to the conclusion that it was created. That perspective also leads me to want to understand (as far as I am able to) the various aspects of the universe. That perspective also leads me to the understanding that I will never be capable of understanding it, but that there is a great deal of enjoyment in trying. That perspective also leads me to want to know who created it and why.
Science and the scientific method are just one of the tools available to me to gain understanding.
YMMV.
Re: Re: Re:5 you are unable to comprehend reality in any very meaningful way
Is that your meaningful comprehension of reality?
Re: Re: Re:6 you are unable to comprehend reality in any very meaningful way
Let me ask you a question in turn – what is your comprehension of the reality that is around you?
Re: Re: Re:7 Let me ask you a question in turn
Answer mine first.
Re: Re: Re:8 Let me ask you a question in turn
The onus is now on you to present an alternative. I have presented an argument for discussion, I have answered your questions in my discussion. If there is a particular problem with my viewpoint, then point it out.
Show you perspective, otherwise, as has been demonstrated by your various responses, you are unable to critically answer or present an opposing view. You have only thrown “potshots” so to speak. I have taken time to answer you questions, but up to this point, you have not been able to show where my reasoning in incorrect. Nor have you presented anything that says you have considered the problems inherent in the models and have solutions that make the models still valid.
If you have such, present it, otherwise I will take it that you know that you have nothing and are simply using your conversational method to hide that fact.
The ball is in your court. Enjoy your day, and I look forward to reading your discourse on the subject.
Re: Re: Re:9 The onus is now on you to present an alternative
No, you were the one trying to present an alternative, only to have it shot full of holes. You tried to claim I was ? unable to comprehend reality in any very meaningful way?, yet all your arguments are based on your presumed ability to ?comprehend reality in any very meaningful way?. What makes you so special, that you can do something I can?t?
The answer is, you?re not. Your arguments fall at the first hurdle of critical thinking.
Re: Science or Pseudo-Science - How can we tell
Umm… All of that would apply to an experiment like “let’s launch an elephant into orbit to test the theory that elephants can survive unprotected in outer space”.
Please, tell us about one of these proposed experiments. Be specific.
Re: Re: Science or Pseudo-Science - How can we tell
One experiment that comes immediately comes to mind is to test if gravity is an effect of electromagnetic interaction between vibrating dipoles: You can chase up the details yourself.
Re: Re: Re: Science or Pseudo-Science - How can we tell
Considering the fact that gravity goes as one over distance squared, and dipole-dipole interaction goes as one over distance to the fourth power, I’d say that’s considerably less plausible than an elephant surviving in vacuum.
Re: Re: Re:2 Science or Pseudo-Science - How can we tell
Your forget the magnitudes of the respective forces and my question to you is – Have you seen the proposed mathematical model, which interestingly enough gives a distance squared and on the same order of magnitude, only attractive force between the two vibrating dipoles?
An additional part of the initial model is that there is also terms which may relate to momentum effects. But further investigation is required and experiments need to be fully devised and done to see if the model has any opportunity of merit.
Re: Re: Re:3 Science or Pseudo-Science - How can we tell
You seem to know just enough about physics to compose this word salad. That may be good enough to impress the people who buy Deepak Chopra’s books, but I’m glad that Jimmy Wales won’t tolerate such garbage. Maybe you can’t tell the difference between science and pseudoscience, but we can.
Re: Re: Re:4 Science or Pseudo-Science - How can we tell
Have you looked into the model in question? Have you attempted to use Mathematica or Maxima (or other products of their ilk) to do an analysis of the model? Have you thought about what experiments could be undertaken that might shed light on the validity of the model?
If not, why not? If so, what are you results?
If you haven’t actually looked at the model in question, then you have not been able to address its short comings or assumptions. If you have looked at the model, what are the shot comings and assumptions that you have found?
Re: Re: Re:3 Have you seen the proposed mathematical model
Yes. Working out the field from a dipole is a basic exercise in undergraduate physics. You can find it on Wikipedia, if you?re interested.
Re: Re: Re:4 Have you seen the proposed mathematical model
It is also a basic exercise in undergraduate electrical engineering. The proviso for the exercises is that it is a static environment with no dynamic component in the position of the charges. I had to do a number of such exercises way back then and in quite varied configurations.
The model I have referred to looks at the dynamic consideration of opposite charges oscillating in their relative positions, which gives rise to different results. It is the residual effects that are being considered and they are inverse distance squared and very, very tiny on the order of the ratio of gravity to electric field strength and always attractive.
Again, to get back to the point being made. This is an alternative model for the effects of gravity for which various experiments can be formulated that would either show the model is defective and hence not worth looking at or produces results in accordance with the model that would allow further study to be undertaken and hence further experiments to be developed.
This is the application of the scientific method. To simply throw the idea and model out without at least doing some initial experiments that are proposed by the model because it disagrees with the currently accepted theories shows a fairly rigid and closed mindset.
Re: Science or Pseudo-Science - How can we tell
“starts to discuss this and are then labelled as “kooks” because they do not agree with the mainstream ideas”
This is not a new development, it is possibly human nature.
For example, in the past mainstream thinking put the earth at the center and everything revolved around it. This has been proven to be incorrect and yet there remains a few who espouse its veracity.
Do you have any examples of these kooks and their pseudo-science? Generalizations and vague declarations do little to support a contention.
Re: Re: Science or Pseudo-Science - How can we tell
Halton Arp and his challenges to the Big Bang Theory.
Re: Re: Re: Science or Pseudo-Science - How can we tell
Who called him a kook and when?
Re: Re: Re:2 Science or Pseudo-Science - How can we tell
Good grief, try reading some of the things that have happened to him and what has been said to ensure that his results are ignored.
Re: Re: Re:3 Science or Pseudo-Science - How can we tell
Good grief … so no one actually called him a kook?
What, exactly, are you attempting to accomplish in your many comments to this thread? It appears you have some grievance with the treatment of one particular theory and then claim all science is therefore suspect as a result. How very unscientific of you. Do you have other examples where your pet theory has been trashed by scientific gang members?
That should be easy...
You mean like this?
Re: You mean like this?
Yes.
I would assume that geographically diverse authors have a lower degree of overlap in their professional contacts/circles than less diverse authors would, resulting in more people who hear about and cite the work in their own research. That is, the citation index might simply be recording the greater reach of the authors rather than the higher quality of the research in such a case.