Success in researching: Looking for high quality evidence
Managing research information
The acquisition of reliable research information requires distinction of source opportunities, methodology in researching and strategy in identifying the quality of evidence. Establishing your approach to a chosen topic with a truthful representation of evidence is essential to achieve the standards of intellectual and moral excellence.
Sourcing good opportunities
(Listed from least reliable to most reliable sources)
Primary sources
Original documents on which other research is based.
Examples are:
- experimental trials; randomised controlled and derivatives
- quasi-experimental studies; non-randomised control studies, before and after studies and interrupted time series
- original research studies; observational (cohort, case-control, case series), retrospective, prospective
double-blind randomised controlled trial
- proceedings of meetings, conferences and symposia
- dissertations (master's degree)
- survey research
- data sets that have been tabulated but not interpreted
- thesis (doctorate degrees)
- interview transcripts
- technical reports
- websites
- patents.
Secondary sources
Interpretations or analyses of primary sources.
Examples are:
- meta analysis articles
- systematic review articles
- review articles
- textbooks
- commentaries
- bibliographies
- database; compilations, indexes
- websites
- article indexes.
Tertiary source
A distilled and collection of information from primary and secondary sources.
- encyclopaedias
- almanacs
- manuals
- textbooks
- medical dictionary
- fact books
- research Quickstarts
- library Course Pages
- pathfinders.
Reliable websites include NIH, Web MD, Mayo Clinic, MedlinePlus, Johns Hopkins, HON as well as government-run websites.
Methodology research and design
Traditional narrative review
Systematic quantitative literature review
Methods of meta-analysis
Traditional reviews
Usually one or more experts in the topic of research presenting a less formalised method enabling summarisation of amounts of information. It accepts differences in questions, designs and context (Booth, Sutton & Papaioannou, 2016).
The approach is to focus on advancing theoretical models, offer perspectives on controversial issues, improve best practice and identify emerging issues (Rumrill Jr and Fitzgerald, 2001).
The broad overview is developed without a hypothesis, study protocol, search strategy or clear methodological approach.
Sometimes evidence based findings, however, those may be prone to bias, with subjective conclusions of findings.
Systematic quantitative literature review
Systematically searching for literature for primary research (published reports and papers) on a specific topic. The explicit, methodological approach requires data entry of reports into a spreadsheet and the best information of the papers providing evidence are summarised using a table to report the findings.
Mapping the literature identifies and structures relevancy of information. Mapping is achieved with information acquired; a systematic method in surveying literature, quantitative measurement of components of research in the field, comprehensive assessment of locations, subjects, variables and responses, structured collection and analysing of literature (Pickering, n.d.).
The analysis is comprehensively narrated to report the current status of research (with the attempt to minimise bias).
Important geographic, scalar, theoretical and methodological gaps can be found in the literature. The benefits can provide better design and justification of the research (Pickering & Byrne, 2014).
Methods of meta-analysis
Conducted by a team of expert, enabling analysis of research studies of a diverse population.
Study design is used to systematically assess research aiming for precise estimates to derive conclusions about the research literature. Critical examination of variability and heterogeneity is required. The benefits enable quantitative review of a large, complex amount of literature. A majority of research is required when identifying existing studies to lead to appropriate conclusions. Large bodies of research now enable opportunities for this method to be feasible (Haidich, 2010).
Simplify your search: Efficiency and action
Searches of subjects and keywords on general and subject specific databases will give you many opportunities. A direction of aim is needed to focus and find relevant information.
Identify the information you need from your sources.
Scope—specific or broad
Information source—primary, secondary or tertiary
Information in context—current or retrospective
Information Level—general, partially technical, highly technical
Ranked information—statistical or non-statistical
Plan a search strategy
Check important features in literature like:
- key research question
- information relevancy
- location of the research
- timing of research
- methodology, analysis and results
- limitations
- researchers results
- research still unexplored
- expert authors (credentials, sources and publishing on similar topics).
Searching
Search with a broad search engine using keywords (change order of words).
Search using specific healthcare research search engines
Reduce to websites with information from organisations.
Search for studies with impact and relevance.
Search in images or news categories (in Google).
Search with keywords and with the word "journal" or "article".
Search with keywords and "pdf".
Use healthcare research search engines
The following is a partial list of some of the tools that the top internet healthcare researchers use.
Online research: Information to consider
Title relevancy to your topic
Demner-Fushman, Hauser & Thoma, (2005) found in their study 92% of titles were effectively informative for classifying clinical publication. Titles were not sufficient for classification of research quality.
The use of words in titles, keywords and abstracts are important for clear and accurate article registration and for representation in search engines (Bonciu, 2005).
Scientific and nonscientific studies
Effective communication between scientific and nonscientific studies are essential in achieving open, democratic policies (Mikulak, 2011). Nonscientific studies offer information relevant to reporting scientific research to a. larger audience of readers.
Relevancy of information of article published
Measuring the impact and relevance of scientific research can be achieved throughout the progression of science into practice.
Sutherland, Goulson, Potts and Dicks (2011) identify phenomenon effecting relevance are:
- societal impacts
- research required facts
- discovery to application time
- politics opportunities
- knowledge transferable to public
Preprinted papers
Preprints is a completed draft of a scientific manuscript without peer review.
Bourne, Polka, Vale and Kiley (2017) suggest preprints are utilised more when licensed and formatted to facilitate reuse. Timestamped and made open to access enables quicker online access transparency in viewing the order of priority relative to other published work.
Preprint are scholarly outlets for work with meaningful results that may not be considered acceptable for journal publication. The quality can be significant. Rapid evaluations can be made to correct errors and make improvements.
Preprints present information that can be used for new and innovated research. Evidence can substantiate and support proposals in the grant application and development in the field.
Scientific abstracts from meetings
Reyes, Andresen, and Palma, (2011) defines scientific abstracts as "indispensable tools" essential in disseminating the latest research. Some databases do not index individual abstracts from a scientific meeting. A survey by BioInformatics LLC & PJA (2008) found 77% of life scientists participated in some type of social media. Of those participating, 85% were understood it influenced their decision-making.
Social technology for collaborative resource management
McKinsey Global Institute (2012), estimates suggest that by fully implementing social technologies, companies have an opportunity to raise the productivity of interaction workers—high-skill knowledge workers, including managers and professionals—by 20 to 25 percent."
Email: A system for productivity
Brooks (2006) found a two month survey of physicians with 4203 participants reported almost two thirds (64%) were using email to contact other healthcare professionals (cited in Goyder, Atherton, M. Car, Heneghan & J. Car, (2015).
Social media: method for finding collaborative contacts
Participation in social media by the general public has increased from the years (2005- 2014) significantly by (8% to 72%). Van Noorden (2014) identified scholars use social media to comment on research (40%), actively discuss research (20%), contact peers (approximately 15%) and discover recommended papers (40%).
A QuantiaMD study found 65% of physicians use social media sites for professional reasons.
British Medical Journal Altmetric offers breakdown information to assist professionals in measuring the online impact of your research with the use of social media.
Blogs are used for research collaboration of professionals and dissemination of information. In a study by Miller and Pole (2010), of 951 health-related blogs, 43% of bloggers were physicians. Postings of external links were relevant and existed in (90.2%) of the blogs.
Librarians and information professionals contribute to medical research teams by helping them make more efficient use of information using Twitter; keeping current research and facilitate networking with professional colleagues (Ford & Tolmie, 2015; Glenn & Rolland, 2010).
Since the previous report (2012), Facebook users have increased from one billion (one-seventh of the worldwide population) to two billion (2017) monthly active users (cited in Constine, 2017).
Adam, Manca, and Bell (2016) found using Facebook improved recruitment rates and was cost effective when advertising for their randomized controlled trial. It is also used for course platforms enabling networking of academic peers, faculty and medical professionals (Guy, 2012).
Lupton (2014) survey used social media networks and identified in a group of (97%) using social media as part of their professional academic work; Twitter (90%), LinkedIn (60%), Facebook (42%), Google+ (21%).
In the U.S. LinkedIn reached over 40% of Physicians and Surgeons to collaborate (Dunlap, 2014). It has shown to be the most popular professional networking site (cited in Ventola, 2014).
Evidence profiling
Outcomes should not be taken into consideration when grading the overall quality of literature evidence.
Cochrane Databases of systematic reviews
The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) Working Group, have established a path to evaluating evidence quality. Their approach to "Grading quality of evidence and strength of recommendations," enable assessment of the judgments effecting recommendations (Grade Working Group, 2004).
Summary
The element of need for systematic review of evidence are:
- study design
- study quality
- consistency
- directness.
Lowest quality of evidence of outcomes is essential in judging overall quality evidence of the study.
Judgements about the strength of recommendations are dependant on the;
- benefit-to-harm balance
- evidence quality
- application
- baseline risk.
Study design ranking
The presence of logical arguments and empirical evidence should support the aim of the study.
Lowered ranking quality requires explicit reasoning.
Important consistencies
Factors identifying importance existing consistencies are the:
- effect direction
- effect size and differences
- significance of the differences.
Where important inconsistencies are evident, a description should be given to separate the estimates of the magnitude of effect for different subgroups.
Direction of evidence
Important differences indirectness are determined by the size of the effect on:
- people
- interventions
- outcomes.
Indirect evidence can offer information identifying no direct comparisons of interventions and investigators, however, direct evidence provides greater certainty (BMJ, 2007).
Torrens University Australia, SSNT Think Education Library, shows an extract from the presentation "How to search the medical literature on the net" (cited in Haffar, 2013). The categorisation from The Oxford Centre of Evidence Based Medicine (n.d.) identifies validity/strength of inference and the bias associated with the levels of evidence (I - V).
Engaging with literature: Higher levels of evidence
Approach quality of evidence
Strategy and an organised method is needed for sound judgment and understanding of benefits and harms of clinical practice (Coleman, Talati, White, 2009).
Clinical studies providing the best evidence
Randomised controlled trial and systematic reviews those trials ensure a high standard for judging the benefits of treatments. Non-randomised (observational) studies have unrecognised invalidating factors interfering with identification of differences between study groups (Barton, 2000).
Hierarchy of importance (most to least)
Randomised controlled trials and derivatives
Controlled observational studies in the middle
Uncontrolled studies and opinions
A flexible approach is advised by Barton (2000). to complement randomised controlled trials and observational studies.
Low bias, quality and strength
The Major Sources of Bias in Clinical Research (Pannucci & Wilkins, 2010) needing to be assessed include trial planning, trial implementation and data analysis.
An estimation of up to 85% of research is "wasted" and avoidable (Chalmers and Glasziou, 2009).
Following this claim Yordanov et al. (2015) performed a simulated study correcting incomplete reporting and identified avoidable waste (42%). Problems were addressed; elimination of patients from the analysis (35%), blinding of patients and reported outcome (19%), blinding when using comparisons (16%), and inadequate methods of intervention when data is missing(15%).
Hewitt, Hahn, Torgerson, Watson & Bland (2005) discovered trials (40%) published in major medical journals used inadequate allocation concealment or did not describe the concealed allocation.
The Centre for Evidence Based Medicine Outcome Monitoring Project (COMPare), researched outcomes of articles (based on impact factor) published in the—New England Journal of Medicine, Lancet, JAMA, The BMJ, and Annals of Internal Medicine (October 2015 to January 2016. Of those articles, protocols and registries, pre-dating trial launch, showed (13%) primary and secondary outcomes were the same (Ioannidis, Caplan and Dal-Ré, 2017).
Adjustments could improve research studies and publications by decreasing the risk of bias from high risk to low risk bias.
Suggested Reading
References
Adam, L. M., Manca, D. P., and Bell, R. C. (2016). Can Facebook be used for research? Experiences using Facebook to recruit pregnant women for a randomized controlled trial. Journal of Medical Internet Research, 18(9), 250. doi:10.2196/jmir.6404
Barton, S. (2000). Which clinical studies provide the best evidence? British Medical Journal, 321(7256). Rerieved from https://www.researchgate.net/publication/12405006_Which_clinical_studies_provide_the_best_evidence_The_best_RCT_still_trumps_the_best_observational_study
BioInformatics LLC & PJA. (2008). The new collaboration: Social media and the life science opportunity. Retrieved from https://www.upf.edu/pcstacademy/_docs/BioInfoSurvey.pdf
Bonciu, C. (2005). Title, abstract and keywords: Essential issues in medical bibliographic research. Revista Medico-Chirurgicala a Societatii De Medici Si Naturalisti Din Iasi, 109(4). Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/16610196
Booth, A., Sutton, A., & Papaioannou, D. (2016). Systematic approaches to a successful literature review. Steele, M. (Ed.). (2nd ed.). Croydon: SAGE Publications Inc.
Bourne, P. E., Polka, J. K., Vale, R. D., & Kiley, R. (2017). Ten simple rules to consider regarding preprint submission. PLoS ONE, 13(5), doi:10.1371/journal.pcbi.1005473
Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. The Lancet, 374(9683), 86–89. doi:10.1016/S0140-6736(09)60329-9
Coleman, C. I., Talati, R., & White, C. M. (2009). Clinician's perspective on rating the strength of evidence in a systematic review. Pharmacotherapy: A pathophysiologic approach, 29(9), doi:10.1592/phco.29.9.1017
Constine, J. (2017, June). Facebook now has 2 billion monthly users… and responsibility. Retrieved from https://techcrunch.com/2017/06/27/facebook-2-billion-users/
Demner-Fushman, D., Hauser, S., & Thoma, (2005). The role of title, metadata and abstract in identifying clinically relevant journal articles. American Medical Informatics Association Annual Symposium Proceedings Archive, 2005. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1560462/
Dunlap, M. (2014, April). Leave Gray’s Anatomy on the bookshelf: How modern healthcare professionals engage with social media. Retrieved from https://business.linkedin.com/marketing-solutions/blog/l/leave-grays-anatomy-on-the-bookshelf-how-modern-healthcare-professionals-engage-with-social-media
Ford, C., & Tolmie, D. (2015). Breaking the limits of time and space: How Twitter is helping #medlibs collaborate and communicate. A descriptive study. Journal of Hospital Librarianship, 16(2), 116-131. doi:10.1080/15323269.2016.1150737
Glenn, E., & Rolland, B. (2010). Librarians in biomedical research: New roles and opportunities. Information Outlook, 4(7). Retrieved from https://www.sla.org/wp-content/uploads/2013/01/2010_OctNovResearchGrant_Article.pdf
Goyder, C., Atherton, H., Car, M., Heneghan, C. J., & Car, J. (2015). Email for clinical communication between healthcare professionals. Cochrane Database Systematic Reviews, 20(2). doi:10.1002/14651858.CD007979.pub3
Grade Working Group; Atkins, D., Best, D., Briss, P. A., Eccles, M., Falck-Ytter, Y., Flottorp, S., Guyat,t G. H., Harbour, R. T., Haugh, M. C., Henry, D., Hill, S., Jaeschke, R., Leng, G., Liberati, A., Magrini, N., Mason, J., Middleton, P., Mrukowicz, J., O'Connell, D., Oxman, A. D., Phillips, B., Schünemann, H. J., Edejer, T., Varonen, H., Vist, G. E., Williams, J. W. Jr., Zaza, S. (2004). Grading quality of evidence and strength of recommendations. British Medical Journal, 328,1490. doi:10.1136/bmj.328.7454.1490
Guy, R. (2012). The use of social media for academic practice: A review of literature. Kentucky Journal of Higher Education Policy and Practice, (1)2. Retrieved from http://uknowledge.uky.edu/cgi/viewcontent.cgi?article=1015&context=kjhepp
Haidich, (2010). Meta-analysis in medical research. Hippokratia, 14(Suppl. 1). Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3049418/
Haffar, S. (2013). "How to search the medical literature on the net" [Online presentation]. Retrieved from http://www.slideshare.net/shaffar75/how-to-search-the-medical-literature-on-the-net
Hewitt, C., Hahn, S., Torgerson, D. J., Watson, J., Bland, J. M. (2005). Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. British Medical Journal, 330,1057. doi:10.1136/bmj.38413.576713.AE
Ioannidis, J. P. A., Caplan, L., & Dal-Ré, L. (2017). Outcome reporting bias in clinical trials: why monitoring matters. British Medical Journal, 356. doi:10.1136/bmj.j408
McKinsey Global Institute. (2012). The social economy: Unlocking value and productivity through social technologies. International, Chui, M., Manyika, J., Bughin, J., Dobbs, R., Roxburgh, C., Sarrazin, H., Sands, G., & Westergren, M.
Mikulak, A. (2011). Mismatches between 'scientific' and 'non-scientific' ways of knowing and their contributions to public understanding of science. Integrative Psychological and Behavioral Science. 45(2), 201-215. doi:10.1007/s12124-011-9157-8
Miller, E. A., & Pole, A. (2010). Diagnosis blog: Checking up on health blogs in the blogosphere. American Public Health Association, 100(8). doi:10.2105/AJPH.2009.175125
Oxford Centre for Evidence-based Medicine – Levels of Evidence (March 2009). (n.d). Retrieved September 25, 2017, from the Centre for Evidence-Based Medicine, Oxford, Centre for Evidence-Based Medicine Web site: http://www.cebm.net/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/
Rumrill, P. D. Jr., & Fitzgerald, S. M. (2001). Using narrative literature reviews to build a scientific knowledge base. Work,16(2). Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/12441470
Pannucci, C. J., & Wilkins, E. G. 2010. Identifying and avoiding bias in research. Plastic and Reconstructive Surgery, 126(2). doi:10.1097/PRS.0b013e3181de24bc
Pickering, C. (n.d.). Systematic quantitative literature reviews. What are they and why use them? [PowerPoint slides]. Retrieved from https://www.griffith.edu.au/__data/assets/pdf_file/0004/504904/What-are-systematic-quantiative-reviews-large-format-slides.pdf
Pickering, C. & Byrne, J. (2014). The benefits of publishing systematic quantitative literature reviews for PhD candidates and other early-career researchers. Higher Education Research and Development, 33(3), 1-24. doi.org/10.1080/07294360.2013.841651
Reyes, B. H., Andresen, H. M., & Palma, H. J. (2011). [The relevance and fate of abstracts presented in scientific meetings]. Revista Medica De Chile,139(1). doi: /S0034-98872011000100001
Sutherland, W. J., Goulson, D., Potts, S. G., & Dicks, L. V. (2011). Quantifying the impact and relevance of scientific research. PLoS ONE, 6(11). doi:10.1371/journal.pone.0027537
University of Canberra. (2014). ‘Feeling better connected’: Academics’ use of social media.
Canberra: Lupton, D.
Van Noorden, R. (2014). Online collaboration: Scientists and the social network. Nature, 512(7514). doi:10.1038/512126a
Ventola], C. L. (2014). Social media and health care professionals: Benefits, risks, and best practices. Pharmacy and Therapeutics, 39(7). Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4103576/
Yordanov, Y., Dechartres, A., Porcher, R., Boutron, I., Altman, D. G., & Ravaud, P. (2015). Avoidable waste of research related to inadequate methods in clinical trials. British Medical Journal, 350. Retrieved from http://www.bmj.com/content/350/bmj.h809
