During the last several years the pandemic has disrupted our research and teaching schedules. Atusyki Morishima at University of Tsukuba has led an effort to create a resource that documents how the various iSchools coped. This project uses crowd-sourcing to gather links to institutional pages about various responses to the Covid crisis. Many of the “pages are often provided in local languages only” and perhaps for that reason “it is not easy to find them by Web search engines.”¹ In order to insure long-term availability, the project will cooperate with the Internet Archive. Users can search by region to find schools that may interest them. Automatic translation is never entirely reliable, but in this case it is relatively easy to find policy information by using Apple or Google or other translation services. Participation has been unexpectedly high. Terms of statistics: 32 of the Asia-Pacific schools have participated, 40 from the European and African region, and 53 schools from the Americas (north and south).² Anyone may use the database for any purpose, including doing research, developing other databases, and discovering how academic institutions internationally have coped with the crisis. In general the focus is on keeping students and staff safe, through measures such as monitoring, masks, training courses, and of course explicit guidelines. Some schools have set up FAQ sites, and some universities have established testing centres that are free for students and staff. Working from home is also an option at many of the schools. It is important to remember that even though the database is a snapshot in time, the links are live and are likely to change as local conditions change and as new regulations come into force. This kind of project shows what we can accomplish when we collaborate internationally. In so far as international travel is becoming more possible, a database like this is especially important when visiting other universities as guest lecturers, or students. 1: https://www.ischools-inc.org/ 2: https://www.ischools-inc.org/results Cover Image by: Martin Sanchez on Unsplash

The 2022 EDUCAUSE Horizon Report, Data and Analytics Edition appeared recently. EDUCAUSE describes itself as a “higher education technology association and the largest community of IT leaders and professionals committed to advancing higher education.” ¹ While the organisation is North American, it strives to think globally. This report presents “four possible future scenarios for postsecondary data and analytics.” ¹ The first scenario discusses how the contemporary data-driven measurement culture demands external evidence about performance, especially in terms of research, and the report notes that this is a challenge for institutions that focus on “serving the ‘whole student’”. (1) Many iSchools must justify their research productivity based on citation indexes, even at a time when criticism about relying on such indexes is growing. ², ³ The second looks at the consequences of dwindling budgets, which is especially a problem for schools that depend on tuition income. This leaves them “searching for answers on how best to support equitable and accessible data and analytics needs”. ¹ The answer often takes the form of cuts. The third scenario talks about a downward trend in the public’s perception of the value of traditional university degrees, and the competition from “for-profit alternative credentialing centers“.¹ The latter may be a mainly a US phenomenon, and probably focuses on students whose educational goals are largely job-centred. The fourth looks at how a drive for efficiency is leading institutions to improve the health of our global ecosystems by redefining the purposes and uses of physical spaces. The COVID pandemic has made home office and virtual meetings commonplace, but not every student or professor sees this as an overall efficiency improvement. Outside of North America, these scenarios are only partially applicable. For example, there is no real evidence that people in Europe or Asia doubt the value of traditional university degrees. Nonetheless many of these scenarios apply broadly to the iSchools membership, the first especially. 1: ‘2022 EDUCAUSE Horizon Report | Data and Analytics Edition’. 2022. https://library.educause.edu/resources/2022/7/2022-educause-horizon-report-data-and-analytics-edition. 2: Khomyakov, Maxim. 2021. ‘Should Science Be Evaluated?’ Social Science Information 60 (3): 308 - 17. https://doi.org/10.1177/05390184211022101. 3: Cochran, Angela. 2022. ‘The End of Journal Impact Factor Purgatory (and Numbers to the Thousandths)’. The Scholarly Kitchen. 26 July 2022. https://scholarlykitchen.sspnet.org/2022/07/26/the-end-of-journal-impact-factor-purgatory-and-numbers-to-the-thousandths/. Cover image by Robynne Hu on Unsplash

A Retraction Watch email¹ recently highlighted an article by Ben Upton about statistical errors in high impact journals. The criticism is important, because many universities treat impact factors as a de facto correlate for quality when making decisions about faculty hiring and promotion. Upton described the origin of the data about the errors: “The analysis compared statistical errors from just over 50,000 behavioural and brain sciences articles and the findings of replication studies with journal impact factors and article-level citation counts. It found that articles in journals with higher impact factors tended to have lower-quality statistical evidence to support their claims and that their findings were less likely to be replicated by others.” ¹ Even though these publications are not themselves Information Science journals, the implications matter as an information quality issue whenever impact factors are used to measure performance. It would be interesting to know exactly which statistical errors were found in the study, if only to be able to warn students against them. The likelihood is that the errors are not simple mathematics, since most scholars today use statistical packages, but rather errors involving poor sampling or errors in understanding the assumptions required for statistical tests. Regardless of the types of error, the implications go beyond individual cases. Upton warns about “long-standing career inequalities” and cites research by Zachary Horne and Michael R. Dougherty: “Citation counts are known to be lower for women and underrepresented minorities [citations 72–74], and there is some evidence for a negative relationship between impact factor and women authorship [citation 75] and so hiring, tenuring or promoting on their basis may perpetuate structural bias.”² Upton goes on to say: “A European Union-backed agreement on research assessment bars signatories from using impact factors in personnel decisions …” 1: Retraction Watch “RW Daily” email post on 18 August 2022. team@retractionwatch.com 2: Upton, Ben. “Papers in High-Impact Journals ‘Have More Statistical Errors.’” Times Higher Education (THE), August 17, 2022. https://www.timeshighereducation.com/news/papers-high-impact-journals-have-more-statistical-errors. 3: Dougherty, Michael R., and Zachary Horne. “Citation Counts and Journal Impact Factors Do Not Capture Some Indicators of Research Quality in the Behavioural and Brain Sciences.” Royal Society Open Science 9, no. 8: 220334. Accessed August 19, 2022. https://doi.org/10.1098/rsos.220334. Cover image by 愚木混株 cdd20 on Unsplash

The iConference in March 2023 will not be the first to be multilingual, but it will be the first to include papers in three languages in addition to English: Chinese, Spanish, and Portuguese. All of these languages are among the world’s top ten most widely spoken first or second languages. Submissions to the iSchools Doctoral Dissertation Award may also be in the original language of the work, even though the review process also requires a 10-page English-language summary. This means that the quality of machine translation could be an issue in reviewing papers and doctoral dissertations, as well as in reading them after the conference. Machine translation is a complex process that involves the cultural connotations of words as well as their dictionary meaning. Related languages translate more reliably than ones with fewer common roots. Machine translation has long been the subject of scholarly analysis. In a recent paper Irene Rivera-Trigueros from Universidad de Granada “… focused on the specialised literature produced by translation experts, linguists, and specialists in related fields …”. ¹ She found that Google was the most used machine translation service. Another study by Han, Jones, and Smeaton (2021) discusses quality assessment and notes that machine translation “outputs are still far from reaching human parity”. ² The authors look also at human judgement factors: “Human assessors are asked to determine whether the translation is good English without reference to the correct translation. Fluency evaluation determines whether a sentence is well-formed and fluent in context.” ² The German-based translation service DeepL claims to offer significantly greater accuracy than its major competitors. ³ Even so, accuracy does not necessarily mean that a translation has the same persuasive power as the original language. A machine translation may get all the facts right, but still neglect nuances of rhetoric. An awareness of the limitations is important when reviewing and reading. 1: Rivera-Trigueros, Irene, (2022), “Machine translation systems and quality assessment: a systematic review” in Language Resources & Evaluation, 56:593–619 https://doi.org/10.1007/s10579-021-09537-5. 2: Han, Lifeng, Gareth J. F. Jones, and Alan F. Smeaton. 2021. ‘Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods’. arXiv. http://arxiv.org/abs/2105.03311. 3: ‘Why DeepL?’ n.d. Accessed 23 August 2022. https://www.deepl.com/en/whydeepl. Cover image by DeepMind on Unsplash.

People sometimes treat information as context-free, but no set of data or form of information can really be understood without enough environmental details to interpret its meaning accurately. An example can be found in a story by Matt Wood, who wrote about an algorithm that “predicts crime a week in advance, but reveals bias in police response.” ¹ The article goes on to say: “Data and social scientists from the University of Chicago have developed a new algorithm that forecasts crime by learning patterns in time and geographic locations from public data on violent and property crimes. The model can predict future crimes one week in advance with about 90% accuracy.” What is different about this model compared to prior ones was the move away from purely spatial models. “Communication networks respect areas of similar socio-economic background,” writes James Evans in the same article, not just formal boundaries. Context is a major reason for why the algorithm performed better with these data than using other models. The model developers did not worry about street boundaries, but focused on “areas of similar socio-economic background”.¹ They split the topography into tiles of about 93 square meters without respect for neighborhood or political boundaries.“ The model can predict future crimes one week in advance with about 90% accuracy.” The model was also tested using data from seven similarly urban cities with similar results. As data the model used “two broad categories of reported events: violent crimes (homicides, assaults, and batteries) and property crimes (burglaries, thefts, and motor vehicle thefts).” The reason for these categories was their greater likelihood of being reported, which meant that the data would be more consistent and reliable. The goal of the algorithm was not as a tool to encourage police concentration in particular areas, but to give a better tool to let researchers “evaluate police action in new ways”. The algorithm also enables voters and politicians new ways to look at a complex problem. 1: Wood, Matt. 2022. ‘Algorithm Predicts Crime a Week in Advance, but Reveals Bias in Police Response | Biological Sciences Division | The University of Chicago’. 30 June 2022. https://biologicalsciences.uchicago.edu/news/algorithm-predicts-crime-police-bias. Photo by GeoJango Maps on Unsplash

Jonathan Berkheim and Ofir Kuperman published an article on The Dark Side of Research: Research Fraud in which they ask “Why do scientists sometimes falsify experimental results?” The authors quote Elisabeth Bik, who suggests: “Most research misconduct is done by researchers who feel a large pressure to publish, and it is easier to publish nice, positive findings, than complicated stories or negative findings. So if the results are not quite what one had hoped for, it is very tempting to change the results a bit to make them look better.” The authors go on to suggest that the checks and balances that are part of the peer-review system are inadequate to catch problems, because peer review is not really designed to catch fraud. Again Bik: “Most peer reviewers will assume the data they are reviewing is real, and might not think of fraud”. Bik also notes that high-impact journals tend to have less fraud, but she adds: “I sometimes wonder if the authors who publish in high-impact journals are just more experienced and better cheaters. Most misconduct is not visible by just looking at the paper; you have to be sitting in the lab next to the person cheating to be able to catch them.” One standard way to combat some forms of research fraud is for journals to encourage replication experiments. Replications are not popular with authors, because a positive result does little to enhance the author’s reputation. Replications tend to be unpopular with editors because the news value is low. Nonetheless science, especially natural science, builds on the expectation that results are replicable, and the fact of a successful replication should matter in the long run. Unfortunately replication is not simple, especially in the social sciences, where nuances of treatment can affect results without implying fraud. As Bik and the authors suggest, there are no easy solutions, but there are steps that the academic community can take to address the problem, including reducing the pressure on scholars to publish. That would take a structural change in how universities evaluate faculty, and (in some countries) a structural change in how governments distribute university funding. Such changes will take time and pressure by respectable external organisations. 1: Wood, Matt. 2022. ‘Algorithm Predicts Crime a Week in Advance, but Reveals Bias in Police Response | Biological Sciences Division | The University of Chicago’. 30 June 2022. https://biologicalsciences.uchicago.edu/news/algorithm-predicts-crime-police-bias. Photo by Josh Nuttall on Unsplash.

An article published on 30 August 2022 in Springer’s journal on “Science and Engineering Ethics” offers a qualitative analysis of research integrity support in the Netherlands, Spain, and Croatia. As the authors write: “It is particularly important that cross-country studies compare the experience of support from the perspective of the study participants because RI [Research Integrity] support may look different in different countries…” That is true – not just for counties but also for different disciplines. The authors chose these three countries “to represent European countries that have national laws, bodies, and codes governing RI, but which are diverse in terms of research and innovation activities (European Commission, 2017), geographical location, language and culture.” The interviews took place “between Oct 2017 and Feb 2018” and involved a total of 59 people. Such interviews are important because individual experience with integrity issues varies greatly. One problem is that looking broadly across a wide range of fields and cultures makes it hard for any study to offer focused suggestions that are not overly general. One of the problems that the study uncovered is that the “provision of RI education was described as piecemeal, often voluntary, and mostly lacking for senior researchers.” The authors emphasized the need for training at all levels of research staff including doctoral students and technicians. The exact nature of that training is not discussed in the article, and that is unfortunate because training needs to address specifics like plagiarism and data falsification and image manipulation in terms that are directly relevant to the researchers themselves. Any successful Research Integrity training program needs to provide participants with a chance to ask their own questions in order to understand potential integrity problems in ways that do not fade into generalities. One of the study’s conclusions is to put “the emphasis of responsibility for RI on institutions rather than individual researchers.” Academic institutions certainly need to take direct and active responsibility, but one of the risks is that the institution tries to provide a single form of training. The experience of those doing training as part of the Information Integrity Academy is that no single approach makes sense for all fields. 1: Evans, Natalie, Ivan Buljan, Emanuele Valenti, Lex Bouter, Ana Marušić, Raymond de Vries, Guy Widdershoven, and the EnTIRE consortium. 2022. ‘Stakeholders’ Experiences of Research Integrity Support in Universities: A Qualitative Study in Three European Countries’. Science and Engineering Ethics 28 (5): 43. https://doi.org/10.1007/s11948-022-00390-5. Photo by Brett Jordan on Pexels

Science Europe established the Coalition for Advancing Research Assessment (CoARA) in September 2022 and important organisations like the European University Association (EUA) and the European Commission are members of the interim secretariat (2). Some of the principles listed in the document are fairly standard, such as complying with “ethics and integrity rules and practises", and ensuring the "independence and transparency of the data infrastructure and criteria necessary for research assessment". CoARA calls on assessment processes to “respect the variety of scientific disciplines, research types (e.g. basic and frontier research vs. applied research), as well as research career stages”(2). The document goes on to emphasise the primary importance of using qualitative evaluation, including peer review for assessment. Some statements are unusually forceful: “Abandon inappropriate uses in research assessment of journal- and publication based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index”. CoARA goes on to warn that evaluation should “[a]void the use of rankings of research organisations in research assessment” since “the international rankings most often referred to by research organisations are currently not ‘fair and responsible’, the criteria these rankings use should not trickle down to the evaluation of individual researchers, research teams and research units.” This is unusually strong and clear language for associations more accustomed to avoiding controversy by making recommendations so bland that they will offend no one. Many governments and many institutions prefer to rely on metrics like impact factors because they offer a way to make the assessment process seem more neutral, even more scientific, by using externally generated numerical data. What the document does not explain is that anyone using index data needs to understand how the meaning varies from discipline to discipline and methodology to methodology. A high impact factor may mean that scholars regard the research as important, or just that the topic temporarily generates a high level of interest that will wane over time. Implementing these sorts of recommendations will be hard, and ultimately the document calls on institutions to “[c]ommit resources to reforming research assessment as is needed to achieve the organisational changes committed to”. Committing resources is no guarantee of improvement, but it represents the will to change. 1: https://coara.eu/coalition/coalition/ 2: ‘The Agreement Full Text’. 2022. COARA (blog). Accessed 9 October 2022. https://coara.eu/agreement/the-agreement-full-text/.

Four authors, two from Leiden University in the Netherlands and two from York University in Canada, raise the question in the title of their London School of Economics blog entry: “Does increasing standardisation of journal articles limit intellectual creativity?” The article begins by citing Rob Warren’s article with the interesting observation that “researchers in American sociology departments have published almost twice as much in recent years compared to the 1990s.”¹ Trying to maximise their own publications affects “how scientists decide which research projects to pick, which collaborations to seek out, and when research projects should be considered completed.”¹ In a scientometric study, the authors found that “perpetual growth in the production of articles is accompanied by a homogenisation of article characteristics. Articles increasingly converge at around 20 pages, they contain between 50 and 60 references, and they are increasingly the product of collaborative authorship (although sole authorship remains prominent).“¹ One consequence of standardisation appears to be that journals in the past had a greater number of “essays, opinion pieces, and more literary writing”.¹ The standardisation has practical advantages for early-career authors in potentially precarious employment situations: “Juggling term-limited project contracts becomes more manageable when treated as the production of a typical form of output, since it allows for calculating investments and payoffs.”¹ Format “homogeneity” also “reduces effort when resubmitting a manuscript to a journal after an initial rejection…”¹ The authors use the term “black-boxing” as a way of labelling research methods to make reuse easier without having to engage in the details of the method too actively. The space saving can be a plus, but also reduces the opportunity for discussion. In the end the question about whether a higher degree of standardisation affects creativity remains open. A highly structured research paper is easier to read, because the reader can anticipate where to find key information without struggling though long and sometimes poorly written paragraphs. The authors worry nonetheless this could discourage “varied intellectual traditions and concepts”¹ The risk is certainly there, but the blog format that the authors used to present their article may also be one of the ways in which creativity can live on. The question for early career researchers is: would a blog post count? 1: Kaltenbrunner et al. (2022). The great convergence – Does increasing standardisation of journal articles limit intellectual creativity?https://blogs.lse.ac.uk/impactofsocialsciences/2022/10/11/the-great-convergence-does-increasing-standardisation-of-journal-articles-limit-intellectual-creativity/

October 25, 2022, #19 Complaints about the lag between the discovery of integrity problems and official retractions in journals are not new, and now Steve McDonald at Monash University’s health evidence unit Cochrane Australia has provided more data by looking “at retractions among the more than 270,000 COVID-19 papers that have been lodged online since the start of the pandemic. The 212 retracted papers investigated were cited 2697 times, a median of seven times per paper.”¹ A detail worth noting is that even though “almost 90% of citations of these papers referenced the retracted paper without mentioning it had been retracted”¹ apparently “80% were published after the retraction”.¹ This data suggests that the slowness of the formal retraction process plays a more significant role than neglecting to check for retraction notices. The time-gap between submission and publication could be a factor too. Retraction decisions are hard for any publisher. Scholars expect journals to take a decision to retract an article seriously and should expect a systematic review process, since a retraction may be career-damaging, but most journals lack a formal infrastructure for such decisions. The software that publishers use for peer review was not intended for the goals and intensity of a retraction investigation. Retraction reviewers may also need access to confidential data not available to the original reviewers. There are exceptions to the slow pace of retractions, of course. The data from Surgisphere for some COVID papers were so plainly fake after an investigation by the Guardian that the publishers took unusually quick action.² A serious problem is how and whether to update a published paper that has cited retracted works. Flagging the paper is insufficient without greater specificity. In the digital world, changing text in an online journal is easy, but many journals and many scholars find the idea of altering already existing publications too Orwellian for comfort without a formal apparatus for flagging the corrections and giving the reasons. Finding fault is easy. The scholarly world needs to work on establishing a commonly accepted mechanism for handling retractions in a timely and transparent manner. 1: McDonald, Steve. 2022. ‘Retraction Inaction: How COVID-19 Exposed Frailties in Scientific Publishing’. Monash Lens. 17 October 2022. https://lens.monash.edu/@medicine-health/2022/10/17/1385133/retraction-inaction-how-the-pandemic-has-exposed-frailties-in-scientific-publishing. 2: Davey, Melissa, Stephanie Kirchgaessner, and Sarah Boseley. 2020. ‘Surgisphere: Governments and WHO Changed Covid-19 Policy Based on Suspect Data from Tiny US Company’. The Guardian, 3 June 2020, sec. World news. https://www.theguardian.com/world/2020/jun/03/covid-19-surgisphere-who-world-health-organization-hydroxychloroquine.

November 08, 2022, #20 At the ASIS&T Conference in Pittsburgh, Pennsylvania, USA, Sara Lafia presented “A Natural Language Processing Pipeline for Detecting Informal Data References in Academic Literature” on behalf of her co-Authors Lizhou Fan and Libby Hemphill, all from the University of Michigan. Their research uses “natural language processing”¹ to link “thousands of social science studies to the data-related publications in which they are used.”¹ They used their own “Entity Recognition (NER) model”¹ to detect informal references with the goal of “connecting items from social science literature with datasets they reference.”¹ Their main source of data came from ICPSR (the Inter-university Consortium for Political and Social Research), which has data from its founding in 1962 They begin by parsing “full text PDFs into structured text documents”¹, which “retains headers, section titles, tables, figures, and footnotes where data references are likely to be found.”¹ Then they applied their NER in order “to detect dataset entities”¹. As Sara Lafia noted, this is very labour intensive work when done by hand, and makes good use of the thousands of ICPSR datasets. The process lets them “address gaps caused by inconsistent data citation practices and make it possible to detect and analyse data references at scale.”¹ They use human-in-the-loop feedback in order to improve accuracy. In the end they will have created a substantial bibliography of “informal references to research data”¹ that could otherwise easily be overlooked. In the paper’s conclusions, the authors make some key observations, including the fact that “awareness of how users interact with data throughout its lifecycle can support the development of data-driven curation and collection policies.”¹ Developing such policies may seem straightforward, but we have in fact far too little information about how users use data in large-scale archives like ICPSR. They also note that better metrics will not only “better represent the diversity of data use”¹ but “may also inspire novel reuse in the long-term”¹. It is easy to underestimate the importance of this reuse. As the scholarly community increasingly emphasises the value of long-term access to research data, understanding of what the data are and how they have been used is invaluable. 1: Lafia, Sara, Lizhou Fan, and Libby Hemphill. 2022. ‘A Natural Language Processing Pipeline for Detecting Informal Data References in Academic Literature’. Proceedings of the Association for Information Science and Technology 59 (1): 169–78. https://doi.org/10.1002/pra2.614.

November 15, 2022, #21 Predatory journals represent a potential risk especially for early-career scholars without any permanent position. The reason is that predatory publications may not count as legitimate, and could even count negatively as a sign of poor judgment. The topic of predatory journals itself goes at least as far back as the list that Jeffrey Beal maintained but was forced to shut down in 2017. An archived version is available. Simon Linacre’s open-access book, "The Predator Effect: Understanding the Past, Present and Future of Deceptive Academic Journals", addresses the problem directly.¹ Linacre cites a 2017 study by Frandsen, which discusses the reasons “why authors want to publish in the first place at this point, and typically it is for one or more of four reasons: to register an idea or experiment or finding; to certify and validate research; to disseminate that research; and to archive the research for future reference.”¹ Further reasons are “the perceived ease with which publications can lead to promotion or a cynical dissatisfaction with the scholarly communications industry as a whole (Frandsen, 2017).”¹ Linacre tells the story of an author who initially published in a predatory journal and paid the required fee. When he learned that the publication was problematic, “a sympathetic senior academic advised he should publish the article again in a different, more reputable journal.”¹ That made things worse, because publishing the same paper twice is considered an ethical violation. The core problem in this story was everyone’s poor understanding about the consequences of these choices. Predatory journals are one of the consequences of the increasing pressure to publish. In “Global South” the pressure to publish in English is particularly strong. Many authors have no simple source for learning which publishers are predatory, and the threat of lawsuits discourages organizations from publishing such lists. This means that it is all that much more important for universities to provide training about how to recognize predatory publishers, which is one of the topics of the Information Integrity Academy. Unfortunately the definition of a predatory publisher is vague. Among the characteristics are a very quick and very minimal peer review process with little real feedback, and predatory publishers mostly charge for publication. These characteristics only serve as warning signals, not as proof, but authors should take such signals seriously. 1: Linacre, Simon. 2022. ‘Deceptive Academic Journals: An Excerpt from The Predator Effect’. Retraction Watch (blog). 8 November 2022. https://retractionwatch.com/2022/11/08/deceptive-academic-journals-an-excerpt-from-the-predator-effect/. 2: Frandsen, T.F. (2019). Why do researchers decide to publish in questionable journals? A review of the literature. Learned Publishing 32: 57–62. https://onlinelibrary.wiley.com/doi/epdf/10.1002/leap.1214.