Guidelines for Evaluating Digital Scholarship

Download or Print PDF

I. Defining Scholarship in the Academic Study of Religion

Scholarship in the study of religion is currently in a period of rapid change. Established modes of research, teaching, and scholarly communication have been transformed by new technologies and digital research methods. Books, journal articles, and other print-based works are now joined by a plethora of new genres of digital publication, while the proliferation of digital tools have redefined what it means to discover, interpret, and disseminate information.

This document aims to guide researchers, teachers, and administrators in evaluating digital scholarship in the study of religion. Building upon the AAR’s Guidelines for Responsible Research Practices, which notes that “Scholars may share the results of their research in multiple media, including peer-reviewed digital scholarship,” it seeks to promote the evaluation of scholarship in new and emerging media in ways that take into account the distinctive characteristics of digital projects and work. In contrast to print—which tends to be single authored, text-based, and completed upon publication—digital scholarship has several peculiar attributes that must be considered when assessing its merit. First, digital scholarship tends to be collaborative in that it not only can involve several scholars at multiple institutions but can also incorporate a variety of professions such as computer programmers, librarians, and even students. Digital scholarship also tends to be multimodal, utilizing multiple forms of media and even multiple platforms simultaneously. In some work, textual or narrative forms of analysis may take a back seat to more visual or auditory displays of knowledge whose contribution is perhaps made through juxtaposition or other nonlinear forms of argumentation. Finally, digital scholarship is also often open ended. In many instances, the launching of a digital project is the beginning of the work rather than its end, and digital scholarship frequently requires maintenance or regular upgrades. What’s more, exemplary digital projects also often make their findings, content, and data available to an open scholarly commons so that other researchers can build upon, remix, or augment prior work. In general we think “open access” to research is an important value, although we also recognize the important role that copyrights can play in promoting certain kinds of scholarship. Good resources to consult when thinking through issues of open access include SPARC (Scholarly Publishing and Academic Resources Coalition, a US-based initiative promoting open access) and OASPA (Open Access Scholarly Publishers Association, an internationally-focused organization of scholarly publishers in open access).

By offering these guidelines, we do not intend to suggest that digital scholarship should be held to a different set of standards. No matter the medium, scholarship should engage in “disciplined reflection” and “critical examination” as outlined in the AAR’s mission. This includes the rigorous analysis or exploration of primary source material or data; the sustained engagement with an established body of scholarly literature; and a clear articulation of the work’s argument, methodological choices, and original contribution. In other words, quality digital scholarship should also always be quality scholarship. The recommendations below aim to place digital scholarship within this broader conversation of scholarly merit. They should be taken as an invitation to a conversation, and not a fixed set of permanent requirements. Indeed, the rapidity with which digital technology changes will necessitate these guidelines be reevaluated at a future date.

II. Guidelines for Evaluating Digital Scholarship

To aid in the evaluation of scholarship produced in digital form, we offer the following overview of the central attributes of digital scholarship as a framework for evaluation. While not all of these areas will apply to all digital projects, and additional attributes may prove central for the evaluation of some projects, this list provides a description of core elements shared by many of the best scholarly digital works. Although the focus of evaluation is often within the context of tenure and promotion, this framework is intended as a guide for the creation and evaluation of digital scholarship at all levels of academic engagement, from undergraduate research to doctoral dissertations and professional scholarly work. As such, we see this framework as a guide to best practices in digital scholarship of religion, one that can be used for both teaching and evaluation. Because the contribution of digital scholarship is tied to the medium of creation and dissemination, we also outline parameters for how the evaluation of digital scholarship should be conducted, and note external evaluative sources which can help identify and articulate the quality and impact of a digital project.

II.a. Central Attributes of Digital Scholarship

1. Content

As with all scholarly work in the study of religion, it is vital that the content of digital scholarship in the study of religion is academically sound and engages with the current scholarship in the field. While that engagement may take many forms, it is incumbent upon the creators to clearly communicate their contribution to the field.

2. Use of digital media

For many digital projects, the digital medium is a central part of the scholarly work, enabling the creator(s) to advance an argument or mode of analysis that would not be possible in print. Creators of digital projects should include some explicit statement of why the digital medium was selected and how it advances the project goals. Digital projects should be evaluated on the effectiveness of the use of the medium to advance the intellectual goals of the project, considering what the medium enables that would not be possible in other formats.

3. Design

The design and organization of the interface of a digital project is a central part of the communication of its content. The design signals and enables the ways readers interact with the scholarly work. As a result, the design of digital scholarly works should be evaluated on elements such as the clarity and effectiveness of interface design, whether graphical or computational; ease of identifying and accessing information; ease of navigating the resources; adherence to established standards of accessibility and ease of use for all users; and coherence between the design and the argument of the project.

4. The Varieties of Digital Scholarship

Digital scholarship is an expansive term that includes many types or genres of scholarly digital work. In evaluating such work, it is important to take into consideration the genre or genres with which the digital work is engaging. For the purposes of evaluation and review, we recommend the following list of genres of digital scholarship, adapted from the Journal of American History Digital History Review Guidelines, as examples of the current range of digital scholarly work. The nature of digital scholarship is that this list will quickly be dated, but to start the conversation, current types of digital scholarship include the following. This list is not exhaustive, and projects may span multiple genres:

• Archive: a collection of digital primary sources.
• Essay, Exhibit, Digital Narrative: something created or written specifically for the Web or with digital methods, that offers interpretation, narrative, or argument. This category can also include maps, network visualizations, or other ways of representing religion data.
• Teaching Resource: a site that provides online assignments, syllabi, other resources specifically geared toward using the Web; or digital apps for teaching, including educational religion content for children or adults, pedagogical training tools, and outreach to the education community.
• Tool: a downloadable, plugin, app, or online service that provides functionality related to creating, accessing, aggregating, or editing digital religion content (rather than the content itself).
• Gateway/Clearinghouse: a site that provides access to other websites or Internet-based resources.
• Social Media: social media as both subject of academic research and means for sharing/publishing scholarship.
• Digital Community: online social spaces that offer a virtual space for people to gather around a common experience, exhibition, or interest.
• Streaming audio and video (Podcasts): video and audio streaming that engages audiences on religious topics and themes.
• Games: challenging interactive activities that educate through competition or role playing, finding evidence defined by rules and linked to a specific outcome. Games can be online, peer-to-peer, or mobile.
• Data sets, APIs: compilations of machine-readable data, shared in a commonly-accessible format, possibly through a CSV file or an Application Programming Interface (API), or data files, which allow others to make use of this data in their own digital work.
• Computational Analysis: application and assessment of algorithms and statistical models (such as natural language processing, machine learning, stylometry) to the data of religions.
• Interface Design: designing and building interfaces (such as web applications and visualizations) focused on user experience with scholarship in digital media.

5. Audience

In the evaluation of digital projects, it is vital to consider the intended audience of the project. This is often linked to the genre of the work, as projects such as blogs, podcasts, games, and teaching resources are most often aimed at a non-academic audience, whereas digital narratives, data sets, and tools are primarily resources for an academic audience. While insisting on a high scholarly standard regardless, a digital project should be evaluated in relation to the intended audience—whether that scholarship is communicated appropriately for the intended audience and in ways that encourage readers to develop nuanced understandings of religion.

6. Sustainability and Preservation Planning

Unlike traditional monographs and articles, which have a standard form and for which there are established preservation processes, the variety of formats of digital scholarship and their reliance on ever-changing technological infrastructure, together with a lack of established practices around their preservation, necessitates that scholars working in the digital medium attend to the sustainability of their work. While no project can guarantee future stability or availability, digital scholarly works should follow best practices in the use of digital technologies and include a plan for the preservation of the various elements of the digital work, such as data, code, and interface.

7. Collaboration

While not all digital projects are collaborative, many rely on teams to bring together the technical infrastructure, create and manage data, design the interface, and craft the content. Some features of collaborative digital projects may be more labor-intensive than others. The scholar is responsible to explain the different roles each collaborator plays and estimate each collaborator’s contribution. However, evaluators should note that digital projects cannot be easily broken down into percentages that equal a whole. Therefore, collaboration should be evaluated as a process in which scholarship is generated, and not as deemed more or less valuable according to conventional assessment standards like title or author order.

II.b. Parameters for Evaluation of Digital Scholarship

One of the distinctive aspects of digital scholarship is the medium of production and dissemination. Because the medium is central to the construction and argument of the scholarly work, the evaluation of digital scholarship must be done in the work’s native environment and with consideration of the collaborative nature in which digital work is produced. We recommend the following parameters for the evaluation of digital scholarship:

1. Evaluation in the work's native environment

Since digital scholarship is designed for use and presentation in a particular medium, it is imperative to evaluate work in the specific native environment in which it was produced. As the College Art Association/Society of Architectural Historians’ Guidelines put it succinctly, "it is crucial that digital work be seen in the environment for which it was designed. Scholars deserve to have their work taken seriously, including the digital contribution. Hence, all work of digital scholarship must be evaluated in its appropriate environment."

2. Media-specific expertise of committee members

When possible, evaluation committees should have members with expertise or experience in the particular forms of media technologies of which they are assessing. Committees that lack members with expertise in digital scholarship should contract colleagues as media-specific evaluators even if they do not possess content expertise in the project's subject field. In this way the committee can engage and evaluate the strengths and weaknesses work on its own terms.

3. Role of contributor in the creation of project

Digital projects are often collaborative products, the work of teams with a range of areas of expertise, from subject matter to technical organization and implementation. In the evaluation of the digital work of a particular scholar, evaluators should take into consideration both the overall success of the project and the role of the contributor in the creation of that project. It is the responsibility of the scholar engaged in digital work to articulate his or her contributions to the digital project, thereby setting up the parameters of evaluation. Collaborators and evaluation committees should be mindful of the ways in which some scholars, women in particular, have historically not received adequate recognition for contributions to collaborative work.

II.c. Additional Evaluative Sources

As with print scholarship, the receipt of grants and awards for the scholarly work, as well as information about its use, are indicators of the reception of the work within the scholarly community. We recommend that those evaluating digital scholarship consider external sources, such as those mentioned below, in evaluating the impact of the scholarly work.

1. Grants received

Unlike traditional peer-reviewed materials that are assessed upon completion, digital scholarship is often continuously developed and improved. Therefore, grant funding is one clear indicator of the impact or potential of digital work. Securing competitive grants demonstrates that after formal peer-review by funding committees the digital work has shown significant potential and/or proven historical success.

2. External Awards

Awards and other recognitions by professional associations, or scholarly organizations, networks, and collectives further confirms that digital work is making a significant contribution to research.

3. Data regarding use of and engagement with project

Digital scholarship often has metrics that demonstrate the type of engagement they have with audiences. For many projects, instruments such as page views, downloads, plays, number of users, engagement, code or data reuse, as well as other alternative metrics can be used as evidence of its impact. For other types of digital projects, influence can be established through the development of new work based on its technologies and/or design. Assessment should be based on both the quality and quantity of public access, scholarly engagement, and audience use since evaluative metrics will not be standardized across digital work. Altogether, data from a project’s standard measuring mechanisms should be clearly explained in order to evaluate how it generates stimulating technical and theoretical contributions to scholarship.

However, these metrics should not be relied upon as a sole indicator of a work’s importance. Just as book sales are not necessarily indicative of the importance of a scholarly work for the development of the field, a high-quality work of digital scholarship may have low use metrics. In addition, as with all humanities scholarship, the importance of a particular work in the scholarly ecosystem often increases over time and is not easily seen in data that privileges current popularity. As such, we caution against the use of metrics as a shorthand for the value or impact of digital scholarly work, recommending instead that the primary mode of evaluation for digital scholarship be qualitative, and that use metrics be considered as providing a supplemental indicator of the current reach of the work.

III. Digital Scholarship for Promotion and Tenure

III.a. Scholar:

Before beginning and throughout the project, the scholar should consult established departmental and/or other institutional standards for evaluating scholarship for promotion and tenure, and should make plans to explain and document the project’s development, progress, and contributions to scholarship in light of those standards. These plans should be discussed with departmental and committee chairs at each stage to ensure everyone has the same expectations.

Scholars should seek support in preparing tenure or promotion portfolios and making a case for its importance, or in preparing to present “dissertations,” from the AAR, from departments, and from colleagues.

Scholars should be transparent, and keep department and/or committee chair informed of collaborations as they unfold.

Scholars should be aware of the processes of evaluation which their departments, schools, and committees will use in evaluating their projects. Scholars should be aware that their departments may weigh different varieties of digital scholarship differently. Scholars should be prepared to help draft those processes if they are nonexistent or inadequate.

III.b. Institution:

Standards and procedures for evaluating scholarship in cases of appointment, tenure, and promotion vary widely among different institutions of higher education. In any case, to avoid confusion or miscommunication with candidates, those standards and procedures should explicitly address the evaluation of digital scholarship as distinct from traditional scholarship in print media. Specifically:

• Evaluation of digital scholarship should be conducted within the work’s native digital environment rather than in the form of print-outs or scans included in review materials along with other print media.
• Every reasonable effort should be made to identify and select external evaluators who are qualified to review the technological and design aspects of the scholarship (e.g., appropriateness of the medium to the project; choice of database, programming language, and web application framework; quality of back-end and front-end design and build; digital continuity; documentation).
• If possible, internal review committees should also consider including at least one member whose expertise is in the digital technologies of the candidate’s scholarly work, even if that committee member’s research expertise is different from that of the candidate.
• Collaborators (internal or external) on the candidate’s project should provide evaluations of the candidate’s contributions to the larger process, including responsibilities in curating and negotiating the relationships of the project team.
• There should be clear standards for measuring evidence of the scholarly influence of the project. Metrics might include: (1) awards, reviews, and other forms of academic peer recognition; (2) grants, fellowships, and other project funding; (3) quality and quantity of access and use by other scholars and students (including links to it in other projects); and (4) evidence of influence in terms of the development of new projects based on its technologies and/or design.
• In addition to the work itself, the candidate’s statement on research should include details that describe and document the project as a means of enabling the committee to evaluate it as both product and process of academic research. This statement might include the following elements, as applicable: (1) the relevance of the project for scholarship, teaching, or research; (2) contributions of the project to scholarship in the field; (3) evidence of originality or innovation; (4) broader applications of the project in other fields; and (5) a project narrative of the process of preparing and completing the project, including benchmarks for evaluation and progress, descriptions of hardware and software requirements for viewing and participating in the project, and the names and contributions of collaborators.
• At certain points in the course of a candidate’s progress toward tenure and/or promotion (e.g., annually or at the time of a third-year review), the candidate might offer a demonstration to department faculty of the project's development process. Reports on and/or recordings of these demonstrations could then become part of the candidate’s file. Such demonstrations could be especially important when evaluating projects that have no clear endpoint or launch date.

IV. Selected Resources and References

This selected list of resources below are either cited in the document above, or served as guides for the crafting of these guidelines.

American Historical Association. “Guidelines for the Professional Evaluation of Digital Scholarship by Historians.”
https://www.historians.org/teaching-and-learning/digital-history-resources/evaluation-of-digital- scholarship-in-history/guidelines-for-the-professional-evaluation-of-digital-scholarship-by-historians

Cantwell, Christopher D., and Hussein Rashid. “Religion, Media, and the Digital Turn: A Report for the Religion and the Public Sphere Program.” Social Science Research Council. December 2015.
http://www.ssrc.org/publications/view/religion-media-and-the-digital-turn/

Cavanagh, Sheila. “Living in a Digital World: Rethinking Peer Review, Collaboration, and Open Access.” Journal of Digital Humanities 1, no. 4 (Fall 2012).
http://journalofdigitalhumanities.org/1-4/living-in-a-digital-world-by-sheila-cavanagh/

College Art Association and the Society for Architectural Historians. “Guidelines for the Evaluation of Digital Scholarship in Art and Architectural History.” January 2016. http://www.collegeart.org/pdf/evaluating-digital-scholarship-in-art-and-architectural-history.pdf

Cooper, Danielle, Roger C. Schonfeld, Richard Adams, Matthew Baker, Nisa Bakkalbasi, John G. Bales, Rebekah Bedard, et al. "Supporting the Changing Research Practices of Religious Studies Scholars." Ithaka S+R. Last Modified 8 February 2017. https://doi.org/10.18665/sr.294119

Fishar, Michelle Millar. “Case Studies and Examples for Evaluating Digital Scholarship.” CAA News Today. February 23, 2016.
http://www.collegeart.org/news/2016/02/23/case-studies-and-examples-for-evaluating-digital-scholarship/

Gala, Alex, Jason Heppler, and Douglas Seefeldt. “A Call to Redefine Historical Scholarship in the Digital Turn.” Journal of Digital Humanities 1, no. 4 (Fall 2012). http://journalofdigitalhumanities.org/1-4/a-call-to-redefine-historical-scholarship-in-the-digital-turn/

George Mason University Department of History and Art History. “Digital Dissertation Guidelines.”
http://historyarthistory.gmu.edu/graduate/phd-history/digital-dissertation-guidelines

Harris, Katherine D. "Explaining Digital Humanities in Promotion Documents." Journal of Digital Humanities 1, no. 4 (Fall 2012).
http://journalofdigitalhumanities.org/1-4/explaining-digital-humanities-in-promotion-documents-by-katherine-harris/

Mandell, Laura. "Promotion and Tenure for Digital Scholarship." Journal of Digital Humanities 1, no. 4 (Fall 2012).
http://journalofdigitalhumanities.org/1-4/promotion-and-tenure-for-digital-scholarship-by-laura-mandell/

Modern Language Association. “Guidelines for Evaluating Work in Digital Humanities and Digital Media.”
https://www.mla.org/About-Us/Governance/Committees/Committee-Listings/Professional-Issues/Committee-on-Information-Technology/Guidelines-for-Evaluating-Work-in-Digital-Humanities-and-Digital-Media

Nowviskie, Bethany. “Evaluating Collaborative Digital Scholarship (or, Where Credit is Due).” Journal of Digital Humanities 1, no. 4 (Fall 2012).  http://journalofdigitalhumanities.org/1-4/evaluating-collaborative-digital-scholarship-by-bethany-nowviskie/

Organization of American Historians. “Digital History Reviews.” Journal of American History.
http://jah.oah.org/submit/digital-history-reviews/

Pells, Rachael. "Understanding the Extent of Gender Gap in Citations." Inside Higher Ed. August 16, 2018.
https://www.insidehighered.com/news/2018/08/16/new-research-shows-extent-gender-gap-citations

Presner, Todd. “How to Evaluate Digital Scholarship.” Journal of Digital Humanities 1, no. 4 (Fall 2012).
http://journalofdigitalhumanities.org/1-4/how-to-evaluate-digital-scholarship-by-todd-presner/

Rockwell, Geoffrey. "Short Guide to Evaluation of Digital Work." Journal of Digital Humanities 1, no. 4 (Fall 2012).
http://journalofdigitalhumanities.org/1-4/short-guide-to-evaluation-of-digital-work-by-geoffrey-rockwell/

Smithies, James. "Evaluating Scholarly Digital Outputs: The Six Layers Approach." Journal of Digital Humanities 1, no. 4 (Fall 2012).
http://journalofdigitalhumanities.org/1-4/evaluating-scholarly-digital-outputs-by-james-smithies/

Authored by:

Theodore Vial, Iliff School of Theology

Timothy Beal, Case Western University

Christopher Cantwell, University of Wisconsin–Milwaukee

Kristian Petersen, Old Dominion University

Jeri E. Wieringa, George Mason University
 

Adopted by the AAR Board of Directors, September 2018