For most of us, the review process is difficult. As an academic, having submissions under review is stressful and challenging. Imposter syndrome can rear its head when we receive a scathing critique or we can scratch our head in utter confusion when we receive positive comments, but have a proposal rejected. As a reviewer, it can be equally challenging to provide clear, substantive feedback that acknowledges authorial effort, but provides constructive critique (especially given the limited time we have to devote to the review process).
For AERA, the review process can be especially challenging. As our Writing and Literacies SIG President Kathy Mills notes, “The biggest challenge [in selecting proposals for AERA] is that AERA is highly competitive, and the administrators only allocate a small ratio of sessions to number of submissions. This is outside both the control of the reviewers and the control of the program chairs at AERA. The number of submissions generally increases each year, while the conference runs the same number of sessions over the same number of days….The typical outcome is that more submissions have to rejected than accepted.”
Keeping all of this in mind, we’re hoping this post can provide some helpful guidelines to make the review process more productive for those whose work is under review (whether your proposal is accepted or not) and reviewers alike.
Here are our top 6 tips for #AERA18 reviewers:
- Start with the positive: Point out what the author has done well in the proposal.
- Offer specific critique: Whenever needed, point to specific aspects of the proposal and rubric where more information or clarity is needed. The best reviewers are those who can find something constructively critical to improve most of the papers. This way, even if the paper ends up not being accepted due to the limit of sessions, there are helpful points for the authors to consider in their development and resubmission to other venues.
- Be aware of the length of your review: You don’t want a review that’s too short or too long. Rather, aim for a few sentences that will give the author enough information about your evaluation of the proposal.
- Keep your language respectful and collegial: Amy Stornaiuolo, our SIG’s current Program Chair, recommends that reviewers “imagine you are responding to someone you have met, face to face.”
- Consider the proposal’s fit: As you review, keep in mind the scope of the SIG and include mention of the extent to which you feel this proposal fits within that scope.
- Ensure alignment between the rating and text of the review: Scores should match the language of the review text so that proposal authors and the Program Chair understand how evaluations were determined. It can be confusing (and frustrating) for authors when they receive little to no critical feedback yet receive a low rating. Particularly when giving an average or poor rating, be sure to include sufficient comments on how the paper could be improved.
With these tips in mind, we* offer reviewers a list of review sentence starters that may help them to craft productive reviews:
The topic is
- grounded in relevant literature
- engaging a new perspective
I (appreciate, value, find significant…)
- clear questions, argument and relevant literature review
- clear and appropriate methods for the research questions
- important participant population
- the diversity of perspectives on the topic
Literature Review/ Perspectives
- are timely and relevant
- explore a strong (breadth, depth) of the literature in this area
Methods and findings
- Are clearly described, given word limitations
- Methods are trustworthy
- Analysis is complete or in final stages
- Implications are clear
- Related to conference theme
- Related to ongoing and/or current concerns in the field
Proposal Limitations (be honest, but positive; be sure the scores match the stated evaluation criteria)
The topic is
- interesting but needs focus
- not related clearly to the SIG goals or conference theme
Literature Review/ Perspectives
- lack (breadth, depth)
- could draw from more current literature in the field
- Unclear or incomplete information related to data collection methods, data analysis methods, participants,context, definitions and concepts, researcher role
- Theoretical arguments are not outlined in terms of premises, questions, or directions.
- Unclear or no evidence of data analysis in progress
- Theoretical analysis is not directed toward a claim or expectation of changing a conceptual understanding for researchers
- Data collection and analysis not clearly defined; do not relate clearly to claims or findings
- Not clearly relevant to current concerns of the SIG/ field
- Not clearly relevant to the conference theme
- Very broad
- Needs to be informed by current research in the field
We appreciate the time and energy put into the submission and review process and hope that these tips can make the process more productive for authors and reviewers! Thank you all for the contributions you make to research in Writing and Literacies.
*List of feedback stems adapted from LRA Reviewer Guidelines