Project Research Update   | watch

 

10 June 2011 - The final assessment went out to both Wikipedians and policy assessors!

15 April 2011 - The first assessment for spring term is nearly complete. THANK YOU Assessment Team!!! The second assessment round will go out next week. The spring term has a lot of interesting topics and many participants, so it looks like the project is having a substantial impact on public policy content.

24 January 2011 - A summary of results from the metric assessment is posted. A full report is coming soon to the outreach wiki.

22 December 2010 - The final round of assessments for this semester is ready. This one will work like the previous round: assessors choose the articles. This is the same set of articles from the Student Pre Assessment; look for big improvements in the articles!

13 December 2010 – The student post assessment requests will be posted here next week, as students are finishing classes this week. Researcher Amy is back from maternity leave and working to get caught up. For assessment team members: email Amy if you would like some recent pics of her daughter!

18 November 2010 – Another round of assessments is available. This round measures the quality of articles students are working on at the revision prior to when the students made their first edits.

3 November 2010 – The next round of assessments is out! Active assessors should see a message on their talk pages. This time, we're letting assessors pick the topics from a randomly selected list.

20 October 2010 – LiAnna Davis (User:Ldavis (Public Policy)) is temporarily taking over assessment while Amy Roth is on maternity leave. All current assessments should be completed by the end of this week. Look for new assessments in November.

14 October 2010 – Preliminary data analysis shows consistent assessment results among Wikipedia reviewers. The next step of the first assessment is to compare Wikipedian results to subject matter expert results. More info.

Article quality assessment is the primary research goal for the WikiProject: United States Public Policy page. As the basis for evaluation overall article quality improvement, article assessment is essential to the Public Policy Initiative (PPI).

Are you interested in research and assessment or just looking to help out? Here's a list of tasks with varying skill and commitment levels, so everyone can participate!

• Tag articles to be included in WikiProject United States Public Policy (WP:USPP):

1. Click on the edit page of an article relating to United States Public Policy
2. Copy the text between the squiggly brackets.
3. Paste the text at the top of the edit page below other project tags.
4. Preview your changes.
5. Save your tag!

• Edit articles.
  

A fall participant in the project said, "If you know something about Absentee ballot, and you edit that page, since Wikipedia is always in the top 5 of a Google search, then everyone who ever looks up Absentee ballot will be better off because you improved that article."

• Assess articles within WP:USPP
  

Follow the steps and watch the tutorial video below.

• Join the Assessment Team.

This team's work is proving that Wikipedians have the highest and most consistent standards of article quality.

Contact ARoth (Public Policy Initiative) (talk · contribs), Amy is the Research Analyst for the project.

At the start of the project, participants worked together to create a quantitative metric so that we could measure improvement in article quality. This quantitative metric shows consistent results among Wikipedians, results in ratings that align with the 1.0 Assessment ratings, and most importantly captures Wikipedia principles, especially neutrality and article quality.

There are two ways to tag articles for WP:USPP, for the article quality rating metric described on this page, paste this code on an article's talk page:

{{WikiProject United States Public Policy
|class = <!-- a class specified here overrides the automatic rating based on the numerical scores -->
|importance = <!-- this works just like the usual WikiProject importance ratings -->
 |comprehensiveness = <!-- 1-10 -->
 |sourcing = <!-- 0-6 -->
 |neutrality = <!-- 0-3 -->
 |readability = <!-- 0-3 -->
 |formatting = <!-- 0-2 -->
 |illustrations = <!-- 0-2 -->
}}

which produces output like this:

You can also use the standard assessment ratings to tag articles using just the class parameter instead of the quantitative metric tag:

{{WikiProject United States Public Policy
|class=
|importance=
}}

This rubric is based Wikipedia's policies and expectations for high-quality articles. It has detailed breakdowns of scores for different aspects of article quality, but it also can translate into the standard Stub/Start/C/B scale and thus feed into the 1.0 assessment system without too much duplicated effort. The language is for what is expected for high-quality articles is mostly adapted from the featured article criteria.

The article covers all significant aspects of the topic, neglecting no major facts or details and placing the subject in context. Any score from 1 to 10 is possible.

• The article is comprehensive, going into appropriate detail about all significant aspects of the topic, and using summary style where appropriate. - 10 points
• The article is mostly comprehensive but falls short in one or more significant aspects of the topic. - 7 points
• The article is well-developed in some aspects but requires major expansion in others. - 4 points
• The article goes beyond a preliminary introduction, with at least some detail beyond a brief overview, but is far from comprehensive. - 3 points
• The article is a stub, consisting of only a paragraph or two of brief introduction to the topic. - 1 point

The article is well-researched. It is verifiable and cites its sources, with inline citations to reliable sources for any material that is likely to be challenged and for all quotations. Any score from 0 to 6 is possible.

• The article is well-sourced, such that readers can determine which information comes from which source. The most appropriate source are used, including journal articles and scholarly monographs where possible. - 6 points
• The article is mostly well-sourced, but has some material that is not sourced or does not use the most appropriate sources. - 4 points
• A significant portion of the article is well-sourced, but the majority of it is not adequately sourced. - 2 points
• The article contains only a bibliography, or only a small portion of the article is well-sourced. - 1 point
• The article does not reference any reliable sources. - 0 points

The article has a neutral point of view, accurately representing significant points of view on the topic without advocating or placing inappropriate weight on particular viewpoints.

• The article follows the NPOV policy. - 3 points
• The article follows the NPOV policy, with only minor exceptions. - 2 points
  • Minor exceptions include subtle imbalances in the ways different comparably significant viewpoints are described, the exclusion of minor but still significant viewpoints when all major viewpoints are covered, etc. Such an article is neutral on the whole, but may have a few small problem areas.
• The article mostly follows the NPOV policy for the viewpoints represented, but other major viewpoints are absent - 1 point
• The article falls significantly short of following the NPOV policy. - 0 points

The prose is engaging and of a professional standard, and there are no significant grammar problems.

• The article has excellent style and grammar and is highly readable. - 3 points
• The article is comprehensible and reasonably clear, but a need for copy editing is apparent. - 2 points
• The organization, style and/or grammar of the article detract significantly from the reading experience. - 1 point
• The article is difficult to understand and requires a thorough re-write. - 0 points

The article is organized and formatted according to Wikipedia standards and generally adheres to the manual of style.

• The article is well-formatted and is mostly consistent with itself and with the manual of style. - 2 points
• The article has modest deficiencies in format and/or deviates significantly from the manual of style. - 1 point
• The article is poorly formatted such that the formatting detracts significantly from the reading experience. - 0 points

The article is illustrated as well as possible using images (and other media where appropriate) that follow the image use policy and have acceptable copyright status. The images are appropriately captioned and have alt text.

• The article is well-illustrated, with all or nearly all the appropriate images and captions. - 2 points
• The article is partially illustrated, but more or better images should be added. - 1 point
• The article has few or no illustrations, or inappropriate illustrations. - 0 points

Numerical scores can be translated into the different classes on the 1.0 assessment scale. For the lower classes, comprehensiveness and sourcing are the main things that differentiate articles of different classes; things like neutrality, style, layout, and illustrations quickly become important as well for the higher tiers of the assessment scale. GA-class and higher require separate reviews, but high numerical scores can indicate whether an article is a likely candidate for one of these ratings. For everything except GA and FA, the ratings are automatically determined by the banner template if detailed scores are present.

• Stub - An article with a 1 or 2 in comprehensiveness is Stub-class.
• Start - An article with a 3 or higher in comprehensiveness that does not qualify for a higher rating is Start-class.
• C - An article must have at least a score of 4 in comprehensiveness and 2 in sourcing to qualify as C-class.
• B - An article must have at least a score of 7 in comprehensiveness, 4 in sourcing, 2 in readability, and 2 in neutrality to qualify as B-class.
• GA - An article with at least 8 in comprehensiveness, 5 in sourcing, 3 in neutrality, 2 in readability, 2 in formatting and 1 in illustrations may be a good candidates to be nominated for Good Article status. (B is the highest rating automatically assigned by a numerical assessment.)
• A - An article with a 10 in comprehensiveness, 6 in sourcing, 2 in readability, 3 in neutrality, 2 in formatting, and 2 in illustrations may be good candidates for an A-class review.
• FA - An article with full points in every category may be a good Featured Article Candidate; even then, additional work may be necessary to comply fully with the manual of style.

The first assessment performed by the WP:USPP assessment team tested the quantitative article quality metric. Three main conclusions can be drawn from the metric assessment.

1. The quantitative article quality metric generates the same scores as the 1.0 rating system when used by Wikipedians.
2. The quantitative metric shows that Wikipedians tend to rate article quality with a high degree of consistency.
3. The metric assessment indicates that Wikipedians are tougher critics of article quality than subject matter experts. (The reasons for this are discussed on the talk page and in the full report.)
   

The assessment team is crucial to the success of the PPI. Their work uses a quantifiable measurement of Wikipedia article quality. Their work shows that Wikipedia assessment of article quality is tougher and more consistent than non-Wikipedian subject matter expert assessment. Their work helps define success of the project. In addition to the Wikipedia assessors listed below there are a few subject matter experts who assess via a Google Group.

User:Bejinhan

User:CasualObserver'48

User:Cordless_Larry

User:Fetchcomms

User:JayJasper

User:Mabeenot

User:Mike_Christie

User:Pjoef

User:RexxS

User:Ronk01

User:MacMed

User:MikeLynch

User:MikeBeckett

If you would like to join the Assessment Team, your participation is welcome, contact ARoth (Public Policy Initiative) (talk · contribs).

1. Metric Analysis - this assessment tests how well the quantitative metric works
2. Article Feedback Tool - this assessment compares Wikipedian, expert, and reader article quality scores
3. Student Pre Test - this assessment benchmarks article quality before students work on them for their class assignment
4. Student Post test - this assessment quantifies article improvement through student work on the project

The first assessment request is posted! Things are pretty exciting this term: LOTS of interesting topics and students are making some big improvements to content. The assessment team sees firsthand the impact of the project.

1. Student Pre Test 2.1 Requested 1 April 2011, Completed 15 April 2011- this assessment benchmarks article quality before students work on them for their class assignment
2. Student Post test 2.1 Requested 9 May 2011, Completed 20 May 2011 - this assessment quantifies article improvement through student work on the project
3. Student Pre Test 2.2 Requested 23 May 2011, Completed 30 May 2011 - this assessment benchmarks article quality before students work on them for their class assignment
4. Student Post test 2.2 Requested 10 June 2011, Completed 20 June 2011 - this assessment quantifies article improvement through student work on the project

Evaluation of the Public Policy Initiative has several layers. More information about evaluation and research for this project can be found on the Evaluation and Research page of the outreach wiki. For more context on article quality experiments, see "Experiments with article assessment" from The Signpost, 2010-09-13. For an overview of the Public Policy Initiative in general, see "Introducing the Public Policy Initiative" from the 2010-06-28 issue. If you would like to participate in project evaluation at a deeper level, please join the discussion on the outreach wiki or contact ARoth (Public Policy Initiative) (talk · contribs).