Key issue | Evaluation questions | Research methods to tackle these issues | Research questions |
Task definition | Were the objectives firmly based on stated local community needs, or through higher-level political goals? Who set the objectives? Were these consulted upon? | Interviews with stakeholders
Desk review of internal project information ( e.g. project inception plan)
Research with participants | How were the project objectives determined?
Which stakeholders were involved in determining the objectives? |
Representative-ness | Assessing the broad engagement by gender, age, socio-economic and educational levels, and in terms of exclusion factors (including IT access, IT skills and others). How were disadvantaged groups engaged / involved? How were barriers addressed? | Web stats
In-depth interviews with stakeholders
Research with participants | How was participation in the Websites measured?
How were participants 'legitimated' as being genuine citizens?
How was participation for a locality benchmarked against local demographics?
What strategies were in place to maximise the participation of excluded or under-represented groups?
Who was excluded or not represented in participation? (these groups will then be targeted for research with participants) |
Early involvement | Were the citizens deeply embedded into the project from the early stage? What were the 'wastage rates', and what were the 'recruitment rates' - in effect how many people lost interest, and how many people were joining the activities? | Interviews with stakeholders
Research with participants | Was your interested sustained throughout and beyond the pilot? If not, why not? If yes, what motivated you to keep participating? |
Community engagement | Was engagement prolonged throughout the pilot? Was engagement sustained? How was the project marketed and disseminated? What was the quality of engagement - was it information sharing, consultation or participation? Did the pilot lead to an increase in public participation? | Interviews with stakeholders
Research with participants | What was done to sustain / rejuvenate interest?
How did you bring more people or new groups on board?
What did people do when they visited the website?
How was the quality of participation measured?
How was the participation lined to policy making? |
Independence | Were the solutions locally designed, or was an external model used? Were local people involved in the design stage? Has conflict and dissent been welcomed and managed as an important mechanism to reach consensus on local issues? | Interviews with stakeholders
Review of internal project information (project plans / models)
Content analysis (to see where debate moved towards) | How did the relationship with external contractors work?
How were partners involved?
Were suggestions and criticisms taken on board?
On what issues was there discussion, debate and consensus? |
Cost -effectiveness | What baseline financial models and data were used? Was the cost-benefit of citizen recruitment clearly argued? How much have other similar models cost/spent/gained? | Review of internal and external project information
Financial data | Was the budget spent according to plan?
What was achieved for the money spent?
What alternative financial models could have been used?
How did the project plan for financial sustainability beyond the funding period? |
Influence | How was the participation assessed in terms of influence on policy? What were the norms of deliberation, and did they change during the project? In effect, what was the initial quality of deliberation, what were the deliberation outcomes, and was there increased political sophistication at the end? Were citizens confident that they see fairer decisions, increased trust in and legitimacy of the local administrations? Was there reduced conflict? | Interviews with stakeholders
Content analysis (to see where debate moved towards)
Interviews with participants | Did you feel you were able to influence policy?
What changes as a result of you being involved in this pilot?
Were you able to raise / influence local issues?
Did you feel your views were being listened top / taken on board /acted upon?
Do you feel you now better understand your local community / local representatives? |
Transparency | Has a content analysis been carried out on Web-logs and other discussion forums? How engaged were the discussions? | Content analysis | Testing for message content on a sample of messages. Did people feel listened to? Were messages acted upon?
What issues were raised/ Were these relevant? |
Structured decision-Making | Was there more of the same activity at local levels - did this project just increase the existing activity levels (known as concurrent validity). Or, if there was an impact on local policy and local engagement did this lead to significant changes in the ways that decisions were debated and resolved at a local level, or was the process unchanged? In effect, was there evidence that organisational change took place in local administrations? What were the planned changes to democratic engagement within the local administrations (known as predictive validity)? | Interviews with stakeholders
Interviews with participants | Had you interacted with local representatives before you used this forum (is it aiding the already active to participate electronically)
Do 'excluded' groups use the mechanism?
What changes have taken place in local democratic structures since this pilot?
What policies / decisions have been influenced? |
Resource Accessibility | Were the resources fully designed-for-all, and were they used by all? Who did not use the resources and why not? Were IT skills / technology a barrier? What IT and other skills were required. | Interviews with stakeholders
Interviews with participants | Which groups did not participate in the pilot and why not?
Were there barriers to participation identified? Were certain parts of the website un-used / under-used? |
Impact on community councils | Did community councils ( CCs) respond to the views expressed on the website?
Were comments relevant? What changes have occurred? Context of wider consultation on future of CCs.
Are CC's best placed to run these types of initiatives? | Interviews with stakeholders | Did CCs respond to the views expressed on the website?
What was the best way for CC's to respond? |
Rollout | Is the model suitable for rollout? Which aspects can be beneficially transferable? Was this project value for money or do CC's prefer to operate in other ways?
Could the impacts have been achieved more effectively through other mechanisms? | Interviews with stakeholders
Interviews with participants
Review / analysis of all incoming project data | What worked well / less well?
What aspects could / should be rolled out?
What aspects could / should not be rolled out? |