PWX AnalytiX Survey Report - Executive Summary
In May 2010, we conducted a survey to help us plot the course of PWX AnalytiX to best serve the water sector and also to validate our pioneering approach to enforce collaboration.
This report discusses the responses, our analysis, and our strategic response.
The top features requested in PWX AnalytiX are:
Show impact of projects by beneficiary counts clearly.
Provide ability to compare the plan (grant application) with the actual (project report).
Highlight visited (and not visited) projects.
Highlight projects which are running late or missing reports.
We need to provide access to these business intelligence features through maps, lists, and search.
In addition, the responses expose a need for a new sector-wide thrust to improve efficiency and collaboration, in both the collaboration and reporting processes.
We received many suggestions and challenges on improving collaboration and efficiency through a common grant application process.
We want to thank the participants in the survey for their valuable input, and also our friends who helped disseminate our input request through their networks.
The first release of PWX AnalytiX was released in September 2010.
AnalytiX Features Most Needed
The top four feature requests are graphed below:
Questions & Response DetailsClick to display
Across all respondents, the primary question requested was regarding beneficiary counts.
Everyone wants to know how many people are being impacted.
The next most requested feature was the ability to see the differences between the grant application and the project report. What changed?
People also want to know which projects have been visited post-completion.
Implementers want to know which of their projects they have visited and which need to be visited.
Everyone wants to know which projects are running late and which are missing data.
The open-ended questions resulted in many ideas, some of which are both helpful and easy to implement, and we will work to deliver those in PWX. Others we will seed and nurture, whose fruits we will deliver in the future.
However, we want to respond to some of the requests that appear to focus on the construction and technology aspects of the project.
Our vision to deliver and verify impact means that project activity needs to extend beyond the technology implementation.
We must include other activities such as the transfer of ownership and behavior change, which can only occur after the ribbon-cutting ceremony as part of the project. We want to engage the water sector in this vision and the discussion to achieve it.
We need a greater focus on post-implementation data acquisition to capture life-cycle events, such as when a measurable change in behavior happens – positive or negative, or where the transfer of ownership takes place and the beneficiaries become self-responsible for the sustainability of their project.
These fuzzy metrics are hard to capture and are composites of many social data points, but they are vital in meeting our common mission to make safe drinking water available to all.
The short-term mission for the PWX team is clear.
First, we have to show beneficiary counts clearly.
We plan to show charts by implementer and by year in PWX AnalytiX 1.0.
Second, we understand the need to provide the capability to compare the plan (grant application) with the actual (project report).
This is hard to do, and we will explore ways of doing this.
We realize that this may involve including the funder for accepting changes to the plan.
We may not be able to provide an adequate response to this requested feature in 1.0.
Third, we will provide the ability to highlight both projects that have and have not been visited.
We will provide the ability to distinguish projects visited by the implementer and those visited by their peers, volunteers, or other travelers.
Fourth, we will highlight projects running late or missing reports.
There are a lot of factors in identifying late projects.
For PWX AnalytiX 1.0 we will only show the projects whose statuses are not shown complete by the stated completion date.
Later, we will add the ability to compare the dates in the proposal, whether the final report contains the required data, whether the funding agency is satisfied, etc.
The preferences of our users to access PWX AnalytiX are graphed below:
As shown in the table in the earlier section, people want to be able to drill down into reports and get to raw data.
Maps are an easy way of visualizing, locating, and drilling down.
Searching and using lists are also popular methods of seeking and viewing project information.
Access is also a part of PWX AnalytiX!
How we access data can also help us in understanding it.
We will provide all access points from a central and easy-to-use console.
This organizing tool will provide the ability to view several pieces of data and make it easier to decide on which to focus on.
More Efficient Selection
We sought ideas for creating a more efficient project selection process.
We have synthesized and summarized the ideas and suggestions that we received:
Increase transparency of the selection process both on donor and implementer side.
Standard application / project templates will reduce work.
Peer review needs to provide feedback cycle and insert transparency into process.
Feedback to implementer about selection; granted or denied, the news should be sent quickly.
Help with training; new process can create new issues.
Past implementer performance will help make decisions.
In addition to looking for incremental improvements in the performance of their existing grantees,
funders should also experiment ("stick their neck out") with selecting projects with new grantees.
The search for funding is very labor intensive now, which can be reduced dramatically.
Constant-shares budgeting prevails, which makes the budgeting process easier for many funders,
but is not necessarily tied to project success rates or regional need reality.
The above list as well as the data from the next three sections indicate that the entire application process
is burdensome to both implementers and funders and making it more efficient, simpler, and transparent will
alleviate much of the pain.
PWX already addresses many of the above suggestions.
Our vision is to create a marketplace where funders and implementers can
interact, collaborate, establish relationships, transact, and build a history of action.
Since our marketplace resides in the social development sector, we want PWX to be an exchange - something more than
a pure marketplace - where we facilitate more collaboration than competition,
we explicitly create more sharing and learning, and teaching, too.
In moving PWX towards this ideal exchange we added peer review to support
collaboration, increase interchange of ideas, and increase decision-making efficiency.
By adding transparency to this exchange, we allow everyone to see the entire set of actions continuously shaping the exchange,
the evolving strategies of its participants, and, most importantly, the impact on the ground.
In terms of specific steps, we will develop a feature to allow funders to provide more feedback in the funding decisions (with or without peer review).
We are continuously working to bring this model to the water sector,
and the results of this survey reinforce our strategies.
We have proven the implementer world is ready to work in this marketplace,
and we now need to work with funders to increase their participation more fully.
Funders' Issues In Selection
The responses to issues facing funders are graphed below:
80% of the funders feel that the process of selecting projects could be made more efficient.
77% of funders would like to see a full history of projects done by the implementer.
55% of funders have no trouble finding projects to fund.
PWX is all about making the funding process more efficient.
Through standardization and peer review, the load on both the funder and the grant applicant is reduced.
PWX is working with implementers to put all their data on PWX so projects funded outside PWX still can be visible within PWX.
A few organizations have provided all their data and made it possible to show of their amazing work, in its entirety, to their peers and the world on PWX.
Implementers' Issues In Application
The responses to issues facing implementers are graphed below:
53% have submitted the same application to different funders.
70% have prepared the same application in different formats.
85% believe that there is value in creating a common grant application format.
PWX is all about making the application process more efficient!
We have created a standard application format which is powerful and simple and look forward to continue working with the community to evolve it.
Creating A Common Application
ChallengesClick to display
PWX has created a standard application format which is powerful and simple and look forward to continue working with the community to evolve it.
PWX has already broken apart the application as shown in the diagram below.
The application thus needs to only focus on the project.
The organization background can be updated as needed.
The history is built up through a one-time upload of projects not funded through PWX and the rest is built automatically as applications are made and projects are executed.
PWX's interactive mapping feature helps see all the work of an organization (through the years) and also that of nearby peers.
PWX's hallmark is transparency! It is revolutionary to share all application history including those not funded.
PWX will work with the WASH community to participate in the definition and acceptance of a common application for WASH projects that highlights the necessary key data elements while leaving space for creativity and uniqueness.
We understand the language variations and in the short-term hope that volunteers help with translation.
Longer-term we hope to have enough activity to justify creating different sections in different languages,
with some basic automated translation to help searching.
We hope to start working with funders to establish KPIs for their projects that feed and merge in with the general process over time.
We will strengthen capacity building activities through our application and also training activities as we get funders interested in that area.
The responses to issues faced by respondents (funders, implementers) who have currently active WASH projects are graphed below:
The first question identified that 47% of those surveyed had projects currently active (either funded or implemented) in the water sector.
Of these 47% of responders:
- 77% find it difficult to find resources to follow up on projects after completion.
- 43% have problems changing projects once in motion; we don't know how many have encountered problems with projects once started.
- 95% find it beneficial to get project updates after completion, compared to 71% of all respondents.
We would like to increase the financial resources to follow up on projects. Maybe we could keep aside 3% of project costs (added on) for such activity.
Having peers visit projects can occur at lower costs (compared to funders and consultants) and more importantly, dramatically increase learning through face-to-face meetings onsite.
Our strategy is to crowdsource the impact assessment process (monitoring and evaluation) to ensure much greater coverage of projects.
This brings a flood of new resources into this extremely important activity.
We want to increase the number of post-completion visits and thus increase the probability that any given project will be visited.
This reduces the burden - specifically targeting visits is a very time-consuming and expensive projects - on the entire water sector.
It is heartening to see that less than half of the people with active projects do not have a problem with changing project direction during implementation,
and we hope its not because they have not encountered problems during implementation that require adjustments.
Such flexibility is key in ensuring greater success at the end.
PWX will support this; through its frequent update feature, the issues showing in projects will be posted
allowing for an open and collaborative adaption.
Time Spent in Applications and Reports
The time spent by those surveyed in the application process and also in the reporting process is graphed below:
80% of the responders spend more than 20% of their time in the application process while 30% spend more than 40% of their time on applications.
The channels used by respondents in the application process are graphed below:
As can be seen, the sector has moved towards electronic communication.
While electronic communication is certainly quicker than earlier modes, it is still inefficient.
This move has aided the sector immensely, but most emails are still printed for discussing, record-keeping purposes, and other reasons
and the email communication - printed or not - can be difficult to compare and standardize on without a central and presentable organizing interface.
75% of the responders spend more than 20% of their time in the reporting process with 29% spending more than 40% of their time on reports.
People in the water sector want more detail (esp. visual) in the report and the ability to drill down into it.
So while the survey shows that a lot of time is spent on reporting, the results are not making a majority of people happy.
PWX exists to make the application process more efficient! We want the sector to move towards a 'marketplace' - an exchange - where
funders and implementers can come together and move proposals forward more efficiently.
One feature of an exchange is creating standardization;
PWX has started by creating a standard application format which is powerful and simple and has evolved over the past five years.
This allows many buyers (funders) and sellers (applicants) to come together quickly and make decisions faster to get on with the real work of implementing projects.
We look forward to working with the community to evolve it.
PWX is also transforming the reporting process, so it is both easier to enter the report and to read it.
Moving towards short quick reports of project progress and impact assessments creates a living history of the project.
These features make it easier to read about, visualize, and digest the transformation happening on the ground.
In addition, by connecting to maps, lists, and searches, reports can be easily identified and connected to other reports and work to see results in a larger context.
The responses to whether we can collaborate during the funding process and the implementation process are graphed below:
Expanding the analysis to beyond our survey responses, we see that water sector (and actually the entire development sector) sees collaboration as a way forward.
Collaboration is a way to increase learning, but is also increasingly regarded as a way to increase efficiency.
Across all demographics our respondents felt that we need to and can collaborate both during the fundraising phase
as well as the project implementation and long-term monitoring phase.
When we layer on the fact (graphed above) that our respondents are increasing using social networking tools,
it sets the stage for increased collaboration in the WASH sector.
The Facebook arrow is green because our respondents have found success in raising money through this channel.
69% did not know the term "crowdsourcing".
Crowdsourcing is the act of outsourcing tasks, traditionally performed by an employee or consultant,
to a large group of people or community (a crowd), through an open call.
Its potential to expand collaboration by
involving more people outside the sector to help track progress is not seen by many in the sector.
The water sector sees collaboration as a way forward and the people in it are increasingly familiar with social networking tools.
Our strategy is to serve those who wish to collaborate and are familiar with online networking processes.
They not only will understand our process that allows for (and enforces) collaboration at each stage of the project life cycle,
but also will easily understand the lingo, the tools, and their use.
We will push our users toward:
- more frequent communication
- creating a steady stream of status updates
- employing more media (photos and videos)
- more interaction (both online and physical)
- increasing acceptance and participation in crowdsourcing
- and, away from long bookish reports not easily viewable online.
Barriers to Reporting Failures
The responses to our request to identify the barriers to reporting the failures in the water sector are graphed below:
Some predictable trends in the responses are visible.
The majority of people feel that the funder relationship is the major reason for not reporting failures.
There are no forums to discuss failures; the website of an implementer is definitely not the place!
Bad publicity is definitely an important deterrent to discussing failures, since stories may be taken out of context.
Clearly, funders have no personal issues in reporting failures!
We feel that peer review allows for more open discussion about problems with earlier projects and approaches.
Our plan to manage all projects by implementors in the network on the website regardless of funder helps put a failure in perspective of the totality of their work.
Finally, in a peer process, sharing failures helps everyone learn and people can also contribute to fixes and changes for the future.
Funders working in isolation cannot put a failure into context easily, but a peer-reviewed application is easier to judge.
The existing PWX strategy makes it the only forum and the best process for reporting and discussing failures.
The survey was closed on May 18, 2010 and the report released on June 25, 2010.
We received 157 responses representing 31 countries shown on the map-chart below.
The breakdown of respondents by organization type are shown on the pie-chart below.
We want to again thank all the respondents for their insight and their time.
We hope this report shows how the survey has helped us with our work and
that, by August we will showcase PWX AnalytiX to help the water sector.