# Community Advisor Assessment Guide

# Guidelines for the application of Scoring Criteria in the Catalyst Fund Process

# General Guidelines

The Community Advisor (CA) is a self assigned role in the Cardano Project Catalyst Community.

Within this role you review proposals in the assessment phase. You are asked to determine how well each proposal matches each of the three criteria: impact, feasibility and auditability. The combined individual score across the three criteria will give an overall score for the proposal.

As a CA you play an important role in the Catalyst funding process by providing guidance to voters who will use it to make their decisions. You are also offering feedback to proposers through your rationale.

Remember: You can not be a CA and a proposer in the same fund. There is an exception: If you have a proposal in the challenges ‘Distributed Decision Making’ and ‘Challenge Setting’ you are still allowed to serve as a CA in the remaining challenges.

Please consider each of the following points as you undertake your assessment(s):

  • Assign for each criteria a score between 1 and 5.
  • For each of the three criteria a score and a rationale must be provided.
  • Each criteria should be considered and assessed on its own
  • Your rationale must explain the score you have given. If a rationale does not correspond with the score given the review may be removed.
  • Any score without a rationale will be filtered out and you will not be compensated.
  • Your rationale should help the voters make their decision, so make sure your considerations, scores and rationales are comprehensible.
  • Consider and communicate what additional information or improvements the proposer could include in order to obtain a higher score in future funds.
  • Submitted assessments can be edited until the review window closes.
  • Be sure that you submit your assessment correctly.

# Guiding Principles

The following principles are designed for CAs and should be followed to help the community fulfil the intention of being a safe and lively environment to explore the fullest potential of human collaboration.

  • Treat proposals and proposers with respect. Misunderstandings are more likely than malign behaviour (see Hanlon's Razor) (opens new window). While it is easy to take a negative view of suggestions/messages/activities that you don't understand or disagree with, instead of wasting your energy on arguments, try to find clarity and common ground. More often than not, it is likely that you (or the proposer, or both) are simply misunderstanding something about each other / the proposal / the challenge or the context. In the assessment phase, you can still contact the proposer. Use this opportunity before making any assumptions.

  • Separate the idea from the proposal. Considering the proposal as it is written, does it express the idea appropriately and provide a clear path to success or does it need more assistance/collaboration/resources to reach its full potential? A great idea poorly communicated can come across as a poor proposal. Offer your critical analysis of the proposals in a respectful, unobtrusive and non-accusative manner. Focus on what is said in the proposal and not who is saying it.

  • Community Advisors offer advice. Offer your best advice and feedback to help proposers write better documentation, explain their ideas more clearly, assess risks or build stronger teams. Be aware that as a CA you should not take over the writing of a proposal. If you find a project that you wish to collaborate on, you have to change your own standing point from a CA to a proposer. The integrity of the system lies in the hands of each of us and our decisions.

  • Be mindful of your time and knowledge. No one expects you to commit more time than you have available. So assess as many proposals as you can and know that it’s enough. To assess one or two proposals is already a phenomenal help for this process. Some proposals will be technical in nature, although it is encouraged that proposers prepare their proposals at a level suitable for most voters. Don't feel that you have to provide an assessment if a proposal is beyond your technical knowledge. Even if a proposal exceeds your technical knowledge, you may still have skills and knowledge appropriate to assess portions of the proposal - in which case, feel free to do so.

# Definitions of quality

These are Catalyst definitions that are applied to voter and advisor participants, including community advisors (CA) and veteran community advisors (VCA) so that:

  • CA are clear on what is expected of them for assessments
  • VCAs are clear on what is expected of them reviewing CA assessments
Guidance on what is expected
Focus Why is this important? EXCELLENT (where most or all are achieved)
CA contemplation of their own ability to assess a proposal CAs have a wide range of backgrounds and experience levels and the reader (proposer or VCA) should be able to discern from the advisor’s feedback which opinions are objectively qualified based on advisor knowledge and which opinions are more subjective. The CA has been able to define which assertions they are better qualified to make in the proposal about its fit to address the Catalyst Challenge, and which assertions are more intuitive, or outside the advisor’s domain of expertise.
CA scoring against Challenge criteria The reader (a VCA or proposer) should be able to understand how the score reflects the advisor’s perspective and feedback provided.*Specifics within the proposal are highlighted to illustrate understanding, and the advisor has provided supportive rationale that is aligned with the criteria. The CA has provided a defendable justification to support the scoring. This justification for scoring clearly highlights specific content in the proposal that is being assessed, and their feedback gives confidence to the reader that the entire proposal was considered and the score plus feedback provided was based on a sound analysis of all the information provided.
CA opinion on the proposal overall Each challenge has its own intended outcomes. The job of assessors and voters is to qualify the validity and appropriateness of the activities described in the proposal and if it is a good use of Cardano Treasury funds to achieve these. CA feedback comments are unambiguous and they demonstrate a strong grasp of the objectives of the Catalyst challenge and the project team’s objectives to address the challenge.
Accountability to the Community Measures being taken so the project team remains accountable to the Community is at the heart of Catalyst. The CA can clearly identify how the proposing team will audit their positive return on intention and, and their scoring is backed up with supporting rationale whether they either agree or respectfully disagree that the plans and budget offer value for money that are appropriate to deliver the project’s intentions.
Positive feedback-loop CAs that provide practical coaching helps the proposer to deliver a better project.*The reader (VCA and proposer) should be able to understand why these guiding points are important to the community advisor and how addressing them would improve scoring the proposal. Actionable coaching points are provided to guide the proposer about what is and isn’t strong about the proposal, and / or what could be improved to become a more compelling proposal in the future.

# Scoring Criteria

# 1) Impact

Scoring the impact of a proposal means that you are assessing whether the proposal addresses the objectives laid out in the challenge or not. As each challenge has its own set of objectives, be sure to be fully aware of them as you conduct your assessment. The objectives can be found in Ideascale under the heading "campaign brief" in the sections about how success looks like and key metrics to measure. Since the challenge settings can be interpreted differently by different people, it is your role to determine why the proposed solution fits into the specific challenge category - based on what the proposer has communicated.

Suggested characteristics of an impactful proposal:

  • The proposal identifies a problem within the challenge which needs to be solved.
  • The proposal clearly articulates how it will add value to the Cardano ecosystem.
  • The key metrics and goals of the proposal align with the outlined challenge settings.
  • The proposal is able to scale to address future challenges.

Score meaning:

  • (1) I strongly disagree
  • (2) I disagree
  • (3) Neutral. I neither agree nor disagree
  • (4) I agree
  • (5) I strongly agree

that this proposal effectively addresses the challenge.

Examples for a considered assessment statement:

This proposal is a great idea. The execution part of it seems to be a bit lacking as the next review sections explain. However, for the challenge brief - it is fitting in. What I am missing is understanding how this fits into the narrative of existing marketplaces for general tokens and how this proposal actually fills the void. Making that distinction clear in the mission brief would be fantastic next time. (Rating given: 4)

As the goal of this challenge is to create Dapps for 2021 to run on Cardano, this proposal is within the challenge settings. However, the statement of what problem is to be solved remains vague. The proposed application on its own has only limited impact. An ecosystem with different use cases must be created here, but this requires further developments in this area in addition to this proposal. If the team is able to build this bridge, then I could see a positive impact for Cardano. Based on the given information, I can rate the impact of this proposal only with a score of 3.

The proposal touches upon the fact that some version of the dashboard is available. However, no information/screenshots are added to help understand it so the proposal remains very abstract.. Would have been great to see this prototype that is mentioned as drafted already - it would make it easier to understand the thought process of the proposed solution and how this addresses the challenge. On the other hand - I appreciate the timeline that goes into years ahead - because it is rather expected that political change is difficult to enact. Introducing a solution like this proposal is a way of challenging the status quo. I believe, in its current form - it may be somewhat too ambitious to convince a government to utilise such a tool. If I read this in the proposal itself - it would make me feel more confident in the thinking behind all of it. (Rating given: 2)

# 2) Feasibility

Scoring the feasibility of a proposal means that you are assessing whether a proposal is likely to be successfully implemented or not. It is your task as a CA to assess the proposal's ability to solve the problem that has been identified. There are no firm rules for this; it will all depend on what information the proposer has provided.

A feasible proposal has considered the following aspects:

  • The proposer provides evidence (references, links, etc.) of relevant skills and experience needed to implement the proposal.
  • The proposer knows, in an identifiable manner, the type and number of team members required to implement the proposal.
  • The budget is outlined clearly and broken down into identifiable items.
  • The budget is reasonable to achieve the set goals.
  • If additional funding is required to implement the proposal, the proposer has provided a verifiable plan in this regard.
  • The proposal offers a realistic timetable to complete the work.
  • The proposal provides a sensible and conductible plan for implementation that shows not only what is being done but also how.
  • The plan describes the resources necessary to implement the proposal.
  • The proposal clearly explains technical aspects like architecture, language and technologies if they are crucial for implementation.
  • The proposer has considered challenges and risks relevant to a successful implementation.

Score meaning:

  • (1) I strongly disagree
  • (2) I disagree
  • (3) I neither agree nor disagree
  • (4) I agree
  • (5) I strongly agree

that it is highly likely that this proposal will be implemented successfully.

Examples for a considered assessment statement:

This team has been participating in the Cardano community for a long time and have been involved with Catalyst from the beginning of the project. I believe they have the skills to inform and educate new members about Catalyst, as they are already doing this. The presented plan is comprehensible but I think it is important to have more clarity in the description of what the membership model is and its relevance in the proposal. The proposers will approach new people for Catalyst through meetups. I think it is important to describe how this will be done in more detail to better understand the approach. (Score given: 3)

The proposer team states that it has the necessary experience. However, they did not provide any links or references to support this statement. My recommendation for a future proposal is to add this information so that the voters are able to validate the experience. The budget provided is very general and therefore I cannot say whether it is appropriate or not. The implementation plan is comprehensible but leaves important questions unanswered: Where do you get the users from? To create the proposed solution is one thing, but operating it successfully is another. If the described aspects are improved, then this proposal has a significantly better chance of obtaining successful funding. Based on the given information, I don’t expect a successful implementation. Therefore, I rate feasibility with a score of 2.

Potentially an intriguing concept but there are some major concerns to me. First, there is a budget figure of 200k USD requested which is half the fund of this challenge. Given that amount, I would expect an appropriate level of depth when addressing how this budget is going to be spent. I tried to open every attachment and read through all segments - but just couldn't locate it. Second - I am unable to establish validity of claims/experience. Third, the proposal itself reads very poorly. It reads like a concept more than an executable proposal. It needs more editing so that it can be assessed properly given its direction. Perhaps reworking it and submitting it at the later stages will work. (Score given: 1)

# 3) Auditability

Scoring the auditability of a proposal means that you are assessing if the information provided is sufficient to audit the progress and the success of the proposal. The audit phase is essential to ensure the implementation. After successful funding the proposer periodically reports the progress to the community. Therefore, the community needs to know what to expect from the proposer up front. As you work through the proposals look for milestones with identifiable measures of success which when achieved can be measured by community.

An auditable proposal has considered the following aspects:

  • A roadmap with milestones and a time horizon for achievement.
  • Metrics/KPIs that define the success of the proposal.
  • Clear understandable description of the targeted problem.
  • Clear understandable solution that addresses the problem.

Score meaning:

  • (1) I strongly disagree
  • (2) I disagree
  • (3) I neither agree nor disagree
  • (4) I agree
  • (5) I strongly agree

that this proposal provides me with sufficient information to assess the progress in attaining its stated goals.

Examples for a considered assessment statement:

The proposer provides a step-by-step plan for the implementation with clear milestones and a defined time horizon. There are also several metrics that measure the success of this project. I recommend adding further metrics like “...” so that more areas of this project are auditable. If the proposed solution is successfully implemented it provides a useful solution to the targeted problem. The only thing I miss in this area is that the proposer has not mentioned possible difficulties and how to deal with them. The quality of the proposal suggests that the proposer is aware of the challenges, but it is easier for outsiders to understand if a statement is made. In summary, I rate auditability with a score of 4.

The proposal lacks some important information, such as roadmap, estimates of content to be produced, details on budget allocation and details about the proposers (it was mentioned that one of the proposers is an SPO, but the pool ticker was not included), I recommend including this information so that the community can have a better feasibility assessment and understand the impact of the proposal. (Score given: 3)

Roadmap and milestones (incl. time horizon) are weakly described. It is only mentioned that further funding will be needed. The given metrics measure what the proposer wants to do, but the success of his actions is not defined (for example: you don't measure the number of emails that are sent alone, you have to measure the response rate). As mentioned above, the description of the solution is inadequate and confusing. Possible challenges are not considered. In conclusion, the progress of this proposal is hard to assess. (Score given: 2)

# Inadequate assessment statements:

Rationales similar to these given below are not in accordance with the standard of the guideline as they do not offer guidance to voters and to proposers. To have your assessment counted towards the overall rating, inadequate assessment statements should be avoided.

For information about the reviewing-the-assessments process, please follow the guidelines for vCA (Veterian Community Advisors) who use a Red and Yellow flagging system to review CA assessments [Link (opens new window)].

# No rationale given:

  • This proposal is only fishing for funds.
  • This proposal is irrelevant.
  • This proposal is missing the point.
  • I can't rate the proposal.
  • Please don’t vote for this proposal.
  • It is a very good proposal. Should be implemented.
  • This proposal [does not] effectively addresses the challenge.
  • It is highly [un]likely that this proposal will be implemented successfully.
  • This proposal [does not] provides me with sufficient information to assess the progress in attaining its stated goals.

# Only a personal opinion given:

  • This proposal is terrible.
  • I like the idea but not the proposer.
  • Is this a joke?
  • Too many similar proposals.
  • Asking for too much funds.

# Unconstructive assessment:

  • The proposal is poorly written, there is not enough information to properly understand it and it’s purpose completely mismatches the challenge goals.
  • The proposer’s experience is not enough to execute the project, and there is no information about hiring experts. Very unlikely to be successful.

# Further aspects of an inadequate assessment:

  • The assessment is given from a biased perspective.
  • It is clear that the proposal was not read through by the CA.
  • It is clear that the scoring criteria and assessment guidelines were not applied or understood by the CA.
  • The CA lacks clear articulation of the rationale for the assessment in a constructive way.

# The assessment has been completed.

# What happens next?


# Reviewing-the-assessments process

This is a self correcting process which precedes the Ballot Submission phase and provides guidance to the Voting Committee.

  1. CA's finish assessments.
  2. An anonymized CSV-file containing all assessments will be passed for peer review by veteran CA's who will filter out substandard assessments.
  3. Proposers will also be invited to report substandard assessments.
  4. Experienced CA's decide which reported assessments should be excluded.
  5. A Catalyst admin removes the results of step 4 and any blank assessments from final published assessments.
  6. A list of all removed assessments will be published and a retrospective to review the process for the next fund will be held.

# Ballot Submission

  • The assessment score and a link to the proposal's URL on the Ideascale innovation platform will be visible from the Catalyst Voting app.
  • All rationales provided by CA's will be attached to the original proposal in Ideascale.

# Anonymity

Identity of the CA will be kept anonymous in Ideascale. However, despite our best efforts and testing, there's always a small risk, because of the nature of Ideascale being a 3rd party tool, and anonymity being an experimental feature, that the CA's identity might become known. Please make sure you understand that risk before assessing proposals.

# Community Advisor Incentive

Goal of incentives is to assure CAs will provide a thoughtful and fair assessment to all proposals, and that each proposal will receive at least three assessments. Therefore:

A total of 5% of the challenge fund amount will be allocated as incentive:

  • 4% will be allocated as rewards for assessments to CAs
  • 1% will be allocated to incentivize those reviewing the assessments (vCAs)

For each proposal on the ballot, three assessments will be randomly selected from all CAs of the proposal to receive an incentive. That means if a proposal received 5 reviews, 3 CAs out of the 5 will receive an incentive.

Incentives will be distributed evenly between all CAs that are randomly selected. In order to receive the incentive the CA will be asked to add a valid Shelley payment address in Ideascale. Rewards will be sent along with proposer rewards at the start of the execution phase for the Fund.

Note: A CA is not eligible for incentives if he receives a red flag for inadequate assessments during reviewing-the-assessments process by vCAs.

Please direct any further questions to the Telegram channel: (opens new window)CatalystCommunityAdvisors (opens new window).* (opens new window)

The Project Catalyst team thanks all our CAs for helping to build the future of Cardano!

Last Updated: 10/5/2021, 1:36:24 PM