# Community Advisor Assessment Guide

# Guidelines for the application of Scoring Criteria in the Catalyst Fund Process

This is a transcription of the Community Agreed Document (opens new window)

# General Guidelines

The Community Advisor (CA) is a self-assigned role in the Cardano Project Catalyst Community.

Within this role you review proposals in the assessment phase. You are asked to determine how well each proposal matches each of the three criteria:impact, feasibility and auditability. The combined individual score across the three criteria will give an overall score for the proposal.

As a CA you play an important role in the Catalyst funding process by providing guidance to voters who will use it to make their decisions. You are also offering feedback to proposers through your rationale.

Important: You can NOT be a CA in challenges that you are participating in as a proposer, implementer or actively working with a proposer in (eg. rewarded mentorship). Assisting and suggesting improvements to proposers as a CA is desirable and encouraged.

Please consider each of the following points as you undertake your assessment(s):

  • Assign for each criteria a score between 1 and 5.
  • For each of the three criteria a score and a rationale must be provided.
  • Each criteria should be considered and assessed on its own
  • Your rationale must explain the score you have given. If a rationale does not correspond with the score given, the review may be removed.
  • Any score without a rationalewill be filtered out and you will not be compensated.
  • Your rationale should help the voters make their decision, so make sure your considerations, scores and rationales are comprehensible.
  • Consider and communicate what additional information or improvements the proposer could include in order to obtain a higher score in future funds.
  • Submitted assessments can be edited until the review window closes.
  • Be sure that you submit your assessment correctly.

# Guiding Principles

The following principles are designed for CAs and should be followed to help the community fulfill the intention of being a safe and lively environment to explore the fullest potential of human collaboration.

  • Treat proposals and proposers with respect. Misunderstandings are more likely than malign behavior (see Hanlon's Razor) (opens new window). While it is easy to take a negative view of suggestions/messages/activities that you don't understand or disagree with, instead of wasting your energy on arguments, try to find clarity and common ground. More often than not, it is likely that you (or the proposer, or both) are simply misunderstanding something about each other / the proposal / the challenge or the context. In the assessment phase, you can still contact the proposer. Use this opportunity before making any assumptions.

  • Separate the idea from the way it is written. Considering the proposal as it is written, does it express the idea appropriately and provide a clear path to success or does it need more assistance/collaboration/resources to reach its full potential? A great idea poorly communicated can come across as a poor proposal. Offer your critical analysis of the proposals in a respectful, unobtrusive and non-accusative manner. Focus on what is said in the proposal and not who is saying it.

  • Community Advisors offer advice. Offer your best advice and feedback to help proposers write better documentation, explain their ideas more clearly, assess risks or build stronger teams. Be aware that as a CA you should not take over the writing of a proposal. If you find a project that you wish to collaborate on, you have to change your own standing point from a CA to a proposer. The integrity of the system lies in the hands of each of us and our decisions.

  • Be mindful of your time and knowledge. No one expects you to commit more time than you have available. So assess as many proposals as you can and know that it’s enough. To assess one or two proposals is already a phenomenal help for this process. Some proposals will be technical in nature, although it is encouraged that proposers prepare their proposals at a level suitable for most voters. Don't feel that you have to provide an assessment if a proposal is beyond your technical knowledge. Even if a proposal exceeds your technical knowledge, you may still have skills and knowledge appropriate to assess portions of the proposal - in which case, feel free to do so.

# Scoring Criteria - Regular Challenges

# 1) Impact

Scoring the impact of a proposal means that you are assessing whether the proposal addresses the objectives laid out in the challenge or not. As each challenge has its own set of objectives, be sure to be fully aware of them as you conduct your assessment. The objectives can be found in Ideascale under the heading "campaign brief" in the sections about what success looks like and key metrics to measure. Since the challenge settings can be interpreted differently by different people, it is your role to determine why the proposed solution fits into the specific challenge category - based on what the proposer has communicated.

Suggested characteristics of an impactful proposal:

  • The proposal identifies a problem within the challenge which needs to be solved.
  • The proposal clearly articulates how it will add value to the Cardano ecosystem.
  • The key metrics and goals of the proposal align with the outlined challenge settings.
  • The proposal is able to scale to address future challenges.

Score meaning:

  • (1) I strongly disagree
  • (2) I disagree
  • (3) Neutral. I neither agree nor disagree
  • (4) I agree
  • (5) I strongly agree

that this proposal effectively addresses the challenge.

Examples for a considered assessment statement:

This proposal is a great idea. The execution part of it seems to be a bit lacking as the next review sections explain. However, for the challenge brief - it is fitting in. What I am missing is understanding how this fits into the narrative of existing marketplaces for general tokens and how this proposal actually fills the void. Making that distinction clear in the mission brief would be fantastic next time. (Rating given: 4)

As the goal of this challenge is to create Dapps for 2021 to run on Cardano, this proposal is within the challenge settings. However, the statement of what problem is to be solved remains vague. The proposed application on its own has only limited impact. An ecosystem with different use cases must be created here, but this requires further developments in this area in addition to this proposal. If the team is able to build this bridge, then I could see a positive impact for Cardano. Based on the given information, I can rate the impact of this proposal only with a score of 3.

The proposal touches upon the fact that some version of the dashboard is available. However, no information/screenshots are added to help understand it so the proposal remains very abstract.. Would have been great to see this prototype that is mentioned as drafted already - it would make it easier to understand the thought process of the proposed solution and how this addresses the challenge. On the other hand - I appreciate the timeline that goes into years ahead - because it is rather expected that political change is difficult to enact. Introducing a solution like this proposal is a way of challenging the status quo. I believe, in its current form - it may be somewhat too ambitious to convince a government to utilise such a tool. If I read this in the proposal itself - it would make me feel more confident in the thinking behind all of it. (Rating given: 2)

# 2) Feasibility

Scoring the feasibility of a proposal means that you are assessing whether a proposal is likely to be successfully implemented or not. It is your task as a CA to assess the proposal's ability to solve the problem that has been identified. There are no firm rules for this; it will all depend on what information the proposer has provided.

A feasible proposal has considered the following aspects:

  • The proposer provides evidence (references, links, etc.) of relevant skills and experience needed to implement the proposal.
  • The proposer knows, in an identifiable manner,the type and number of team members required to implement the proposal.
  • The budget is outlined clearly and broken down into identifiable items.
  • The budget is reasonable to achieve the set goals.
  • If additional funding is required to implement the proposal, the proposer has provided a verifiable plan in this regard.
  • The proposal offers a realistic timetable to complete the work.
  • The proposal provides a sensible and conductible plan for implementation that shows not only what is being done but also how.
  • The plan describes the resources necessary to implement the proposal.
  • The proposal clearly explains technical aspects like architecture, language and technologies if they are crucial for implementation.
  • The proposer has considered challenges and risks relevant to a successful implementation.

Score meaning:

  • (1) I strongly disagree
  • (2) I disagree
  • (3) I neither agree nor disagree
  • (4) I agree
  • (5) I strongly agree

that it is highly likely that this proposal will be implemented successfully.

Examples for a considered assessment statement:

This team has been participating in the Cardano community for a long time and have been involved with Catalyst from the beginning of the project. I believe they have the skills to inform and educate new members about Catalyst, as they are already doing this. The presented plan is comprehensible but I think it is important to have more clarity in the description of what the membership model is and its relevance in the proposal. The proposers will approach new people for Catalyst through meetups. I think it is important to describe how this will be done in more detail to better understand the approach. (Score given: 3)

The proposer team states that it has the necessary experience. However, they did not provide any links or references to support this statement. My recommendation for a future proposal is to add this information so that the voters are able to validate the experience. The budget provided is very general and therefore I cannot say whether it is appropriate or not. The implementation plan is comprehensible but leaves important questions unanswered: Where do you get the users from? To create the proposed solution is one thing, but operating it successfully is another. If the described aspects are improved, then this proposal has a significantly better chance of obtaining successful funding. Based on the given information, I don’t expect a successful implementation. Therefore, I rate feasibility with a score of 2.

Potentially an intriguing concept but there are some major concerns to me. First, there is a budget figure of 200k USD requested which is half the fund of this challenge. Given that amount, I would expect an appropriate level of depth when addressing how this budget is going to be spent. I tried to open every attachment and read through all segments - but just couldn't locate it. Second - I am unable to establish the validity of claims/experience. Third, the proposal itself reads very poorly. It reads like a concept more than an executable proposal. It needs more editing so that it can be assessed properly given its direction. Perhaps reworking it and submitting it at the later stages will work. (Score given: 1)

# 3) Clarity/Auditability

Scoring the auditability of a proposal means that you are assessing if the information provided is sufficient to audit the progress and the success of the proposal. The audit phase is essential to ensure the implementation. After successful funding, the proposer periodically reports the progress to the community. Therefore, the community needs to know what to expect from the proposer up front. As you work through the proposals look for milestones with identifiable measures of success which when achieved can be measured by community.

An auditable proposal has considered the following aspects:

  • A roadmap with milestones and a time horizon for achievement.
  • Metrics/KPIs that define the success of the proposal.
  • Clear understandable description of the targeted problem.
  • Clear understandable solution that addresses the problem.

Score meaning:

  • (1) I strongly disagree
  • (2) I disagree
  • (3) I neither agree nor disagree
  • (4) I agree
  • (5) I strongly agree

that this proposal provides me with sufficient information to assess the progress in attaining its stated goals.

Examples for a considered assessment statement:

The proposer provides a step-by-step plan for the implementation with clear milestones and a defined time horizon. There are also several metrics that measure the success of this project. I recommend adding further metrics like “...” so that more areas of this project are auditable. If the proposed solution is successfully implemented it provides a useful solution to the targeted problem. The only thing I miss in this area is that the proposer has not mentioned possible difficulties and how to deal with them. The quality of the proposal suggests that the proposer is aware of the challenges, but it is easier for outsiders to understand if a statement is made. In summary, I rate auditability with a score of 4.

The proposal lacks some important information, such as roadmap, estimates of content to be produced, details on budget allocation and details about the proposers (it was mentioned that one of the proposers is an SPO, but the pool ticker was not included), I recommend including this information so that the community can have a better feasibility assessment and understand the impact of the proposal. (Score given: 3)

Roadmap and milestones (incl. time horizon) are weakly described. It is only mentioned that further funding will be needed. The given metrics measure what the proposer wants to do, but the success of his actions is not defined (for example: you don't measure the number of emails that are sent alone, you have to measure the response rate). As mentioned above, the description of the solution is inadequate and confusing. Possible challenges are not considered. In conclusion, the progress of this proposal is hard to assess. (Score given: 2)

# Scoring Criteria - Community Challenge Setting

Community-defined challenges are critical for Cardano community self-governance, and generate a unique way to achieve Cardano's mission.

Challenge setting is different from other challenges as it's about proposing new challenges, whether regular challenges, or about proposing initiatives to address challenges.

  1. # Alignment

This challenge is critical to achieve Cardano's mission.

  1. # Feasibility

The Catalyst community has the capacity to address this challenge. Bonus if there's an established challenge team.

  1. # Verifiability

Success criteria and suggested metrics are set correctly to measure progress in addressing the challenge.

# Catalyst Fund8 Challenge Setting Strategic Goals

Fund8 Challenge Setting has taken input from community and IOG stakeholders as part of a horizon scanning activity to help focus and centre Community Challenge Setting on some core themes and attributes.

Challenges should be aligned with the Fund8 strategic goals:

  1. Prepare a group of people willing and able to make contributions to the ecosystem.
  2. Turn Cardano into an open source project.
  3. Accelerate the growth and evolution of developer & app ecosystem.

If Fund8 Challenge Setting proposals do not meet one or more of the above attributes, then assessment prioritisation by CAs and voters should favour proposals which do meet these goals.

Learn more about what Challenge Setting is about from this time stamped presentation by James Dunseith of Gimbalabs: https://youtu.be/nP0r5KwNtQk?t=4790

# Definitions of a quality assessment

These are Catalyst definitions that are applied to help participants, including Community Advisors (CA), veteran community advisors (vCA), voters, and proposers to understand how assessments and review feedback should be provided so that:

  • CA are clear on what is expected of them for assessments
  • vCAs are clear on what is expected of them reviewing CA assessments

These definitions have been set in consultation with Catalyst Community Advisors (CAs) and veteran Community Advisors (vCAs)

Guidance on what is expected
Focus Why is this important? EXCELLENT (where most or all are achieved) GOOD
Positive feedback-loop _CAs that provide practical coaching help the proposer to deliver a better project._The reader (vCA and proposer) should be able to understand why these guiding points are important to the community advisor and how addressing them would improve scoring the proposal. Actionable coaching points are provided to guide the proposer about what is and isn’t strong about the proposal, and / or what could be improved to become a more compelling proposal in the future. CA has provided feedback about the strengths and weaknesses in the proposal; however the feedback either lacks clear actionable coaching points or the action points appear to be not so carefully considered to be useful to the proposer.
CA contemplation of their own ability to assess a proposal Individual CAs may come from a wide range of backgrounds and experience levels. CAs are encouraged to provide sufficient information regarding their relevant qualifications and knowledge regarding a proposal. Alternatively, they can refer to expert sources of information when providing feedback. Lack of such information may indicate a more subjective opinion to the reader (voters, proposers, vCAs). The CA has been able to define which assertions they are better qualified to make in the proposal about its fit to address the Catalyst Challenge and feasibility of plans, versus which opinions are more intuitive, or outside the advisor’s domain of expertise. The CA has provided no clarity about which of their assertions are made based on their own domain expertise, and which are more intuitive or they have less experience or knowledge in.OrIt is unclear how the CA has arrived at their judgments if the assertions made are outside of the CAs domain expertise, or in the case that the CAs domain expertise remains unknown to the reader.
CA scoring against Challenge criteria _The reader (a vCA or proposer) should be able to understand how the score reflects the advisor’s perspective and feedback provided._Specifics within the proposal are highlighted to illustrate understanding, and the advisor has provided supportive rationale that is aligned with the criteria. The CA has provided a reasonable and thoughtful justification to support the scoring. CA feedback addresses specific content assessed in this proposal. CA feedback is comprehensive and demonstrates that the entirety of the proposal was considered. CA score and comments are well balanced and logical, indicating that the feedback was based on a sound analysis of all the information provided The CA has made a fair attempt to justify their scoring against the challenge criteria; however the feedback generally lacks specificity or clarity to support their rationale and scores.
CA opinion on the proposal overall Each challenge has its own intended outcomes. The job of assessors and voters is to qualify the validity and appropriateness of the activities described in the proposal and if it is a good use of Cardano Treasury funds to achieve these. CA feedback comments are unambiguous and they demonstrate a strong grasp of the objectives of the Catalyst challenge and the project team’s objectives to address the challenge. CA commentary demonstrates a basic understanding about the project’s objectives to address the challenge however there are gaps identified in their feedback or the feedback lacks clearly documented understanding
Accountability to the Community Measures being taken so the project team remains accountable to the Community is at the heart of Catalyst. The CA can clearly identify how the proposing team will demonstrate their positive return on intention. CA scoring is backed up with supporting rationale whether they either agree or respectfully disagree with the realism of the plans and budget to deliver value for money appropriate to the project’s intentions. The CA has provided feedback however there are gaps identified and feedback is missing related to the proposers auditability, project plans or budget appropriateness to deliver the project’s intentions.

# Inadequate assessment statements:

Rationales similar to these given below are not in accordance with the standards of these guidelines as they do not offer guidance to voters and to proposers, and will be filtered out. To have your assessment counted towards the overall rating, inadequate assessment statements should be avoided.

For information about the reviewing-the-assessments process, please follow the guidelines for vCA (Veteran Community Advisors).

# No rationale given:

  • This proposal is only fishing for funds.
  • This proposal is irrelevant.
  • This proposal is missing the point.
  • I can't rate the proposal.
  • Please don’t vote for this proposal.
  • It is a very good proposal. Should be implemented.
  • This proposal [does not] effectively addresses the challenge.
  • It is highly [un]likely that this proposal will be implemented successfully.
  • This proposal [does not] provides me with sufficient information to assess the progress in attaining its stated goals.

# Only a personal opinion given:

  • This proposal is terrible.
  • I like the idea but not the proposer.
  • Is this a joke?
  • Too many similar proposals.
  • Asking for too much funds.

# Unconstructive assessment:

  • The proposal is poorly written, there is not enough information to properly understand it and it’s purpose completely mismatches the challenge goals.
  • The proposer’s experience is not enough to execute the project, and there is no information about hiring experts. Very unlikely to be successful.

# Further aspects of an inadequate assessment:

  • The assessment is given from a biased perspective.
  • It is clear that the proposal was not read through by the CA.
  • It is clear that the scoring criteria and assessment guidelines were not applied or understood by the CA.
  • The CA lacks clear articulation of the rationale for the assessment in a constructive way.

# The assessment has been completed.

# What happens next?

# Reviewing-the-assessments process

This is a self correcting process which precedes the Ballot Submission phase and provides guidance to the Voting Committee.

  1. CA's finish assessments.
  2. An anonymized CSV-file containing all assessments will be passed for peer review by veteran CA's who will filter out substandard assessments. (Tool such as the vCA-tool (opens new window) exist to assist with this).
  3. Proposers will also be invited to report substandard assessments.
  4. Experienced CA's (known as vCAs - veteran Community Advisors) decide which reported assessments should be excluded.
  5. A Catalyst admin removes the results of step 4 and any blank assessments from final published assessments.
  6. A list of all removed assessments will be published and a retrospective to review the process for the next fund will be held.

# Ballot Submission

  • The assessment score and a link to the proposal's URL on the Ideascale innovation platform will be visible from the Catalyst Voting app.
  • All rationales provided by CA's will be attached to the original proposal in Ideascale.

# Anonymity

Identity of the CA will be kept anonymous in Ideascale. However, despite our best efforts and testing, there's always a small risk, because of the nature of Ideascale being a 3rd party tool, and anonymity being an experimental feature, that the CA's identity might become known. Please make sure you understand that risk before assessing proposals.

Note: A CA is not eligible for incentives if he receives a red flag for inadequate assessments during reviewing-the-assessments process by vCAs.

Fund 6 Reward Scheme Upgrade for Catalyst Community Advisers

Goal of incentives is to assure CAs will provide a thoughtful and fair assessment to all proposals, and that each proposal will receive at least three assessments. Therefore:

A total of 5% of the challenge fund amount will be allocated as incentives for CAs and vCAs:

  • 4% will be allocated as rewards for assessments to CAs
  • 1% will be allocated to incentivize those reviewing the assessments (vCAs)

Goals of the reward upgrade:

  • Grow the community of Community Advisors sustainably, keeping a core of qualified professionals, with fair rewards as an integral part
  • Provide incentives for high quality reviews
  • Provide comprehensive analysis for high budget proposals
  • Ensure that each proposal is reviewed and CAs reviews are quality assured

in order to provide Community Advisors with:

  • Transparent reward scheme
  • Predictable reward amount dependent on quality level of the review;
    if there are risks - they must be understandable and have clear strategies to be handled
  • Support for an option for gradual skill improvement: seasoned reviewers would get greater rewards for high quality reviews, new reviewers would still have a reason tocontribute.

For more details on this new rewarding algo - please reference Catalyst town hall slides here (opens new window)and video explanation here (opens new window).

# Main changes to rewards from previous funds

Implemented since Fund7

  • No major/breaking changes introduced

Implemented since Fund6

For Community Advisors:

  • The number of assessments rewarded per proposal increases from 3 to 5.
  • Assessments are going to be classified as Excellent / Good / Filtered Out.
  • Excellent assessments will receive 3 times more rewards than Good ones
  • Filtered out assessments will not receive rewards and will not influence the final score
  • Each proposal rewards budget fits 2 Excellent and 3 Good assessments.
  • If there are more than 2 Excellent or 3 Good assessments, a lottery system is used.
  • Bonus rewards are available (20% of total CA rewards budget) for assessments. ‘Approved’ by the community, proportional to the proposal budget.

Veteran Community Advisors (vCAs)

  • Rewards are proportional to the number of reviews made

Please direct any further questions to the _Telegram channel: CatalystCommunityAdvisors (opens new window)

The Project Catalyst team thanks all our CAs for helping to build the future of Cardano!