All tasks have an optional review. As an admin you set this review frequency between 0 and 100% of the tasks. The moment each task is complete the reviewer is able to access the task. This is per task, so work can flow freely. The reviewer can pass or reject the task. Optionally they can leave a general comment, or a more specific issue on a specific annotation. The results of this review ratio can be further analyzed eg "Who always is getting them wrong".
Fully integrated Quality Control Slideshow helps reviews view each annotation one at a time.
Fully integrated issue tracking.
You can access an Annotation capability, without risking real work, through the Exams practice function
Each time an Annotator goes to start an Exam, it's a fresh copy. If you are familiar with Video Game instances it's similar to that concept.
Afterwards there is a process to grade and optionally award credentials.
These credentials can be optionally required on future tasks.
Basically it allows you to enforce the exam requirement - so a user can't start working on a job till the exam that's relevant to it is complete.
For the grading process, for a few data types this can be nearly automatic, but for many this will require a human review.
Diffgram is one of the only options I'm aware of that offers this level of depth of integrated Exams.
Exams Benchmark Introductions
See annotations, time spent, and more, all per user.
Easily see the status of a given batch of work.
This lets you set different steps, etc. This is a super flexible workflow concept.
Use Interactive Annotation Automation to easily create customized quality assurance scripts.
Automatic Task Distribution
To manage workflow at scale it takes a team. And that team has distinct roles, like admin and annotator. Diffgram has persona focused user experiences optimized for every member of your team.
Annotate. Resume. Browse
Quickly identify what data to label and reduce costs labeling data that doesn't improve the model.
Easily query data after the fact to find errors.
Skip downloading and unzipping massive datasets. Explore data instantly through the browser.
Visualize in seconds multiple datasets. Compare models without extra computation.
(More automatic surfacing features coming soon)
Please feel free to create a new issue
Updated almost 2 years ago