Release Criteria

i. All Phase 1 features are implemented and pass functional user acceptance testing.

ii. Security testing validates system robustness.

iii. A limited set of legal documents is available in a test environment (Need documents to test on).

iv. Multiple document upload of multiple document types (Phase 1 will be limited to .pdf and .docx document types).

v. Acceptable confidence and accuracy levels of LLM using zero-shot approach.

1. Azure OpenAI comes pre-trained with vast amounts of anonymized data, so you typically don't need to pre-train it yourself. It's ready to use out of the box, which means you can utilize it for various tasks like document vectorization without needing additional training. This is often referred to as zero-shot learning. In some cases it would require fine-tuning.
2. Confidence
   • Evaluate quality of embeddings themselves (intrinsic evaluation methods like similarity analysis or clustering)
   • Retrieval score (each document gives a score based on the similarity of query made and embeddings themselves)
   • Thresholding to control minimum required similarity for considering a document relevant.
3. Accuracy
   • Qualitative evaluation
   • Targeted benchmarking - use small, curated dataset to calculate accuracy metrics.
   • Similarity based metrics
4. Notes
   • Baselines
     • Compare the model against simpler baselines (like keyword matching) to gauge the relative improvement.
   • Iteration
     • Zero-shot is often an iterative process. Use the mentioned evaluation methods to identify weaknesses and refine the embedding and retrieval approach over time.

vi. Acceptable metrics for back-end based on performance levels for stress and load testing of APIs for a maximum concurrent user. Use of 20 users.

vii. Acceptable lighthouse metrics for front-end based on performance, SEO, accessibility, and best practices.