Grader Internals

The Grader is the central orchestration component of omegaUp's evaluation system. Written in Go, it manages submission queues, coordinates Runners, and determines final verdicts.

Architecture Overview

flowchart TB
    subgraph Frontend
        PHP[PHP Controllers]
    end

    subgraph Grader
        API[Grader API]
        Queue[Queue Manager]
        Dispatch[Runner Dispatcher]
        Validator[Output Validator]
    end

    subgraph Runners
        R1[Runner 1]
        R2[Runner 2]
        R3[Runner N]
    end

    subgraph Storage
        Git[GitServer]
        DB[(MySQL)]
    end

    PHP -->|HTTPS + Client Cert| API
    API --> Queue
    Queue --> Dispatch
    Dispatch --> R1
    Dispatch --> R2
    Dispatch --> R3
    R1 -->|Results| Validator
    R2 -->|Results| Validator
    R3 -->|Results| Validator
    Validator --> DB
    API --> Git

Queue System

The Grader maintains eight priority queues to handle different submission types:

Queue Types

Queue Routing Logic

flowchart TD
    Submit[New Submission] --> IsContest{In Contest?}
    IsContest -->|Yes| IsUrgent{Urgent Contest?}
    IsContest -->|No| IsRejudge{Rejudge?}

    IsUrgent -->|Yes| IsSlow1{Slow Problem?}
    IsUrgent -->|No| IsSlow2{Slow Problem?}

    IsSlow1 -->|Yes| SlowUrgent[Slow Urgent Queue]
    IsSlow1 -->|No| Urgent[Urgent Queue]

    IsSlow2 -->|Yes| SlowContest[Slow Contest Queue]
    IsSlow2 -->|No| Contest[Contest Queue]

    IsRejudge -->|Yes| IsSlow3{Slow Problem?}
    IsRejudge -->|No| IsSlow4{Slow Problem?}

    IsSlow3 -->|Yes| SlowRejudge[Slow Rejudge Queue]
    IsSlow3 -->|No| Rejudge[Rejudge Queue]

    IsSlow4 -->|Yes| SlowNormal[Slow Normal Queue]
    IsSlow4 -->|No| Normal[Normal Queue]

Slow Problem Definition

A problem is considered "slow" if: - Time limit exceeds 30 seconds - Or problem is explicitly marked as slow

Slow Queue Constraint: Only 50% of Runners can simultaneously process slow queue submissions to prevent resource starvation.

Runner Coordination

Runner Registration

1. Runners connect to Grader on startup
2. Register with their capabilities (memory, CPU)
3. Re-register every 60 seconds as a heartbeat
4. Grader removes Runners that miss heartbeats

Dispatch Algorithm

The RunnerDispatcher.dispatchLocked() method:

1. Selects highest-priority submission from queues
2. Finds an available Runner (round-robin, no affinity)
3. Creates a GradeTask with 10-minute deadline
4. Dispatches to Runner via HTTPS

Priority Order:
Urgent → Slow Urgent → Contest → Slow Contest → 
Normal → Slow Normal → Rejudge → Slow Rejudge

Execution Flow

sequenceDiagram
    participant G as Grader
    participant R as Runner
    participant S as Sandbox

    G->>R: Dispatch submission
    R->>R: Download input files
    R->>R: Compile source code

    loop Each test case
        R->>S: Execute in sandbox
        S-->>R: Output + metrics
        R->>R: Compare output
    end

    R-->>G: Results (verdicts, times)
    G->>G: Calculate final score
    G->>G: Update database

Verdict Determination

Verdict Types

Scoring Logic

1. Per-test-case verdict: Each test case gets a verdict
2. Weight assignment:
3. If /testplan exists: parse explicit weights
4. Otherwise: equal weights (1/N for N cases)
5. Group scoring: Cases grouped by filename prefix (before first .)
6. Final score: Weighted sum of group scores

Testplan Format

# Group weights (optional)
[group1]
weight = 30

[group2]  
weight = 70

# Test case to group mapping
group1.case1.in
group1.case2.in
group2.case1.in
group2.case2.in

API Endpoints

The Grader exposes these internal APIs (accessed via HTTPS with client certificates):

Submission Management

Status & Control

Configuration

The Grader is configured via JSON:

{
  "Grader": {
    "BroadcasterURL": "https://broadcaster:32672/broadcast/",
    "GitserverURL": "http://gitserver:33861",
    "GitserverAuthorization": "OmegaUpSharedSecret secret omegaup:grader",
    "V1": {
      "Enabled": true
    }
  },
  "Runner": {
    "PreserveFiles": false
  },
  "Logging": {
    "Level": "info"
  }
}

Key Settings

Monitoring

Status Endpoint Response

{
  "status": "ok",
  "grader": {
    "running": true,
    "queue_length": {
      "urgent": 0,
      "contest": 5,
      "normal": 12,
      "rejudge": 3
    },
    "runners": {
      "total": 4,
      "available": 2
    }
  }
}

Metrics Port

The Grader exposes Prometheus metrics on port 6060:

• Queue lengths by type
• Runner availability
• Grading latencies
• Error rates

Security

Authentication

• PHP Frontend uses client certificates to authenticate
• Shared secret for internal service communication
• All traffic encrypted via HTTPS

Sandboxing

All code execution happens in sandboxed Runners (see Runner Internals).

Source Code

The Grader is implemented in the quark repository:

• cmd/omegaup-grader/ - Main entry point
• grader/ - Core grading logic
• runner/ - Runner coordination

Related Documentation

• Runner Internals - Code execution details
• System Internals - Full request flow
• Features: Grader - High-level overview