Job Queues

The jobQueue library in Go aims to serve as a Cron replacement, enabling the scheduled and event-driven execution of tasks in an organized manner.

Library Requirements

Core Requirements

1. Job Management Structure: A structure to catalog all jobs, providing methods to add and remove jobs.

2. YAML Import: A function to import job definitions from a YAML file.

3. Job Exclusivity: Jobs should be either concurrently runnable or exclusive.

4. System Resource Monitoring: Real-time monitoring of system resources like CPU, memory, etc., consumed by the jobs.

5. Priority and Distribution: Each job should have a priority. Jobs should also be evenly distributed across available resources.

6. Single and Recurring Jobs: Support for both one-time and recurring jobs.

7. Logging: Each job should have a log written that includes Process ID, memory usage, start time, end time, and exit code.

8. Priority Escalation: If a job is not run due to its priority, its priority should escalate over time.

Additional Requirements

1. Error Handling: Mechanisms for effective error handling, especially for failed or terminated jobs.

2. Notification System: Optionally, an interface for notifications on job failures or completions.

3. Resource Limits: Ability to set resource limits per job.

4. Job Dependencies: Optional support for jobs that depend on the successful completion of other jobs.

5. Testability: The library should be easy to test, ideally via unit tests.

6. Documentation: Comprehensive API documentation and a guide on how the library works.

Documentation

JobData Structure

Fields

• Id (JobIDType)

  • Unique identifier for the job.

• Priority (int)

  • The priority level of the job in the queue.

• Exclusive (bool)

  • Specifies whether the job should be executed exclusively or not.

• MaxRuns (int)

  • The maximum number of times the job should be executed.

• Concurrency (int)

  • The maximum number of concurrent executions of the job.

• LastRun (time.Time)

  • Timestamp for when the job was last executed.

• NextRun (time.Time)

  • Timestamp for the next scheduled execution of the job.

• Logs ([]JobLog)

  • An array of log entries for the job.

• Schedule (string)

  • A cron expression that defines the execution schedule.

• Status (JobStatus)

  • The current status of the job (e.g., Running, Completed).

• Timeout (time.Duration)

  • The maximum duration the job is allowed to run.

• Retries (int)

  • The number of times the job will be retried if it fails.

• RetryDelay (time.Duration)

  • The delay before retrying the job if it fails.

• ResourceLimits (struct)

  • Resource limits for CPU and Memory.

    • CPULimit (float64)

      • CPU limit for the job.

    • MemoryLimit (uint64)

      • Memory limit for the job.

• Dependencies ([]JobIDType)

  • List of job IDs this job depends on.

• Tags ([]string)

  • Tags associated with the job for easier categorization.

• Metadata (map[string]interface{})

  • Metadata for extending the structure with additional information.

• Stats (JobStats)

  • Job-related statistics.

Job Structure

Inherits fields from JobData and adds additional functional fields.

Fields

• Runnable (Runnable)

  • A function or method that defines the action of the job. This is what gets executed.

• scheduleImpl (cron.Schedule)

  • Internal cron schedule implementation that interprets the Schedule string in JobData.

• *TelemetryHooks ([]func(JobLog))

  • An array of functions that will be called for telemetry during the job's execution. These hooks can update a JobLog object.

• Failover (func() error)

  • A function that gets called if the primary execution fails for some reason. It's meant for fail over mechanisms.

• ctx (context.Context)

  • A context that can carry deadlines, cancellations, and other request-scoped values across API boundaries and between processes.

• mu (sync.Mutex)

  • Mutex for synchronizing access to the job's fields, making the job safe for concurrent use.