Video Processing: SRS Alignment & Job Flow

Hey guys, let's dive into the Video Processing Service (VPS) and how we can align it perfectly with the System Requirements Specification (SRS). This is crucial for the entire system to work seamlessly, especially when dealing with encode/decode job orchestration, cryptography integration, and job lifecycle management. Our goal is to make sure the VPS not only meets but exceeds the standards laid out in the SRS, ensuring smooth operation and data security. So, let's break down the essential steps needed to bring the VPS up to speed, focusing on the key areas that need attention.

Decoding the SRS Responsibilities

First off, let's get on the same page by reviewing the responsibilities the VPS shoulders according to the PQC-VStego_SRS_v1dot7.md document. Understanding these requirements is the cornerstone of our development efforts. The VPS needs to be a well-oiled machine, handling various tasks with precision and efficiency. Key responsibilities include:

• Kafka Topic Consumption: The VPS must diligently listen to and consume messages from the job.request.encode and job.request.decode Kafka topics. This means the service should be actively monitoring these topics, ready to spring into action whenever a new job request pops up. This is our entry point for job processing.
• Encode Job Execution: For encode jobs, the VPS has a series of important steps. First, it fetches a secret file from the File Service. Then, it calls the Cryptography Service's /encrypt-and-sign endpoint to get a signed and encrypted payload. After that, it dives into the fascinating world of steganography using ffmpeg to embed this payload into a video. Once the stego video is ready, the VPS uploads it to the File Service. Finally, it publishes a job.completion message to Kafka, signaling that the job is done.
• Decode Job Execution: Decode jobs work a bit differently. The VPS fetches or streams the stego video. Using ffmpeg and LSB extraction, it pulls out the payload. It then calls the Cryptography Service's /verify-and-decrypt endpoint. Finally, it uploads the decrypted secret to the File Service and publishes a job.completion message to Kafka.
• Streaming Operations: All of this has to happen without creating temporary files, using stdin/stdout piping. This is to ensure an efficient and resource-friendly operation. File uploads and downloads need to leverage streaming APIs, maximizing speed and minimizing resource consumption. This is a critical aspect for optimized performance.

This is a lot, right? But fear not; we'll break it down step by step to ensure we get everything right!

Identifying Current Implementation Gaps & Problems

Now, let's take a hard look at the current state of the VPS. We need to identify any areas where the implementation falls short of these requirements. This self-assessment is key to finding and squashing any bugs. Here's a rundown of the major gaps and problems we need to tackle:

• Kafka Job Consumption: Currently, the service has a JobConsumerService and an implementation, but it's not actually listening to the crucial job.request.encode or job.request.decode topics. The methods are essentially empty placeholders. This means the service isn't even aware of incoming job requests, which is a major roadblock.
• Cryptography Service Integration: While the steganography logic for embedding and extracting is in place, there's no actual communication with the Cryptography Service endpoints during the encode and decode workflows. This is a big problem because it undermines the security measures that are supposed to protect our data.
• Job Completion: The service doesn't publish job.completion messages to Kafka after a job finishes (or fails). This is a critical part of the job lifecycle. Without these messages, the Orchestration service won't know whether a job has finished successfully or if something went wrong. This breaks the expected workflow and makes it hard to manage and monitor jobs.
• Streaming and File Handling: The service uses ffmpeg correctly with pipe:0/pipe:1 for streaming, which is great. However, file uploads are currently going to GridFS instead of using the direct streaming approach. This may require some coordination with the Orchestration changes to optimize file handling. This can cause bottlenecks and affect overall performance.

These gaps highlight areas where we need to focus our efforts to bring the VPS into alignment with the SRS.

Action Items: Bridging the Gaps

Alright, it's time to roll up our sleeves and fix these problems. Here's what we need to do to bring the VPS in line with the SRS:

1. Implement Kafka Listeners:

   • We'll add @KafkaListener methods for the job.request.encode and job.request.decode topics. This is the starting point, enabling the VPS to listen for and react to job requests.
   • Inside these listeners, we'll implement the job-handling logic in the handleEncodeRequest and handleDecodeRequest methods. This is where the core work of encoding and decoding happens.

2. Integrate Cryptography Service:

   • For encoding, we'll fetch the secret from the File Service. Then, we will call the /encrypt-and-sign endpoint (using streaming for efficiency). Finally, we'll embed the resulting payload into the video.
   • For decoding, we'll extract the payload from the stego video. We will then call the /verify-and-decrypt endpoint (also using streaming). Afterward, we'll upload the decrypted result.

3. Produce Job Completion Events:

   • We'll make sure that after a job is successfully processed (or if it fails), the VPS publishes a KafkaJobCompletion message to Kafka. This message will include the job's status, the ID of the output file (if applicable), and any errors that occurred. This ensures the job lifecycle is properly managed and monitored.

4. Streaming-first File Handling:

   • We'll ensure all file operations use streaming APIs to avoid creating temporary files. This is important for performance and resource management.
   • We'll coordinate with the Orchestration service to remove unnecessary GridFS uploads for carrier and stego files. This will optimize file handling and streamline the workflow.

5. Testing and Compliance:

   • We'll add comprehensive integration tests to cover the complete encode and decode job flow. This will make sure everything works correctly, including cryptography and file streaming.
   • We'll rigorously validate that all SRS constraints (C-7, C-6, etc.) are met. This is a must for ensuring the service is compliant with the specifications and all regulatory requirements.

By following these steps, we'll bring the VPS up to speed, address all the gaps, and ensure it functions properly.

Referencing Key Documents

Here are the critical documents that we'll be using as references during this process:

• SRS: PQC-VStego_SRS_v1dot7.md (specifically focusing on FR-B-4, FR-B-5, C-6, and C-7).
• SteganographyServiceImpl.java: This is where the steganography magic happens. We'll be updating this file with the cryptography integration.
• JobConsumerServiceImpl.java: This is where we'll add the Kafka listener implementations.
• HybridController.java: This file, located in the cryptography-service, contains the endpoints we need to call for encryption, decryption, and signing.

Priority and Labels

Priority: High — This is a critical task, as it's required for the proper functioning and SRS compliance of the entire system.

Labels: enhancement, bug — These labels will help categorize the issues and track their progress.

That's the roadmap, guys. By tackling these action items, we'll make sure our Video Processing Service is reliable, secure, and compliant with all the specifications. Keep up the excellent work, and let's get it done! Remember, this is about ensuring the system works correctly and that our data is secure. Let's make it happen!