Testing Marcus: Dark Mode Toggle Project

by SLV Team 41 views
Testing Marcus: Dark Mode Toggle Project

Hey guys! Let's dive into a cool test scenario to see how well Marcus can handle a straightforward project from start to finish. This is all about validating his ability to manage tasks intelligently and stick to the rules we set. Specifically, we're going to see how he tackles adding a dark mode toggle to a Chrome extension using plain old vanilla JavaScript. This isn't just a simple test; it's a deep dive to ensure Marcus can understand project requirements, create efficient task patterns, and execute them perfectly. This test helps us validate core functionalities, making sure Marcus is not just good but great. The main goal here is to make sure that Marcus correctly identifies the work needed, creates the right tasks, and completes the project autonomously and efficiently. So, buckle up; we're about to explore the inner workings of Marcus!

The Dark Mode Challenge: Project Setup

Alright, let's set the stage. The project is simple: add a dark mode toggle to a Chrome extension. We'll kick things off with a specific command: create_project("Add dark mode toggle to chrome extension with vanilla JavaScript", "dark-mode-feature", {"complexity": "standard"}). This command is the starting point, the prompt that will set Marcus in motion. The core of this test lies in the setup phase, where the correct project initialization ensures the whole process runs smoothly. The "vanilla JavaScript" constraint is key here, making sure Marcus does not use any extra frameworks. Think of it as Marcus going back to basics. This constraint is not just a preference; it's a critical part of the test. The project is designed with a straightforward task: implement the dark mode toggle. We're expecting one single task, which simplifies the process and makes it easier to track Marcus's performance. This atomic feature approach allows us to focus on the core functionality without extra layers of complexity. This initial setup sets the expectation for a streamlined process, which will be our main validation step.

Expected Flow Breakdown

  1. Project Initiation: The process starts with a single command. The command serves as the directive, and it includes details like the project description, feature identification, and the level of complexity. The objective of this setup is to ensure that Marcus can accurately interpret the initial requirements. It's the first test of Marcus's understanding of our needs.
  2. Feature Classification: Upon receiving the command, Marcus is expected to classify the feature as atomic. This means it is a single task that is easy to manage. The success in this phase is key to making sure that Marcus can break down the projects correctly.
  3. Task Creation: We expect Marcus to create only one task: "Implement Dark Mode Toggle." This indicates that Marcus understands the scope of the project and its requirements.
  4. Autonomous Execution: An agent will pick up this task and start working on it independently, implementing the dark mode toggle, committing the changes, and pushing them to a branch. This part tests the autonomous action of the agent and shows how it works with the system.

Success Criteria: What We're Looking For

Here’s what needs to happen for us to say Marcus nailed it. The success criteria are designed to offer a detailed evaluation of Marcus’s capabilities in managing and executing tasks. These criteria offer a clear path to test and validate the project end-to-end.

  • Single Task Creation: Only one task should be created, focusing solely on the implementation aspect. This approach validates Marcus’s ability to understand and not overcomplicate tasks.
  • Vanilla JavaScript Adherence: The task description must respect the "vanilla JavaScript" constraint. There should be no signs of using any external frameworks.
  • Autonomous Task Completion: The agent assigned to the task needs to complete the implementation without needing any outside help.
  • Git Commit Integrity: The Git commit message must include the task ID in the format: feat(task-XXX): implement dark mode toggle. This demonstrates the seamless integration of task management into the development workflow.
  • Dedicated Branching: The code must be pushed to a dedicated branch, which helps maintain a structured and organized version control system.
  • Completion Reporting: The agent should report its completion and request the next task. This is a critical step in a continuous, streamlined workflow.
  • Graceful Exit: Upon completion of all tasks, the agent should exit gracefully, indicating an efficient and complete execution.

Validation Steps: The Testing Process

Here’s a breakdown of the steps we’ll take to make sure Marcus does everything right.

  1. Test Environment Setup: Setting up a test environment is the first thing. We need to prepare Marcus and a single agent in a controlled environment.
  2. Project Initiation: We will run the create_project() command with the specified description and parameters. This initializes the project, which is the starting point for Marcus.
  3. Task and Classification Verification: After the project creation, we'll check how many tasks were created and what the project classification is. This verifies that the initial understanding and breakdown by Marcus is correct.
  4. Agent Execution: Launch the agent and observe its actions as it executes the task autonomously. Observe its behavior to see if it works as expected.
  5. Git History Review: The most important step here is to check the Git history to confirm the correct commits. This verifies that the task ID is included and that the code is pushed to the designated branch.
  6. Branching and Traceability: We will check the branch naming to confirm that everything is organized and follows the expected pattern. This shows how well Marcus keeps the workflow organized.

Expected Results: What We Hope to See

Here's a look at what we’re hoping to see during the test. We're setting these expectations to measure Marcus's performance accurately. These metrics will allow us to assess the accuracy, efficiency, and adherence to the project constraints. The expectations are designed to reflect the ideal outcomes we anticipate.

  • Task Count: We're expecting only one task. This directly validates Marcus's ability to efficiently break down the work.
  • Complexity Classification: The classification should be atomic. This shows that Marcus understands the scope of the project.
  • Task Pattern: The task pattern should be "implementation". This confirms that Marcus chose the right approach for the project.
  • Execution Time: The estimated time is between 15-30 minutes. This gives us an idea of how fast Marcus can complete a task.
  • Git Commits: Expecting 1-2 commits. This will confirm the correct workflow.
  • Constraint Violations: We're aiming for zero violations, particularly no framework mentions. This validates Marcus's ability to stick to the given constraints.

Code References: Where to Find the Good Stuff

These code references are key to understanding the inner workings of Marcus. This helps us trace the functionalities that are responsible for the project management, task pattern selection, and constraint handling.

  • Task Pattern Selection: The src/ai/advanced/prd/advanced_parser.py::_select_task_pattern() (PR #112) is where Marcus decides how to break down the task. This file is critical for understanding the intelligence of the system.
  • Constraint Propagation: src/ai/advanced/prd/advanced_parser.py::_generate_task_description_for_type() (PR #114) handles the constraints, ensuring that Marcus does not use disallowed frameworks. This is what makes Marcus follow all the rules.
  • Agent Workflow: You can learn about the agent's workflow by looking at prompts/Agent_prompt.md. This gives you an understanding of how the agent executes the tasks and works with Marcus.

Related Aspects: The Big Picture

This test is more than just about adding a dark mode toggle; it’s a key part of our larger plan. This is connected to our broader validation goals and the Value Propositions.

  • VALUE_PROPOSITIONS.md Phase 1: This is part of the initial validation phase.
  • Natural Language to Execution: It validates our claims that Marcus can understand natural language and execute commands on its own.
  • Intelligent Scaling: Demonstrates how Marcus can handle projects of different complexities.
  • Constraint Enforcement: It proves that Marcus can follow the project's rules.

Alright, that's the plan, guys! Let's get this test running and see Marcus in action! We're excited to see how he handles the challenge! Let's get started!