Testing an Elastic Agent with an AI Agent (Warp + Claude)

Test an Elastic Agent using AI Agent generated code

In this post, I want to discuss using Generative AI to generate test scripts for an Elastic Agent configuration. I would not create these scripts by hand; it takes too much effort. Working together with an AI Agent makes it easy to create these test scripts.

I used Warp together with Claude to create the code base. The code base is available here:

https://github.com/jettro/test-elastic-agent

Background story

#

I am currently creating an observability platform for a client, using the Elastic Stack. It is essential to ingest logs or events from multiple applications. Each application has different requirements. One application produces a lot of auditing logs. The application owners are only interested in some of these logs. You have multiple solutions, each with its advantages. The default approach is to start with an Elastic Agent pulling log files, piping the logs through Logstash or sending them through an ingestion pipeline in Elasticsearch. For this use case, the number of logs is high, and the logs of interest are just a small percentage. Therefore, we want to filter the logs at the source, within the agent, before sending them off to Elasticsearch.

Warp

#

Warp started as an intelligent terminal or console. It made the console feel more like a regular application. Copying and pasting and other interactions were straightforward. With the introduction of LLMs and later Agentic applications, Warp brings me the best way to start new projects, debug problems in running applications, and assist me with those command-line tasks that I forget how they work. A few days ago, I started paying for Warp to have more Agent interactions after using it to create complete Java project skeletons for a workshop we are making.

The best thing about Warp? You can try it absolutely for free. When using the agent part, you run into a limit fast. However, paying for a month to try it out is worth your money. Please use this link to download Warp. If enough people try it out, I get a free t-shirt :-)

Your invite to Warp-the agentic development platform *Warp is an AI agent platform that lets you run multiple agents in parallel to complete any development task.*app.warp.dev

The config for the Elastic Agent

#

An Elastic Agent configuration contains an input and an output component. Outputs for an Agent are limited to Elasticsearch, Kafka and Logstash. Sending the logs to Elasticsearch is the best choice. I can run Elasticsearch as a Docker container on my machine. The input is a file with sample logs. Each line contains the log level, like: DEBUG, INFO, WARN, ERROR.

The input element can contain processors. One processor is the drop_event processor. We can use this processor to drop all lines containing INFO or DEBUG. Below is the config for the agent.

inputs:
  - id: sample-logs
    type: filestream
    data_stream.type: logs
    data_stream.dataset: jettro.app
    data_stream.namespace: prod
    paths:
      - /Users/jettrocoenradie/Development/personal/elastic-agent-test/sample.log
    processors:
      - drop_event:
          when:
            or:
              - contains:
                  message: "INFO"
              - contains:
                  message: "DEBUG"

outputs:
  default:
    type: elasticsearch
    hosts: [ "http://localhost:9200" ]
    username: "elastic"
    password: "yourStrongPassword"

agent.logging.level: debug
agent.logging.to_files: false
agent.logging.to_stderr: true
agent.logging.metrics.enabled: false
agent.monitoring.enabled: false

Prompting your solution

#

I want to present here the prompt that generated the whole project. However, I had to learn more about Elastic Agents. I had to try out different paths and learn from my own mistakes. In the end, it did take me about 4 hours to make it work. For the following situation, an hour would be enough to do it again.

Some tips

#

Be clear about what you want. The more information you give about your requirements, the better Warp can work with it.

Do read all the output, yes, it takes longer, but you can learn from it.

You want the agent to do something, tell it what to do, do not ask questions.

If you want something big, you can try it all at once; sometimes the agent surprised me with the steps it takes. But to follow what it is doing, giving it subtasks can provide better results.

If something works, be sure to commit to git, which makes it easier to rollback. You can use other best practices, like working in branches. You can also tell the agent to do that.

Tell the agent to write the README.md file. The content helps you and others to understand the goal of the project, and in my case, learn about all the options that the shell scripts have.

The solution

#

The key features for the solution are generated as well. This is from the README file.

Key Features

#

• Automated Testing: Complete test suite that validates processor functionality
• Dynamic Test Data: Generates fresh, realistic log data for each test run
• Data Stream Safe: Properly handles Elasticsearch data streams and rollover scenarios
• Comprehensive Validation: Tests document counts, content filtering, and data structure
• Enhanced Diagnostics: Detailed troubleshooting when tests fail
• Easy Debugging: Helper scripts for manual inspection and troubleshooting
• Color-coded Output: Clear, readable test results with pass/fail indicators
• Production Ready: Safe cleanup operations that preserve data stream integrity

The script test-agent.sh performs all the steps, I’ll post a YouTube video at the end of the blog showcasing the output. Besides the whole script, you can also manually run the script and call the verify command to check the records that are ingested. Below is a sample from the documentation with the available commands.

# Document counts
./query-elastic.sh count     # Total documents
./query-elastic.sh infomsg   # INFO message count
./query-elastic.sh warn      # WARN message count
./query-elastic.sh error     # ERROR message count

# Data inspection
./query-elastic.sh sample    # Show recent documents
./query-elastic.sh all       # Complete summary + test result
./query-elastic.sh info      # Data stream structure

# Test data generation
./query-elastic.sh generate     # Generate 25 fresh log entries
./query-elastic.sh generate 50  # Generate 50 fresh log entries

# Maintenance
./query-elastic.sh delete    # Clean test data
./query-elastic.sh refresh   # Refresh data stream

I do not post all the generated code here—just a glimpse to understand how thorough the agent is in developing your solution. You still have to look at the code yourself. In my case, the agent made an error in cleaning up as we are using a data stream, not an index. I also had to tweak some other parts, but I did not change the code.

Concluding

#

I like this use case of testing components like Elastic Agents using these scripts. They are not reusable, but the process to generate the scripts is.