Creating a Beat based on Metricbeat resolving Init and create metricset failures

I suspect that you are reading this post after having trouble creating a new Elastic Beat based on Metricbeat. Bud do not worry! We have you covered and solved the issues for you!

Long story short, a few days ago we wanted to create a new Metricbeat. After reading and executing step by step the official guide “Creating a Beat based on Metricbeat” we stumpled upon a series of issues like Step 3 - Init and create the metricset failure. After spending 2 days trying to resolve our local dependencies and solve the issues in our development environment, we decided to make everything in a sterilized environment, hence a Docker container.

To begin with, we used Go language official Docker image with tag golang:1.10.2-stretch and we run everything in it. So in a shell, we executed:

docker run -it --rm --name go1.10 golang:1.10.2-stretch /bin/bash

When the container is started, a new shell is loaded and we can download all the dependencies needed, by executing inside it:

apt-get update \
 && apt-get install -y wget curl git vim \
 && echo "set number" | tee -a /etc/vim/vimrc \
 && echo "syntax on" | tee -a /etc/vim/vimrc \
 && echo "colorscheme evening" | tee -a /etc/vim/vimrc  \
 && wget https://bootstrap.pypa.io/get-pip.py \
 && python get-pip.py \
 && rm get-pip.py \
 && pip install virtualenv \
 && echo -n -e "\n\nOur environment:\n===================================\n" \
 && go version \
 && echo -n "Python version: " \
 && python --version \
 && echo -n "virtualenv version: " \
 && virtualenv --version \
 && echo "===================================" \
 && echo -n "Fetching metricbeat source code: " \
 && go get github.com/elastic/beats/metricbeat \
 && echo "[OK]" \
 && cd ${GOPATH}/src/github.com/elastic/beats \
 && echo "Checking out commit 2619e137c1f7eb031fc89132045117a85c7ef818 to new branch in order to avoid bug" \
 && git checkout -b working 2619e137c1f7eb031fc89132045117a85c7ef818 \
 && cd $GOPATH \
 && sed -i 's/@-cp -r module\/[*]\/_meta\/kibana _meta\//@-cp -r ${GOPATH}\/src\/github.com\/elastic\/beats\/metricbeat\/module\/*\/_meta\/kibana _meta\//' ${GOPATH}/src/github.com/elastic/beats/metricbeat/Makefile \
 && echo "Elastic beats environment has been initialised successfully! You can now generate a new Metricbeat!"

If everything works as expected you should see something like this in the output:

Our environment:
===================================
go version go1.10.2 linux/amd64
Python version: Python 2.7.13
virtualenv version: 16.0.0
===================================
Fetching metricbeat source code: [OK]
Checking out commit 2619e137c1f7eb031fc89132045117a85c7ef818 to new branch in order to avoid bug
Switched to a new branch 'working'
Elastic beats environment has been initialised successfully! You can now generate a new Metricbeat!

The environment is set up correctly. Let’s create a new Metricbeat. So execute:

python ${GOPATH}/src/github.com/elastic/beats/script/generate.py --type=metricbeat

and respond to the input as the example:

root@1e407b4d73a7:/go# python ${GOPATH}/src/github.com/elastic/beats/script/generate.py --type=metricbeat
Beat Name [Examplebeat]: weatherbeat
Your Github Name [your-github-name]: manios
Beat Path [github.com/manios/weatherbeat]:
Firstname Lastname: Christos Manios

The newly created Metricbeat will be stored in ${GOPATH}/src/github.com/manios/weatherbeat/.

Navigate to the Metricbeat directory. Execute the following to avoid weird errors when creating a metricset:

cd  ${GOPATH}/src/github.com/manios/weatherbeat \
&& cp -r $GOPATH/src/github.com/elastic/beats/metricbeat/scripts . 

Now we are ready to create a new Metricset. The metricset module will be called skgtemperatures and the metricset itself hydro. According to the official guide we execute make setup and the output will be the following:

mkdir -p vendor/github.com/elastic/
cp -R /go/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git
make create-metricset
make[1]: Entering directory '/go/src/github.com/manios/weatherbeat'
New python executable in /go/src/github.com/manios/weatherbeat/build/python-env/bin/python
Installing setuptools, pip, wheel...done.
docker-compose 1.22.0 has requirement requests!=2.11.0,!=2.12.2,!=2.18.0,<2.19,>=2.6.1, but you'll have requests 2.19.1 which is incompatible.
# Work around pip bug. See: https://github.com/pypa/pip/issues/4464
find /go/src/github.com/manios/weatherbeat/build/python-env -type d -name 'dist-packages' -exec sh -c "echo dist-packages > {}.pth" ';'
Module name: skgtemperature
Metricset name: hydro
Module skgtemperature created.
Metricset hydro created.
make[1]: Leaving directory '/go/src/github.com/manios/weatherbeat'
make collect
make[1]: Entering directory '/go/src/github.com/manios/weatherbeat'
# Work around pip bug. See: https://github.com/pypa/pip/issues/4464
find /go/src/github.com/manios/weatherbeat/build/python-env -type d -name 'dist-packages' -exec sh -c "echo dist-packages > {}.pth" ';'
make[1]: Leaving directory '/go/src/github.com/manios/weatherbeat'

If everything works as expected then if you execute a tree command, to module directory you will probably have the following structure:

-- skgtemperature
    |-- _meta
    |   |-- config.yml
    |   |-- docs.asciidoc
    |   `-- fields.yml
    |-- doc.go
    `-- hydro
        |-- _meta
        |   |-- data.json
        |   |-- docs.asciidoc
        |   `-- fields.yml
        `-- hydro.go

That’s it! I hope it helps!