Can not run elasticsearch as root sonarqube Error - SonarQube Elasticsearch do not run as root

When you tried to configure SonarQube, you may experience the below error:

2018.10.15 17:32:13 ERROR es[][o.e.b.Bootstrap] Exception
java.lang.RuntimeException: can not run elasticsearch as root
        

 ~[elasticsearch-5.6.3.jar:5.6.3]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) 

SonarQube does not support being run as root user, so we need to create regular user like sonar and give privilege to that use.

Fix for the above issue:
create a user called sonar by executing below command:
sudo useradd sonar

sudo chown -R sonar:sonar sonar-install-folder/

Edit the sonar.sh start script and change the #RUN_AS_USER to be RUN_AS_USER=sonar

sudo vi /opt/sonarqube/bin/linux-x86-64/sonar.sh

Change 
RUN_AS_USER=sonar

now start the server 

sudo sh /opt/sonarqube/bin/linux-x86-64/sonar.sh start

This should fix that issue.

Jenkins Scripted Pipeline - Create Jenkins Pipeline for Automating Builds, Code quality checks, Deployments to Tomcat - How to build, deploy WARs using Jenkins Pipeline - Build pipelines integrate with Bitbucket, Sonarqube, Slack, JaCoCo, Nexus, Tomcat

What are Pipelines in Jenkins?

- Pipelines are better than freestyle jobs, you can write a lot of complex tasks using pipelines when compared to Freestyle jobs.

- You can see how long each stage takes time to execute so you have more control compared to freestyle.

- Pipeline is groovy based script that have set of plug-ins integrated for automating the builds, deployment and test execution.

- Pipeline defines your entire build process, which typically includes stages for building an application, testing it and then delivering it. 

 - You can use snippet generator to generate pipeline code for the stages you don't know how to write groovy code.

- Pipelines are two types - Scripted pipeline and Declarative pipeline. Click here to know the difference between them.

Pre-requisites:

Install plug-ins
1. Install Pipeline stage view, Deploy to container, Slack, Jacoco, Nexus Artifact Uploader and SonarQube, pipeline stage view plug-ins (if already installed, you can skip it)

Steps to Create Scripted Pipeline in Jenkins

1. Login to Jenkins

2. Create a New item

3. Give name as MyfirstPipelineJob and choose pipeline

4. Click ok. Pipeline is created now

5. Under build triggers, click on poll SCM, schedule as

H/02 * * * *

6. Go to Pipeline definition section, click on Pipeline syntax link. under sample step drop down, choose checkout: Checkout from version control. enter bitbucket or GitHub Repository URL, and enter right credentials. Click here to learn to use PAT if you are using GitHub. scroll down, click on Generate Pipeline script. Copy the code.

7. Now copy the below pipeline code highlighted section into Pipeline section in the pipeline. Please copy stage by stage

8. Change Maven3, SonarQube variables and also Slack channel name as highlighted above in red as per your settings.

9. For Nexus Upload stage, You need to change the Nexus URL and credentials ID for Nexus (which you can grab from Credentials tab after login)

10. For Dev Deploy stage, you can copy credentials ID used for connecting to Tomcat.

Pipeline Code:

node {

    def mvnHome = tool 'Maven3'
    stage ("checkout")  {
       copy code here which you generated from step #6
    }

   stage ('build')  {
    sh "${mvnHome}/bin/mvn clean install -f MyWebApp/pom.xml"
    }

     stage ('Code Quality scan')  {
       withSonarQubeEnv('SonarQube') {
       sh "${mvnHome}/bin/mvn -f MyWebApp/pom.xml sonar:sonar"
        }
   }
  
   stage ('Code coverage')  {
       jacoco()
   }

   stage ('Nexus upload')  {
        nexusArtifactUploader(
        nexusVersion: 'nexus3',
        protocol: 'http',
        nexusUrl: 'nexus_url:8081',
        groupId: 'myGroupId',
        version: '1.0-SNAPSHOT',
        repository: 'maven-snapshots',
        credentialsId: '2c293828-9509-49b4-a6e7-77f3ceae7b39',
        artifacts: [
            [artifactId: 'MyWebApp',
             classifier: '',
             file: 'MyWebApp/target/MyWebApp.war',
             type: 'war']
        ]
     )
    }
   
   stage ('DEV Deploy')  {
      echo "deploying to DEV Env "
      deploy adapters: [tomcat9(credentialsId: '4c55fae1-a02d-4b82-ba34-d262176eeb46', path: '', url: 'http://your_tomcat_url:8080')], contextPath: null, war: '**/*.war'

    }

  stage ('Slack notification')  {
    slackSend(channel:'channel-name', message: "Job is successful, here is the info -  Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})")
   }

   stage ('DEV Approve')  {
            echo "Taking approval from DEV Manager for QA Deployment"     
            timeout(time: 7, unit: 'DAYS') {
            input message: 'Do you approve QA Deployment?', submitter: 'admin'
            }
     }

stage ('QA Deploy')  {
     echo "deploying into QA Env " 
deploy adapters: [tomcat9(credentialsId: '4c55fae1-a02d-4b82-ba34-d262176eeb46', path: '', url: 'http://your_tomcat_url:8080')], contextPath: null, war: '**/*.war'

}

  stage ('QA notify')  {
    slackSend(channel:'channel-name', message: "QA Deployment was successful, here is the info -  Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})")
   }

stage ('QA Approve')  {
    echo "Taking approval from QA manager"
    timeout(time: 7, unit: 'DAYS') {
        input message: 'Do you want to proceed to PROD Deploy?', submitter: 'admin,manager_userid'
  }

}

stage ('PROD Deploy')  {
     echo "deploying into PROD Env " 
deploy adapters: [tomcat9(credentialsId: '4c55fae1-a02d-4b82-ba34-d262176eeb46', path: '', url: 'http://your_tomcat_url:8080')], contextPath: null, war: '**/*.war'

}
}

11. Click Apply, Save

12. Now click on Build. It should execute all the stages and show pipeline view like this.

You can watch the Scripted pipeline video in my YouTube channel:

Pipeline code for uploading build artifacts to Nexus from Jenkins - How to upload build artifacts to Nexus from Jenkins

Here below is the pipeline code for uploading build artifacts to Nexus from Jenkins:

Prerequisite:
1) Install Nexus Artifact Uploader plug-in.

2) You also need to add Nexus credentials and use the credentials ID from Jenkins

stage ('Nexus Artifact upload')
    {
        nexusArtifactUploader(
        nexusVersion: 'nexus3',
        protocol: 'http',
        nexusUrl: 'ec2-18-223-182-14.us-east-2.compute.amazonaws.com:8081',
        groupId: 'myGroupId',
        version: '1.0-SNAPSHOT',
        repository: 'maven-snapshots',
        credentialsId: '2c293828-9509-49b4-a6e7-77f3ceae7b39',
        artifacts: [
            [artifactId: 'MyWebApp',
             classifier: '',
             file: 'MyWebApp/target/MyWebApp.war',
             type: 'war']
        ]
     )
    }

How to create S3 bucket in AWS using Terraform - Create S3 bucket in AWS using Terraform

Terraform is an infrastructure orchestration tool for creating web services in AWS automatically. You can use Terraform for provisioning S3 bucket in AWS.

sudo vi create_s3.tf

resource "aws_s3_bucket" "mybucket" {
  bucket = "my-tf-test-bucket"
  acl    = "public-read"
  website {
    index_document = "hello.html"
   routing_rules = <<EOF
[{
    "Condition": {
        "KeyPrefixEquals": "docs/"
    },
    "Redirect": {
        "ReplaceKeyPrefixWith": "documents/"
    }
}]
EOF
  }

  tags= {
    Name        = "My bucket"
    Environment = "Dev"
  }
}

Once you create the above file, execute terraform plan and then terraform apply to create S3 bucket in AWS.

Ansible Playbook for provisioning a new EC2 instance in AWS - Create a new EC2 Using Ansible Playbook

Please find the Ansible Playbook for provisioning a new EC2 instance. Please follow the below steps in the machine where you installed Ansible.

Steps to create EC2 instance using Ansible:

1. Login to AWS console, click on username and go to My security credentials.
2. Continue on security credentials, click on access keys
3. Create a new access key if you dont have one. Make sure you download the keys.
4. Login to EC2 instance using Git bash or ITerm where you installed Ansible.

execute the below command

sudo vi ~/.boto

add below three lines in the above file, replace the ?? with access key and secret key values.

[Credentials]

aws_access_key_id = ??

aws_secret_access_key = ??

5. Edit Ansible hosts or inventory file
sudo vi /etc/ansible/hosts 
Add the below two lines in the end of the file:
[localhost]
local

6. cd ~
7. mkdir playbooks  

8. cd playbooks

Create Ansible playbook

9. sudo vi create_jenkins_ec2.yml 
(copy the below content in green color)

edit the create_jenkins_ec2.yml to make sure you update the key which is red marked below:

---

 - name:  provisioning EC2 Lab Exercises using Ansible

   hosts: localhost

   connection: local

   gather_facts: False

   tags: provisioning

   vars:

     keypair: MyEC2Key

     instance_type: t2.small

     image: ami-07c1207a9d40bc3bd

     wait: yes

     group: webserver

     count: 1

     region: us-east-2

     security_group: my-jenkins-security-grp

   

   tasks:

     - name: Create my security group

       local_action: 

         module: ec2_group

         name: "{{ security_group }}"

         description: Security Group for webserver Servers

         region: "{{ region }}"

         rules:

            - proto: tcp

              from_port: 22

              to_port: 22

              cidr_ip: 0.0.0.0/0

            - proto: tcp

              from_port: 8080

              to_port: 8080

              cidr_ip: 0.0.0.0/0

            - proto: tcp

              from_port: 80

              to_port: 80

              cidr_ip: 0.0.0.0/0

         rules_egress:

            - proto: all

              cidr_ip: 0.0.0.0/0

       register: basic_firewall

     - name: Launch the new EC2 Instance

       local_action:  ec2 

                      group={{ security_group }} 

                      instance_type={{ instance_type}} 

                      image={{ image }} 

                      wait=true 

                      region={{ region }} 

                      keypair={{ keypair }}

                      count={{count}}

       register: ec2

     - name: Add Tagging to EC2 instance

       local_action: ec2_tag resource={{ item.id }} region={{ region }} state=present

       with_items: "{{ ec2.instances }}"

       args:

         tags:

           Name: MyTargetEc2Instance

10. now execute the ansible playbook by
sudo ansible-playbook create_jenkins_ec2.yml

Fix the warnings by executing below command

pip install --upgrade requests==2.20.1

If everything is good, you should see the new instance created on AWS console. make sure you are able to connect to that instance.

That's it!! That is how you create a new EC2 instance using Ansible.

How to push a code change into Bitbucket remote repository? - Code push into Bitbucket using Git bash

Let us see how to push a code change using Git bash or iTerm on your local machine into bitbucket.

After configuring a freestyle job or pipeline using Jenkins for building and deployment, you would like to make a code change to make sure Jenkins have started automated builds/deployments.

Pre-requistes:

SSH keys set up and uploaded into Bitbucket.

Implementation steps:

1. Go to bitbucket, select repository which you already have setup MyWebApp.
If you already cloned, move to step # 6.

2. Click on clone

3. copy SSH clone url

4. go to git bash window

5. paste the url and enter

git clone git@bitbucket.org:username/repo_name.git

6. cd repo_name

change repo_name per your name of the repo you created in Bitbucket

7. cd MyWebApp/src/main/webapp

8. sudo vi index.jsp

Change Hello world into below yellow highlighted:

<html>

<body>

<h2>Howdy Folks !!! Welcome to DevOps!</h2>

</body>

</html>

9. press escape, enter :wq!

10. git add index.jsp

11. git commit -m "made change to jsp"

12. git push

Now refresh the browser and click on Source to see code changes you made in your git bash window.

Create SonarQube instance using Terraform - Setup SonarQube instance using Terraform

Execute the below command after login to EC2 where you installed Terraform:

navigate to project-terraform folder where you have created already tf files.

cd ~/project-terraform

if you are using Apple laptop or EC2 instance, execute below command:

sudo vi sonar.tf 

for Windows laptop use below command:

notepad sonar.tf

Copy the below content with green background:

Change the key name marked red below per your key name:

resource "aws_vpc" "sonar" {

  cidr_block = "172.16.0.0/16"

  instance_tenancy = "default"

  tags = {

    Name = "sonar_vpc"

  }

}

 resource "aws_security_group" "security_sonar_group_2023" {
      name        = "security_sonar_group_2023"
      description = "security group for Sonar"

      ingress {
        from_port   = 9000
        to_port     = 9000
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
      }

     ingress {
        from_port   = 22
        to_port     = 22
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
      }

     # outbound from Sonar server
      egress {
        from_port   = 0
        to_port     = 65535
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
      }

      tags= {
        Name = "security_sonar"
      }
    }

    resource "aws_instance" "mySonarInstance" {
      ami           = "ami-0b9064170e32bde34"
      key_name = "your_aws_ssh_key"
      instance_type = "t2.micro"
      vpc_security_group_ids = [aws_security_group.security_sonar_group_2023.id]

      tags= {
        Name = "sonar_instance"
      }
    }

# Create Elastic IP address for Sonar instance

resource "aws_eip" "mySonarInstance" {

  vpc      = true

  instance = aws_instance.mySonarInstance.id

tags= {

    Name = "sonar_elastic_ip"

  }

}

Execute below command:
    terraform plan
and then
    terraform apply

Now you will see a new EC2 instance being created in AWS console.

How to enable code coverage report using JaCoCo plug-in - Code coverage Report using JaCoCo, Maven and Jenkins - Code Coverage in Pipeline

Code coverage is important aspect for maintaining quality in Agile development. There are different ways to manage code quality. one of the effective ways is to measure code coverage by using plug-ins such as JaCoCo, Cobertura.

We will see how to enable code coverage for your Java project and view coverage report in Jenkins UI.

step # 1: Add Maven JaCoCo plugin in POM.xml under MyWebApp in bitbucket Repo

<build>
<finalName>MyWebApp</finalName>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.7.201606060606</version>
<executions>
<execution>
<id>jacoco-initialize</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>jacoco-report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

Step 2 : Add JaCoCo plug-in in Jenkins:

Step 3:

For Freestyle Job:
Enable in Jenkins job to view code coverage report by going to post build action and add Record JaCoCo coverage report

For Pipelines:
Add below code in your existing pipeline:

stage ('Code coverage') {
jacoco()
}

Step 4 : Run the job by clicking Build now

Step 5:
 Click on the job to view code coverage report.

How to enable Multi Factor Authentication (MFA) for your AWS account? - Enable MFA to AWS account - Secure AWS Account

It is very important to secure your AWS account. Especially for those who is creating new AWS account, as hackers are potentially targeting new accounts.

Let us see the how to secure AWS accounts by enabling multi factor authentication?

1. Login to AWS.
2. Click on your user name. Click on My Security Credentials.

3. Continue to security credentials.

4. Click on Multi-factor authentication, click on Activate MFA

5. Now choose A Virtual MFA device

6.  Click on Next step

7. Now download google authenticator or Microsoft authenticator on your smart phone
8. After downloaded, click on + scan bar code. Scan the above bar code.
you need to enter first number and wait until it expires, enter second number in code 2 and then click on Activate Virtual MFA.

9. Once entered both numbers, click on Activate Virtual MFA.

This is how you secure AWS account by enabling MFA.

How to integrate SonarQube in Azure DevOps - Setup SonarQube code analysis in VSTS or Azure DevOps

Please find steps below for integrating SonarQube with Azure DevOps or Visual Studio Team Services:


Pre-requisites:
1. Make sure you install SonarQube plug-in/Add-on in VSTS (Azure DevOps) using below URL:
https://marketplace.visualstudio.com/acquisition?itemName=SonarSource.sonarqube

Once added SonarQube plug-in, click on proceed to Organization..

Steps:
1. Login to Azure DevOps. Go to Azure Pipelines. Edit your pipeline

2. Click on Add tasks

3. Enter Sonar

4. Add Prepare Analysis configuration
5. move up this task.
It should be like shown below:

6. Edit prepare sonar analysis configuration task

7. Click on New service connection

 8. Enter name as mySonarServer and put in SonarQube url including port number and use the token generated

9. Choose that name and select Integrate with Maven or Gradle

10. Edit maven task and add clean install sonar:sonar

11. Run the build, it should integrate with Sonar and should do code analysis.

How to create Jenkinsfile and configure Pipeline as Code - Jenkins Pipeline As code

Please find steps below for configuring your existing pipeline as a code (Jenkinsfile).

Pre-requistes:

1. Project setup in Bitbucket or GitHub
2. Jenkins and Tomcat (web container) set up.
3. Maven installed as well
4. Sonarqube setup and integrated with Jenkins

Add Jenkinsfile (pipeline code) to MyWebApp in BitBucket.

Step 1.

Go to Bitbucket and choose the Repo where you setup MyWebApp.

Step 2

Click on three dots ... and Add File

Step 3
Enter Jenkinsfile as a file name

 

Step 4

Copy and paste the below code

node {

def mvnHome = tool 'Maven3'

stage ('Checkout') {

checkout scm
}

stage ('Build') {

sh "${mvnHome}/bin/mvn clean install -f MyWebApp/pom.xml"

}

stage ('Code quality scan') {

withSonarQubeEnv('SonarQube') {

sh "${mvnHome}/bin/mvn sonar:sonar -f MyWebApp/pom.xml"

   }

}

    stage ('Nexus upload')
    {
        nexusArtifactUploader(
        nexusVersion: 'nexus3',
        protocol: 'http',
        nexusUrl: 'ec2-18-223-182-14.us-east-2.compute.amazonaws.com:8081',
        groupId: 'myGroupId',
        version: '1.0-SNAPSHOT',
        repository: 'maven-snapshots',
        credentialsId: '2c293828-9509-49b4-a6e7-77f3ceae7b39',
        artifacts: [
            [artifactId: 'MyWebApp',
             classifier: '',
             file: 'MyWebApp/target/MyWebApp.war',
             type: 'war']
        ]
     )
    }

stage ('DEV Deploy') {

echo "deploying to DEV tomcat "

sh 'sudo cp /var/lib/jenkins/workspace/$JOB_NAME/MyWebApp/target/MyWebApp.war /var/lib/tomcat8/webapps'

}

stage ('DEV Approve') {

echo "Taking approval from DEV"

timeout(time: 7, unit: 'DAYS') {

input message: 'Do you want to deploy?', submitter: 'admin'

     }

 }

stage ('Slack notification') {

slackSend(channel:'channel-name', message: "Job is successful, here is the info - Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})")

  }

}



Step 5
Click on Commit and enter comments

Click on Commit

That's it. Pipeline as a code - Jenkinsfile is setup in BitBucket.

Create Pipeline and Run pipeline from Jenkinsfile

1. Login to Jenkins
2. Click on New item, give some name and choose Pipeline and say OK

3. Under build triggers, choose Poll SCM,
Enter H/02 * * * *

4. Under Pipeline section. click on choose pipeline script from SCM

5. Under SCM, choose Git

6. Enter SSH or HTTPS URL of repo and choose credentials - SSH private key if you are using SSH url. or user/password of GitHub if you are using HTTPS url.

Script path as Jenkinsfile

7. Click on Apply and Save
8. Click on Build now.
You should see pipeline running and application is deployed to Tomcat.