Monday, August 13, 2018

DevOps FAQs page | DevOps Wiki Page | DevOps Troubleshooting Page

Welcome to DevOps FAQs page. The objective of this page is to help you provides fixes for the issues you are facing during building pipelines.

Issue # 1 - Not able to connect to EC2 instance from my local machine and have No such file or directory error.
Root cause and fix:
1. Make sure you have downloaded the above keys under downloads directory.
2. Once you verified, you need to ssh from downloads directory on your local machine. make sure you type below command:

cd downloads (press enter)
and then copy ssh url copied from AWS console (and press enter)


Issue # 2 -  Unable to connect to Bitbucket in Jenkins


Root cause and fix:

There are two issues in the above picture.
1. you need to select add button and add your Bitbucket credentials and select it from drop down after adding it.
2. Do not add Jenkins credentials, it should be Bitbucket credentials.

Issues # 3 - There is no pom in this directory



Root cause and fix

Path of pom.xml was not specified in the build job. you need to click on advanced and add the path of pom.xml. in our case it will be MyWebApp/pom.xml



Issue - 4 Tomcat Deployment failed or not able to access WebApp after deployment


root cause and fix: 

Tomcat deployment steps are not configured correctly in Jenkins job. Click on post build action, add deploy to container step and enter tomcat url and credentials per below screenshot:



Also make sure, tomcat credentials are entered correctly per below screenshot in tomcat-users.xml
Execute the below command
sudo vi /var/lib/tomcat9/conf/tomcat-users.xml 

make sure your file looks like below file:



make sure you restart tomcat as well:
sudo systemctl restart tomcat9



Issue # 5 - SonarQube server can not be reached Error



Root cause and Fix:
When ever you stop EC2 instance and re-start, public DNS name always changes. So you will have to change it every time you restart.
You need to re-enter new sonarqube server URL under Jenkins --> Manage jenkins --> Configure system. scroll down and under sonarQube configuration 


enter SonarQube URL. no need to create token again. old token is enough.


Issue # 6  - Not able to publish into GitHub or Bitbucket using git push


root cause and fix:


This could be due to SSH keys are not set up correctly from the machine you are trying to push the code.


Issue # 7 - Unable to push into Bitbucket or GitHub, getting rejected error




root cause and fix:

When ever you do git push, if changes are not pushed and getting above messages such as rejected. 
 
Solution:
You need to perform git pull first by syncing your local repository with remote repo changes.


Issue # 8 - Unable to create EC2 instance
 




root cause and fix:


Re-name the security group to something else, or use the same group instead of creating another one.


Issue # 9 - Not authorized. Please check the properties sonar.login and sonar.password


root cause and fix:
You will run into this issue if you have setup Sonar Token in Jenkins.
You need to login in SonarQube as an administrator and create the token. Copy the token in Jenkins --> Manage Jenkins--> sonarqube section.




Issue # 10 - Puppet Login credentials error.


root cause and fix:
Make sure you have your access keys and secret keys in the below file and also no space before [default]

sudo vi ~/.aws/credentials
[default]

aws_access_key_id = ?
aws_secret_access_key = ?


Issue # 11 - Unable to upload WAR file into Nexus,  WAR file is not uploading into Nexus





Root cause and fix:
You need to choose as Nexus 3 from drop down instead of Nexus2




Issue # 12 - Unable to access Jenkins or Tomcat from the browser






Root cause and Fix:


You need to enter public dns name from step # 4 in below screen. 
Go to AWS console, select EC2 instance, click to connect and copy public dns name value from step #4.



Issue # 13 - Tomcat Deployment failed in Jenkins - Invalid username and password


Root cause and Fix:
It could be due to invalid tomcat user name and password. You need enter right username and password and also select it from drop down.

Also make sure, tomcat credentials are entered correctly per below screenshot in tomcat-users.xml
Execute the below command 

sudo vi /var/lib/tomcat10/conf/tomcat-users.xml 
Make sure you have the below entries properly per screenshot:
<role rolename="manager-script"/>

<user username="tomcat" password="password" roles="manager-script"/>



Make sure you restart tomcat  once you make the above changes:
sudo systemctl restart tomcat10


Issue # 14 - Unable to upload WAR file into Nexus,  Not Authorized error




Root cause and Fix:
Make sure you are using right credentials for Nexus which is admin/admin123, and also you need to select from drop down

Issue # 15 - Invalid Credentials ID when applying TerraForm

Root cause and Fix:
Make sure your AWS access keys and secret keys are correct in your aws.tf file.

Issue # 16 - SonarQube does not match any configured installation



Root cause and fix:

Go to Jenkins ->Manage Jenkins- > Configure system, sonarqube server installation, make sure the variable name correct. It should match with below name:

stage ('Code quality scan') {

withSonarQubeEnv('SonarQube') {

sh "${mvnHome}/bin/mvn sonar:sonar -f MyWebApp/pom.xml"

   }


Issue # 17 - SonarQube is stopped Error 



Root cause:

Your sonar.properties may not be setup correctly. It could be due to three different reasons:

1. verify below sonar.proeprties
sudo vi /opt/sonarqube/conf/sonar.properties
uncomment the below lines by removing # and add values highlighted yellow
sonar.jdbc.username=sonar
sonar.jdbc.password=password







Next, Add the below line
sonar.jdbc.url=jdbc:postgresql://localhost/sonarqube


2. Switch to sql shell by entering
psql









Execute the below lines (one by one)


ALTER USER sonar WITH ENCRYPTED password 'password';

CREATE DATABASE sonarqube OWNER sonar; 
 
GRANT ALL PRIVILEGES ON DATABASE sonarqube to sonar;
 
\q





type exit tom come out of postgresSQL.

3. Make sure you have setup sonarqube as a service properly.

Execute the below command:
sudo vi /etc/systemd/system/sonar.service

# copy the below green highlighted commands in the above file
[Unit]
Description=SonarQube service
After=syslog.target network.target

[Service]
Type=forking

ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonarqube/bin/li
nux-x86-64/sonar.sh stop
LimitNOFILE=131072
LimitNPROC=8192
User=sonar
Group=sonarGroup
Restart=always
 
[Install]
WantedBy=multi-user.target


Issue # 18 - Unable to do git clone - make sure you have correct access rights error


root cause and fix

When ever you have this error, it means you have not uploaded your SSH keys correctly into bitbucket or github. Make sure you generate ssh keys by executing the below command:
ssh-keygen 
(enter four times)
Copy the ssh-keys like below and upload into Bitbucket or GitHub.
cat ~/.ssh/id_rsa.pub

Issue # 19 - Non-Readable POM - No such file or directory
When you try to build in Jenkins, you may get this issue.




Root cause and fix
Java project may not be setup in BitBucket properly. You will have to setup MyWebApp in Bitbucket on the repo you are using in Bitbucket.

Re-visit lab # 2 on how to set up java project or use the repo that has already MyWebApp setup correctly. 

Issue # 20 - sudo: no tty present and no askpass program specified Error

Root cause and fix:
You need to give permission for Jenkins to do deployment.
Login to instance where you installed Jenkins, type the below command

sudo visudo


look for root and add the changes highlighted in yellow at the below line of root:

# User privilege specification
root ALL=(ALL:ALL) ALL
jenkins ALL=(ALL) NOPASSWD: ALL
After making the changes, Ctrl O, press enter to save. Ctrl X to exit the file.

Now with the above changes, you are able to execute the build jobs which require sudo permissions.

Issue # 21 -  Unable to connect to GitHub in Jenkins


Root cause and fix:

There are two issues in the above picture.
1. you need to select add button and add your GitHub credentials and select it from drop down after adding it.
2. Do not add Jenkins credentials, it should be GitHub credentials.

Issue # 22 -  Maven Path issue 


Root cause and fix:
1.Remove space before or after /usr/share/maven

Issue # 23 -  Puppet Error: Could not run: Could not find type ec2_instance



Root cause and fix:
You need to install puppet-labs-aws module
sudo /opt/puppetlabs/bin/puppet module install puppetlabs-aws
 
Issue # 24 -  SonarQube Quality gate issue Error

hudson.remoting.ProxyException: net.sf.json.JSONException: Invalid JSON Stringat org.sonarsource.scanner.jenkins.pipeline.WaitForQualityGateStep$Execution.checkTaskCompleted(WaitForQualityGateStep.java:234)

Root cause and fix:
Make sure you fix SonarQube url by removing / in the end under Manage Jenkins->configure system 
you need to remove / in the end of URL. there should not be any / after 9000, it should be something like below:

right url - http://sonarqube_url:9000
Wrong url - http://sonarqube_url:9000/

Issue # 25 -  Error can not perform an interactive login from a non TTY device


Root cause and fix:
remove existing access + secret keys if you have attached in EC2 instance by performing below command:
sudo rm ~/.aws/config
sudo rm ~/.aws/credentials

Then you need to create an IAM role with AmazonEC2ContainerRegistryFullAccess policy and attached the IAM role to Ec2 instance.

Issue # 26 -  No valid Credentials for Terraform AWS provided found

Root cause and fix:
make sure you have AWS credentials setup using aws configure or add Create IAM role with right policy and attach that IAM role to ec2 instance.

Issue # 27 - stderr: remote: Support for password authentication was removed on August 13, 2021. Please use a personal access token instead 

Root cause and fix:
How to Fix for the above error by creating Personal Access Token:GitHub removed password authentication support from August 13, 2021 instead it recommends to use either OAuth or Personal Access Token.
Click here to learn how to create Personal Access Token.

Issue # 28 - Certificate verification failed: The certificate is NOT trusted | Jenkins installation Error Fix
If you try to install Jenkins on Ubuntu 18.0.4/20.0.4 instance, you may experience this problem.

Running sudo apt-get update on my AWS EC2 Ubuntu 18.04.01 LTS instance fails because Certificate verification failed: The certificate is NOT trusted.

Err:6 https://pkg.jenkins.io/debian-stable binary/ Release

Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 199.232.66.133 443]

Fix for the above error

You need to install certificates to overcome this error:

sudo apt install ca-certificates

sudo apt-get update

Now try installing Jenkins

sudo apt install jenkins -y

Issue # 29 - Error from server (BadRequest): container "mypythondockerrepo" in pod "my-python-deployment-745b64d6c6-dgwn5" is waiting to start: image can't be pulled

Root cause and fix:

For Deploying Docker images from ACR into AKS Cluster

The following command allows you to authorize an existing ACR in your subscription and configures the appropriate ACRPull role for the managed identity.

az aks update -n myAKSCluster -g myResourceGroup --attach-acr myacrrepo31

Issue # 30 - Errors running EKS Cluster 


Root cause and fix:
remove existing access + secret keys if you have attached in EC2 instance by performing below command:
sudo rm ~/.aws/config
sudo rm ~/.aws/credentials

and then setup access keys

Issue # 31 - Errors acquiring the state lock 


Root cause and fix:

Make sure you add AmazonDynamoDBFullAccess policy to the role and attach EC2 instance.

Issue # 32 - Error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found


Root cause and fix:
your EC2 instance does not have enough privilege to perform terraform apply. Make sure you add IAM role to EC2 instance or setup access keys + secret keys 

Issue # 33 -  no hosted parallelism has been purchased or granted in Azure Devops


Root cause and fix:
Microsoft has temporarily disabled the free grant of parallel jobs for public projects and for certain private projects in new organizations. However, you can request this grant by submitting a request. Submit a ticket using below url to request increased parallelism in Azure DevOps.  Please note that it takes us 2-3 business days to respond to your free tier requests.


Issue # 34 - Kubernetes Deployment Error using helm chart

Error: kubernetes cluster unreachable: exec plugin: invalid apiversion "client.authentication.k8s.io/v1alpha1"


Root cause and fix:
Downgrading helm version to 3.8.2 would resolve the issue.

curl -L https://git.io/get_helm.sh | bash -s -- --version v3.8.2


Issue # 35 - SonarQube Admin Password Reset or Forgotten SonarQube Admin password
Let's say you have setup SonarQube using Docker or Docker Compose, you have forgotten the admin password for SonarQube. This article helps you to reset/recover admin password. If you changed and then lost the admin password, you can reset it using the following steps.
Fix:
As we have configured SonarQube using Docker compose, We need to login to PostgreSQL running inside postgres docker container and execute update command to reset to default password. check below link:

https://www.coachdevops.com/2022/10/how-to-recover-admin-password-for.html

Issue # 36 - Classic Pipeline Editor is Disabled in Azure DevOps, how to Enable it?

If classic editor option is disabled in Azure DevOps, you may see something like below in your Azure DevOps project. We can enable it by changing the settings at project or organizational level.

Fix:


Organization Settings--> Select Pipelines --> Settings

Make sure 'Disable creation of classic build pipelines' and 'Disable creation of classic release pipelines' is turned off, to have classic editor shown after creating a project.


Finally disable the options highlighted in the image below