Troubleshooting Haaukins
Solutions to common problems.
In this guideline, the way of handling troubles and possible reasons of troubles will be explained with some solutions.
The issues that you may face during active usage of Haaukins
- No space left on device
- VM Import failed
- Config file not found
- Guacamole 500 Error
- Pool overlaps with other one on this address space
- Unable to create database client
- Certificate Issue
Issues which can be seeing in setting up development environment
No space left on device
This error could be caused due to various reasons which includes redundant inodes, bad blocks in your volume or an application which fills your /
root directory.
However, if you are facing this error after active usage of Haaukins platform you may need to configure your docker volumes path, the content of docker volumes should not be written to /
root path.
Check this guide for solving No space left on device
error and see whether it is caused due to docker or not.
There is another reason that you may face with this error when you are using Haaukins. In the project, we are using ioutil.Tempfile
which is under the hood connected to os.Tempfile
and if you do not specify first parameter for that function,
it will use the value of os.TempDir
as first parameter and it will check and return following values.
Definition of how os.TempDir` is finding out which directory to use for temporary files.
TempDir returns the default directory to use for temporary files. On Unix systems, it returns $TMPDIR if non-empty, else /tmp. On Windows, it uses GetTempPath, returning the first non-empty value from %TMP%, %TEMP%, %USERPROFILE%, or the Windows directory. On Plan 9, it returns /tmp. The directory is neither guaranteed to exist nor have accessible permissions.
If your $TMPDIR
not set and you are using linux, then Haaukins will use /tmp
directory for writing any temporary files.
You may need to set it to a place which is generally used for keeping data, e.g /data/tmp
, rather than /tmp
under root path.
If any of them did not work for you create issue
Alternatively,symbolic links could be very useful for using a directory which is not under root
You can bound a symbolic link to /tmp
as follows;
ln -s /data/tmp /tmp
Creating symbolic link means that the data will actually reside in /data/tmp
however, the data can be accesible and usable from /tmp
.
VM Import failed
This issue can be caused for different reasons, it could be not enough space on the folder where VM will be important or bad VM file (-bad ova file-), in this case some operations should be done in order to fix it.
Import VM failed due to no space
In our case, we have faced with import error which means that when we try to import VM, it failed due to not having space on the server. In this case, an inspection made the reason of failing very clear.
All data regarding to VM and other applications stayed on root
path which is not a common case for managing servers. root
path should contain essential programs and installations where operating system is requiring or an application which should be under a user' home folder. Therefore, root
path should be as clean as possible, all related data to an application should be stayed under data
path/folder.
Example output of this error:
10:02AM DBG getting path lock first_time=true path=/<user-daemon-path>/frontends/kali.ova
10:03AM ERR Error while creating new lab VBoxError [import /<user-daemon-path>/frontends/kali.ova --vsys 0 --vmname kali{ecc3ea71}]: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interpreting /<user-daemon-path>/frontends/kali.ova...
OK.
0%...10%...20%...30%...
Progress state: VBOX_E_FILE_ERROR
VBoxManage: error: Appliance import failed
VBoxManage: error: Could not create the imported medium '/<path-to-vms>/VirtualBox VMs/kali/kali-disk001_2.vmdk'.
VBoxManage: error: VMDK: cannot write allocated data block in '/<path-to-vms>/VirtualBox VMs/kali/kali-disk001_2.vmdk' (VERR_DISK_FULL)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component ApplianceWrap, interface IAppliance
VBoxManage: error: Context: "RTEXITCODE handleImportAppliance(HandlerArg*)" at line 886 of file VBoxManageAppliance.cpp
Disks:
vmdisk1 42949672960 -1 http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized kali-disk001.vmdk -1 -1
Virtual system 0:
0: Suggested OS type: "Debian_64"
(change with "--vsys 0 --ostype <type>"; use "list ostypes" to list all possible values)
1: VM name specified with --vmname: "kali{ecc3ea71}"
2: Number of CPUs: 2
(change with "--vsys 0 --cpus <n>")
3: Guest memory: 1024 MB
(change with "--vsys 0 --memory <MB>")
4: Sound card (appliance expects "", can change on import)
(disable with "--vsys 0 --unit 4 --ignore")
5: Network adapter: orig NAT, config 3, extra slot=0;type=NAT
6: IDE controller, type PIIX4
(disable with "--vsys 0 --unit 6 --ignore")
7: IDE controller, type PIIX4
(disable with "--vsys 0 --unit 7 --ignore")
8: Hard disk image: source image=kali-disk001.vmdk, target path=/<path-to-vms>/VirtualBox VMs/kali/kali-disk001_2.vmdk, controller=6;channel=0
(change target path with "--vsys 0 --unit 8 --disk path";
disable with "--vsys 0 --unit 8 --ignore")
In order to overcome such a situation, it is always nice to have exclusive data folder for each users on the server where they reside their data regarding to their applications/research.
Create data folder for each user on the system
In general, servers have data path which is much higher than normal root
path. In order to make root
path as clean as possible, all data under /home/${USER}
should be migrated into data
path
Changing default home folder can help to overcome the problem.
$ sudo su
$ mkdir /data/${USER}
$ chown -R ${USER}:${USER} /data/${USER}
$ usermod -d /data/${USER} ${USER} # will change default home dir to /data/${USER}
$ mv /home/${USER}/* /data/${USER}
Following bash script could be used automated way of handling the operation.
#!/bin/bash
# do not include the user who logged in to server
# make sure that user has admin privileges
declare -a users=("user1" "user2" "user3")
for user in "${users[@]}"
do
mkdir /data/$user
chown -R $user:$user /data/$user
# or do whatever with individual element of the array
usermod -d /data/$user $user
mv /home/$user/* /data/$user
done
Do that operations for all users (-except the one who logged in to server-) who consumes a lot of places in terms of data.
Afterwards, there will be no problems regarding to spaces until, /data
path is full.
Import failed due to bad ova file
It is very clear that the problem is directly related to corrupted ova file, in those cases update ova file with non-corrupted file.
Import failed due to no permission
In some cases, permission is denied for importing VMs, in order to overcome, change the permission of the folder where VMs are generated with correct permissions.
Permissions are changed through chmod
command.
Examples:
$ chmod +rw /vms/
It will add to the permission of /vms
write and read permissions.
Config file not found
Basically it clarifies config file is missing, when you are running daemon of Haaukins or from source code, either you need to clarify config file by adding --config
flag at the end of file.
Like;
go run main.go --config=/<absolute-path-to-config>/config.yml
]
For demonized version of Haaukins, you can provide config path after binary such as ;
<path-to-binary>/hknd --config=/<absolute-path-to-config/config.yml>
Keep in mind that Haaukins is looking for config.yml file on the same directory with binary, which means that if config.yml file on the same directory with Haaukins binary, there is no need to provide absolute path of configuration file.
Guacamole 500 Error
This error might happen for some reasons which are;
- VRDE feature is NOT enabled on virtualbox. It needs to be installed by following
$ VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-5.2.44.vbox-extpack
The version of extension version differs according to version of virtual box that you are having on the server.
VM might NOT be active (Not running state, check it)
Be careful about updating and downgrading the kernel version, it may cause serious headaches Make sure that Docker and Vboxmanage have been installed correctly.
For Haaukins specific; check resume functionality on teams to make sure that suspended VMs started without error. If there is an error happened restart VM which throws the error.
Guacamole mysql is NOT able to run. (Crashing)
Mysql requires following configuration file to be placed.
Pool overlaps with other one on this address space
This issue has been resolved completely, the reason of this error was that Haaukins is creating random Docker network, using some common private IP addresses. For instance, when you run docker-compose or a docker container, default docker network will assign an IP address from 192.168 or 172 which are common network range which Docker daemon is using. Haaukins was also using that range to generate random Docker networks, which has been updated from the version 2.4.2 on Haaukins project, the issue resolved.
Prune System
docker system prune -f
Prune volumes
docker volume prune -f
Unable to create database client
Having problems regarding to database client might be happened due to certificates error, or not running healthy haaukins-store, in these cases there are some things to care of ;
Ensure haaukins store is running correctly
It is always good to be sure about docker status of haaukins store, it can be checked through
docker-compose logs -f
which will feed your stdout with logs, if everything seems ok and no problem, you can close watching logs. If there is an error or something on logs, try to fix it with proper approach.Check version of store and haaukins daemon
In some cases, it might be the case where daemon and store do not share same version which means that some functionalities and features might not work. In those cases, client might not be able to create connection with database due to proto file differences. (- contract differences- ) Ensure that versions are matching, like features and functionalities released in both programs.
Check certificates
Since haaukins and store are using secure gRPC calls, certificates are required to be in place, however certificates which daemon (-for db client-) and store should share same certificates to have a secure gRPC connection. Make sure that there is no problem regarding to certificates.
Check ports
Configuration files for both client and daemon is crucial to run the program correctly, hence, it is good habit to check all values in configuration file. Especially, providing port numbers for store and daemon is quite important for communicating, check out them. If there is no issue regarding to configuration file and if you are still getting error when you run the program, check logs in depth.
Certificate Issue
Certificates are crucial for any component of haaukins which provides secure communication between clients and server, for this reason, it is quite important to setup auto-renew of certificates for all domains where haaukins component is using. For a domain where there is no certificate issued yet, following script can help to retrieve certificate from Let’s Encrypt, before using the script make sure that you are able to add TXT record on domain manager like Cloudflare.
Keep in mind that, example.domain.com
should be changed with your domain which you would like to get certificate on.
#!/bin/bash
# /etc/letsencrypt
# WHAT: This is the default configuration directory. This is where certbot will store all
# generated keys and issues certificates.
#
# /var/lib/letsencrypt
# WHAT: This is default working directory.
#
# certonly
# WHAT: This certbot subcommand tells certbot to obtain the certificate but not not
# install it. We don't need to install it because we will be linking directly to the
# generated certificate files from within our subsequent nginx configuration.
#
# -d
# WHAT: Defines one of the domains to be used in the certificate. We can have up to 100
# domains in a single certificate. In this case, we're obtaining a wildcard-subdomain
# certificate (which was just made possible!) in addition to the base domain.
#
# --manual
# WHAT: Tells certbot that we are going to use the "manual" plug-in, which means we will
# require interactive instructions for passing the authentication challenge. In this case
# (using DNS), we're going to need to know which DNS TXT entires to create in our domain
# name servers.
#
# --preferred-challenges dns
# WHAT: Defines which of the authentication challenges we want to implement with our
# manual configuration steps.
#
# --server https://acme-v02.api.letsencrypt.org/directory
# WHAT: The client end-point / resource that provides the actual certificates. The "v02"
# end-point is the only one capable of providing wildcard SSL certificates at this time,
# (ex, *.example.com).
#
docker run -it --rm --name letsencrypt \
-v "/etc/letsencrypt:/etc/letsencrypt" \
-v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
quay.io/letsencrypt/letsencrypt:latest \
certonly \
-d example.domain.com
-d *.example.domain.com \
--manual \
--preferred-challenges dns \
--server https://acme-v02.api.letsencrypt.org/directory
Once certificates are retrieved and placed, you have to have auto-renew as cron job or inside a docker environment in order to do not deal with renewing certificates manually all the time.
You can either download and use certbot-auto renew
command from directly host or you can perform same thing through docker,
for host integration following steps should be enough :
$ cd /etc/letsencrypt
$ wget https://dl.eff.org/certbot-auto && chmod a+x certbot-auto
$ ./certbot-auto renew
It will check certificates and renew them if they are about to expire, you can add that task into cron job.
$ crontab -e
0 0 * * 0 cd /etc/letsencrypt/ && ./certbot-auto renew
It will run every week at 00:00 on Sunday.
For Docker based approach, similar steps can be achieved as well, there are plenty of examples regarding to it, you may check following approach or create new one;