Reach us Join us
Contact our team to know more about our services
select webform
By submitting, you acknowledge that you've read and agree to our privacy policies, and Opcito may use the information provided for business purposes.
Reach us Join us
Become a part of our team
select webform
By submitting, you acknowledge that you've read and agree to our privacy policies, and Opcito may use the information provided for business purposes.
Akash Surwase 22 Nov 2019
Backup and restore Neo4j Graph Database using Ansible
22 Nov 2019

Backup and restore Neo4j Graph Database using Ansible

Increasing data volumes are like double-edged swords. The good thing about that is - a lot of data means a lot of information and a lot of value that comes with it. However, it also means a lot of technology to store and analyze this information is on its way. Thankfully, the graph database, with its agility, performance, and flexibility, is here to help us through this.

A graph database uses graph structures with two major elements for semantic queries, viz., nodes and relationships. Nodes are the entities in the graph, and relationships represent the connections between two nodes. In normal databases, data is the most important entity but a graph database treats the relationships between data with equal importance. Instead of narrowing down the approach towards data with a predefined model, the data is analyzed based on the interrelation between two nodes. Let’s take an example that will help you understand what I am talking about. Just to make sure everyone gets it, I will select everyone’s favorite topic.

Neo4J Graph structure

In this particular example, New England Patriots, Tom Brady, and Greater Boston are the nodes. The relationship between these nodes is represented over the arrow, and the arrow represents that these relationships are directional. The additional attributes associated with the nodes are properties as represented below the node.

Neo4j is open-source and provides an ACID-compliant transactional backend. Neo4j is referred to as a native graph database because of the efficient implementation of the property graph model down to the storage level. This means that the data is stored exactly as you whiteboard it, and the database uses pointers to navigate and traverse the graph. When it comes to production scenarios, Neo4j assures cluster support and runtime failover. Apart from these and the efficient store, process, and raise queries, here are some of the features and advantages that make Neo4j a popular choice among graphical databases -

  • It follows the Property Graph Data Model.
  • It supports UNIQUE constraints, which ensures the uniqueness of the data stored.
  • It contains a UI (Neo4j Data Browser) to execute CQL Commands that help to create and alter different storage unit for data. Neo4j CQL query language commands are in a readable format and very easy to learn.
  • The ACID(Atomicity, Consistency, Isolation, and Durability) rules support and ensure the validity of the data.
  • Support for Cypher API and Native Java API makes it easy to develop Java applications./li>
  • It is very easy to represent and retrieve/navigate connected data as well as it represents semi-structured data very easily.

Recently, in one of the projects I was working on, we wanted to automate the Neo4j database backup and restore process. We used Ansible’s configuration management capabilities to automate this Neo4j database Backup and Restore. Ansible is open-source and I love the way it enhances the scalability, consistency, and reliability of any IT environment. You can use Ansible to automate tasks such as - provisioning servers you need in your infrastructure or for configuration management or application deployment.

Now, let us see how the actual backup and restore process is like.

Backup Role

In case of unwanted scenarios like accidental deletion of data or a database instance gone down, to revert to the previous database state, we need to have the latest backup file of the running database. To save the existing Neo4j database, I have created this Backup role that can take a backup of the existing Neo4j instance and push the backup file to the S3 bucket.

Here is an Ansible playbook to take the backup of the Neo4j Database -

---

# tasks file for neo4j_backup
- name: Create a neo4j backup file with proper permissions
file:
path: /var/lib/neo4j/import/
state: touch
owner: neo4j
group: adm
mode: '0644'
- name: Copy cypher file on neo4j instance
template:
src: backup.cypher.j2
dest: /tmp/backup.cypher
- name: Taking a backup
shell: cat /tmp/backup.cypher | cypher-shell -u neo4j -p neo4j123
- name: install boto packages
pip:
name: boto botocore boto3
executable: pip-3.3
- name: Copy file to S3 bucket
vars:
ansible_python_interpreter: /usr/bin/python3
aws_s3:aws_access_key: ""
aws_secret_key: ""
bucket: neosrc: /var/lib/neo4j/import/
mode: put
object: ""
environment:
PATH: /usr/bin/python3

To execute the backup command in the cypher-shell, I have created a backup.cypher.j2 file in the roles template folder -

backup.cypher.j2
BEGIN
CALL apoc.export.graphml.all('', {useTypes:true, storeNodeIds:false});
:COMMIT

: To back up the existing Neo4j database, I have created the neo4j-backup role. The backup.cypher.j2 file will help to execute the backup command. The task written in the tasks/main.yml file will execute the cypher file with the help of a cat command to log in to the Neo4j database using the database credentials. Once the process is completed, the backup file will get pushed to the specific S3 bucket in AWS.

To execute the Neo4j Backup role, I have created a neo4j-backup.yml playbook -

---
- name: Neo4j-Backup
  hosts: neo4j
  gather_facts: true
  remote_user: ubuntu
  become: true
  roles:
	- neo4j_backup
To run this neo4j-backup role, the following command can be used:
ansible-playbook -i inventory neo4j-backup.yml -e AWS_ACCESS_KEY_ID=****************** -e AWS_SECRET_ACCESS_KEY=********************** -e environment_name=poc -vvv

Restore Role

Now that we have a backup file with us, let’s see how to restore your database to the previous state. To restore the Neo4j database, download the latest backup file from the S3 bucket and restore it on the newly created database instance.

Here is an Ansible playbook to restore the Neo4j database -

---

# tasks file for neo4j_restore
- name: install boto packages
pip:
name: boto botocore boto3
executable: pip-3.3
- name: Download file from S3 bucket
vars:
ansible_python_interpreter: /usr/bin/python3
aws_s3:
aws_access_key: ""
aws_secret_key: ""
bucket: neo4j-backup
object: ""
dest: /home/ubuntu/
mode: get
environment:
PATH: /usr/bin/python3
- name: Copy cypher file to restore neo4j database
template:
src: restore.cypher.j2
dest: /tmp/restore.cypher
- name: Restoring Backup
shell: cat /tmp/restore.cypher | cypher-shell -u neo4j -p neo4j123

To execute the restore command in the cypher-shell I have created a restore.cypher.j2 file in the roles template folder -

restore.cypher.j2
:BEGIN
CALL apoc.import.graphml('./', {batchSize: 10000, readLabels: true, storeNodeIds: false});
:COMMIT

To restore the backed-up Neo4j database on a new instance, I have created the neo4j-backup role. In this, I have created the restore.cypher.j2 file to execute the backup command. After that, in tasks/main.yml file, I have written the task to execute the cypher file with the help of cat command to log in to the neo4j database using database credentials. To restore the database, you need to pass the backup file that is on the S3 bucket with date and time.

To execute the Neo4j Backup role, I have created a neo4j-backup.yml playbook -

---

- name: Neo4j-Restore

  hosts: neo4j

  gather_facts: False

  remote_user: ubuntu

  become: true

  roles:

	- neo4j_restore

To run this neo4j-restore role, use the following command -

ansible-playbook -i inventory neo4j-restore.yml -e

 AWS_ACCESS_KEY_ID=************************ -e

 AWS_SECRET_ACCESS_KEY=******************** -e

 BACKUP_FILE=neo4j_backup_file.graphml -vvv

Using this backup role you will be able to take the backup from the Neo4j instance and push the backup file to the S3 bucket. Similarly, you can download the backup file pushed to the S3 bucket on the Neo4j instance to restore the Neo4j Database using the restore role. This was a quick rundown of how you can backup and restore Neo4j Database using Ansible. If you have any queries or want to share suggestions to ease the restore and backup process, feel free to comment.

Related Blogs

Subscribe to our feed

select webform